22 January 2006

The spam problem part 2: The dismal science

In Part 1 I took a “from first principles” look at the spam problem, and concluded that the only way to actually solve the problem was to make people pay to send e-mail.

Now, it’s time to look at what I mean by that—because there are almost as many ways to implement “pay to send” as there are ways to implement filtering.

This is going to be a bit more technical than part 1. I’m going to assume you know basically how SMTP e-mail works. If not, there are tutorials available.

2.1: Reverse the charges

One approach I’ve considered in the past is making e-mail a “pull” rather than “push” protocol. While I came up with the idea myself, D.J. Bernstein has a similar proposal he calls Internet Mail 2000.

Currently, a blast of spam immediately spreads out across thousands of mail servers, occupying lots of disk space and eating the bandwidth of innocent third parties. But, what if e-mail wasn’t immediately sent to the recipient? Suppose it worked like this instead:

  • You send e-mail to John via your ISP’s mail server. Your ISP’s server stores the e-mail and sends a brief “ping” to the John’s ISP.
  • John’s ISP records the fact that there is mail waiting for John at your ISP.
  • Some time later, John logs in. His mail client is handed a list of URLs of locations where there is mail waiting for him.
  • John’s e-mail client connects to each location and fetches the mail.
  • Once it has confirmation that the mail has been collected by John, your ISP’s mail server can delete its copy, file it for your reference, or whatever.

In other words, e-mail becomes more like RSS. There are some major advantages to this way of doing things:

  • Confirmation is instant and reliable. Your ISP’s mail server can tell you for certain whether John’s e-mail client has picked up the message you sent him.
  • Unlike many other anti-spam proposals, this one lets mailing lists keep working. In fact, they can become more efficient, as a single copy of the e-mail can reside on a single server and be available to all list recipients. Going away for two weeks? Doesn’t matter, your mailing list traffic won’t be piling up in your inbox and overflowing your mail quota.
  • There’s no way to hide the source of the e-mail without making it undeliverable.
  • Similarly, attempting to pretend to be someone else would make the recipient’s mail client connect to the impersonated party’s mail server, get a “no such message” response, and you wouldn’t see the forgery
  • There are no bounce messages, so you don’t have to deal with them. Which is good, because a lot of badly-written software doesn’t deal with them properly.
  • Notifications can be very small, so your allocated inbox can be enormous without taking up lots of disk space. For example, if we allow 80 characters for a URL, a typical cheap 5MB ISP mailbox could accept over 60,000 new e-mails before overflowing.
  • The fetching protocol could be made to work like NNTP, POP3 and IMAP—you could look at the headers and decide whether to read the e-mail, and if you decided to just delete it, the bandwidth of transmitting the message across the Internet would never be wasted.
  • If you send an e-mail then realize you made a mistake, you will often be able to un-send it.

Now consider what happens if someone sets up an account under such a system, and uses it to spam 1,000,000 people.

  • If half a dozen people complain within half an hour, and the ISP yanks the account, anyone checking their e-mail later on won’t even see the spam.Instead of 1,000,000 copies of the spam traveling across the Internet, maybe a few hundred do.
  • If everyone fetches the spam, the sender is effectively subjected to a distributed denial-of-service attack, automatically. If it can’t cope, the mail recipients’ client programs will time out and the recipients won’t see the spam. This means that cheap throwaway accounts, which won’t be set up with fat connections to deal with massive incoming load, will no longer be useful for spamming.
  • A commercial spamming business can’t send out copies of their spam to hundreds or thousands of recipients per copy, and rely on other people’s mail servers to multiply up the e-mail load—instead, they have to pay for every byte of every copy fetched by a recipient. Basically, they now inherently have to pay for the Internet bandwidth that they consume.

I don’t think that making e-mail a “pull” protocol would eliminate spam, but I think it would reduce it drastically, as well as having other benefits.

When I proposed “pull” e-mail to the IETF Anti-Spam Research Group, one major objection was: “What about botnets?”

In case you aren’t aware, a lot of spam is currently sent using networks of hijacked Windows machines that have been compromised by trojan horse programs, or back doors installed when the user browser the web using Internet Explorer. The objection to “pull” e-mail was: What about the user who suddenly gets a bandwidth bill from his ISP because his Windows machine was hijacked and used to send thousands of spam messages?

My answer is simple: That’s not a bug, it’s a feature.

Currently any clueless yutz can connect his Windows PC to the Internet and make a nuisance of himself, and not have to pay the price. Furthermore, Microsoft can continue to release buggy, security-hole-ridden code, and not pay any price for that either. Instead, the rest of us pay. Our mailboxes fill with Windows viruses being e-mailed, bounce messages caused by Windows viruses using our e-mail addresses fraudulently, and spam sent via Windows botnets.

Maybe “pull” e-mail would mean that Windows users who didn’t keep their systems clean would face unexpected ISP charges. Maybe Windows users would need to take out insurance against Microsoft’s shoddy software making them liable for bandwidth bills. Maybe, just maybe, that would make them consider a smarter option—like not using Windows. Or maybe there would be a big class action lawsuit against Microsoft for foisting dangerously crappy software on a public too technically unknowledgeable to know better.

So yeah, it’s all good.

2.2: Hashcash

A second approach to making the sender pay to send e-mail is called “hashcash”.

The idea is that the recipient e-mail system checks for a special “stamp” in the message. The stamp is designed to be very hard to compute, so that a typical desktop PC will have to sit and process for a few seconds to work it out.

The idea is that the sender is “paying” with CPU time, something that has financial value but isn’t actually money. Since most people don’t send many e-mail messages, they’ll be able to compute all the hashcash “stamps” they need and then some, without really being inconvenienced. Similarly, a business would be able to dedicate a PC or a more powerful server to generating hashcash “stamps” for the hundreds or thousands of system-generated e-mails it needs to send, and e-mail generated by humans could be stamped by the user’s desktop machine.

Spammers, the theory goes, simply won’t have the CPU power to generate millions and millions of hashcash “stamps”. And as computers get faster, the hashcash “postage” required can be raised.

It’s certainly an ingenious idea. Unfortunately, I don’t think it’s practical.

For starters, it’s quite hard to calibrate how much CPU time a hashcash stamp should require. There are a lot of less well-off people out there with computers that have clock speeds in the hundreds of megahertz, or even less. There are also a lot of mobile devices with low power CPUs—I can send e-mail from my phone, and frankly I like that.

Proponents of hashcash will say “Ah, well, you could have a server that generated the hashcash for your CPU-starved mobile device.” The problem is, once you do that, you’re back to present-day SMTP, except that now the SMTP server is chewing even more resources.

Then there are the environmental or aesthetic considerations. My laptop has a long battery life partly because the CPU sits idle a lot of the time, and when it does the system automatically switches into a low power mode until I next press a key. I don’t want my laptop to have to spend 30 seconds generating a hashcash stamp for each e-mail I send when I’m working through my e-mail backlog, because that will cut into my battery time noticeably.

The most damning problem, though, is the aforementioned botnets. A spammer with 10,000 trojaned Windows boxes can compute 10,000 hashcash stamps at once.If I have the ability to send out a few hundred messages—which is reasonable, if I run a mailing list—then that means the spammer has the ability to send a million spams.

So, that’s about it for hashcash, I think.

3.1: Pay actual cash per message

When the British postal service faced the spam problem in the 1830s, the solution was the postage stamp, invented in 1837. Digital postage stamps have been proposed as a solution to the e-mail spam problem.

There are a number of problems with digital postage stamps, however; some are technical, some cultural.

  • The net has no central authority that could decide postage rates and sell stamps. The legacy of Verisign’s mismanagement of the domain name system is that nobody is going to want to put a similar entity in a position of power over the net’s e-mail infrastructure.

  • User acceptance is a problem. Nobody wants to have to pay every time they send e-mail to their friends.

  • Many useful mailing lists would simply die if they had to pay for digital stamps for every message sent. You could kiss Yahoo Groups goodbye for starters. To see the effect of imposing per-event fees on previously free activity, just take a look at the meetup.com fiasco.

These problems aren’t insurmountable, but they do require a slightly different approach to the plain “digital stamp” approach.

3.2: Attention bonds

I’m going to go ahead and state a few things which I consider to be absolute requirements for any system that involves people paying actual money to send e-mail:

  • There must be a way to exempt mailing lists, friends, family, customers, etc. This is pretty simple; you just need to add whitelisting.

  • The fee for the message must be set by the recipient. I am the only one who can decide what my time is worth. In fact, most people vary their estimates of how much their time is worth based on context, and I would want to be able to do the same for our hypothetical new e-mail system.

    For instance, I would probably want to set a high fee for sending an e-mail to my mobile phone and actually interrupting me immediately; a lower fee for an e-mail to my regular inbox; and a tiny fee for delivery to a “dogwash” folder that I read when I’m done with my high priority stuff. I’d probably also exempt mail that was a reply to mail I had sent, mail about my software, and so on.

  • The sender must be able to limit the fees he will accept. Ideally the sender would be presented with the fee at sending time; if that is not possible, the sender will at least need to be able to set a limit for how much he is willing to pay.

  • The expected cost, for well behaved users, must be zero—or so close to zero as to be unimportant. This is the key to acceptability. If I’m an ordinary person sending non-abusive, non-spammy e-mail, it should remain cost-free for me to do so.

    The way to implement this requirement is quite subtle: rather than making the fee a postage stamp, you make it a refundable bond. That is, when I send you an e-mail, I put up (for example) 10¢ in cash as part of my promise that the e-mail will be worth your while to read. When you read the e-mail, you decide whether you agree with me that the e-mail was a waste of your time or not. If it was, you pocket the 10¢. If not, however, the money is refunded to me.

    Recall that as discussed in part 1, it’s implausible to have a machine decide whether something is spam or not. The beauty of this scheme is that you don’t need to try!

  • Participation must be optional, and introduction must be graceful. Some people are never going to accept even the possibility of paying for e-mail, so it must be possible to set the fee for a given e-mail address to zero. In addition, while the system is being introduced there will be many people who don’t have the ability to use it, so it must be possible to set up backup addresses with zero fees for use during the transition period.

  • The system must be decentralized. A single transaction processor is not going to work for sound political reasons.

So, here’s an outline of how I see the system working. I’ll go through several scenarios. First, a mailing list:

  • Julie on the pet skunk list e-mails a picture of an exceptionally cute skunk.

  • The message arrives at YahooGroups. Yahoo does the usual header rewriting before redirecting a copy to me.

  • My mail server looks at the incoming e-mail. It sees that it was sent to the alias I have set up for skunk mail (known only to me and YahooGroups), and that it came from YahooGroups. That matches a whitelist entry, so the mail is delivered free of charge.

Now let’s imagine a friend sending me e-mail:

  • Dan writes me an e-mail and hits send.

  • The mail client contacts my e-mail server, and does the initial negotiation. The server recognizes Dan’s e-mail address, and replies with a very small bond requirement—say, 5¢.

  • Since 5¢ is less than Dan’s configured maximum of 50¢, the client goes ahead and sends the mail with 5¢ cash attached.

  • I receive the e-mail and read it. I click the normal button to move to the next message; that is, I don’t click the “This was spam!” button. The mail client interprets that as acceptance of the message.

  • The 5¢ bond fee is refunded to Dan.

Note that in this example, the bond I charge friends is not zero. If it was, a spammer who found out my friends’ e-mail addresses could fake e-mail as being from them, and flood my mailbox for free. Obviously, people who are happy to take that risk could set a bond of zero for friends’ e-mail addresses if they wanted. Note, however, that the expected cost for Dan is the same either way–zero!

Finally, let’s imagine some random person on the Internet sending me e-mail:

  • Hugh Cares writes an e-mail asking if I’ve updated RedPill to work on Tiger. He hits the Send button.

  • His e-mail client does the same negotiation as Dan’s. This time, because the sender address isn’t recognized, the server responds with my “random bozo” bond requirement of 50¢.

  • Hugh is a penniless college student, so this exceeds the discretionary limit he has configured in his e-mail client. The client puts up a dialog box asking him to confirm that he wants to put up the cash and send the e-mail.

  • Hugh decides to check MacUpdate first, finds the answer to his question, and deletes the e-mail. Everybody wins!

So we see that this system can have beneficial effects even for a population of non-abusive users of e-mail, in that it may help to discourage unnecessary e-mail. (If you don’t get unnecessary e-mail, please tell me your secret.)

I think attention bonds are the only option for ending spam that will actually work. In part 3, I’ll go through a few of the objections commonly brought up when the possibility of involving actual cash in e-mail sending is raised.

Related posts

If you liked that post, you might like one of these:

© mathew 2017