A screeching voice of the minority

Apple has built technology that can scan all your photos for forbidden content before you even upload them

Apple’s new plans for scanning people’s photos in iOS 15 have come under fire this week. Here’s an outline of how the new system works.

First of all, some smart software developers came up with an algorithm called NeuralHash that can turn an image into a digital hash code — a string of hexadecimal characters representing a large binary number. A hash might look like this:


Note that the hash contains absolutely no useful information about what’s in the image. Also, note that there’s no way to turn the hash back into the image.

The algorithm is carefully designed so that changes to the image such as cropping it, rotating it, adjusting the exposure or masking parts of it, still result in the same code. That’s so that criminals can’t just disguise the image with a trivial change.

A government-funded organization called the National Center for Missing & Exploited Children (NCMEC) collects photos showing child sexual abuse. They process the photos with NeuralHash, and make the list of forbidden hash codes available to tech companies. This is done to help the tech companies scan for known child abuse photos quickly, without needing to expose moderators to the material. Pretty much every cloud provider uses the database to scan photos you upload to their servers, including Facebook and Google.

What’s new is that with iOS 15, Apple will be embedding a copy of NeuralHash into iOS, along with a copy of the government database of forbidden hash codes. Your phone will run your images through NeuralHash, and compare to the list of forbidden hash codes. If there’s a match, some data will be uploaded to Apple’s servers when you upload the image. The data contains part of the information needed to identify you — but not all of it. However, if you have enough forbidden images, Apple will receive enough information to unmask your identity, and they will do so, and pass the information on to law enforcement. (The problem of requiring a certain number of flags before unmasking the user is handled by a technically complicated technique known as private set intersection.)

Note that so far, Apple are only planning to do the data upload if you choose to upload your images to iCloud.

So to summarize:

  • Government gives Apple a list of hex codes that it says are the hashes of child sexual abuse material.
  • Apple checks your photos against this list of forbidden image hashes. (For now, only if the image is going to be uploaded.)
  • If there are enough matches, Apple reports you to law enforcement.

You can probably immediately see some problems with this.

In the past, companies scanned your images on their servers. If you didn’t upload the image, it wouldn’t be scanned. Now, with Apple’s system, for the first time it has become technically possible for all your images to be scanned whether you upload them or not.

Sure, Apple say that they’re not going to scan and report photos unless you upload them to iCloud, but there’s nothing in the technology to enforce that. It’s a pretty safe bet that once the technology is rolled out, Apple will be under immense pressure to just scan everything. Politicians will no doubt ask why Apple is choosing to protect child abusers when it could be scanning their photos for CSAM. Do you think Apple will be able to stand up to that pressure?

Another big problem is that Apple doesn’t know if the forbidden hash codes are actually from child sexual abuse material, and nor do we. We are expected to take the government’s word for it. You might be happy to believe the National Center for Missing & Exploited Children, but would you be happy to trust China’s Ministry for Information Industry?

We already know that China considers this image harmful enough that people have gone to jail for years for sharing it:

Tank Man

And yes, China’s WeChat filters the image.

I’m betting the Chinese government are already preparing a list of hashes of forbidden images for Apple to search every iPhone for. I’m also certain the Chinese government aren’t going to be satisfied with Apple only checking images if they’re uploaded to iCloud.

But that’s China, right? Well, in the US police have used facial recognition software to try to identify Black Lives Matter protesters. Social media content has been used to identify and track protesters. If you’re right wing, look at how privately taken video and photos have been used to track down the January 6th Capitol rioters, and imagine how much easier it would be if police could get a copy of every relevant photo along with the identity of the person who took it.

Apple has already admitted that their content scanning will be rolled out to other countries. I bet the more authoritarian of those countries can hardly wait to tell Apple to go ahead and run it across all photos regardless of whether they’re being uploaded.

So while scanning for child sexual abuse material isn’t new, there’s something new and incredibly dangerous about making it possible for everyone’s own phones to do the scanning automatically on every photo. It’s a dangerous technology to have built, even if Apple promises not to turn that option on.

You might think that it’s worth it to catch child predators and prevent abuse, but sadly this kind of photo scanning isn’t much use for that.

First of all, the hash database only catches old photos that the NCMEC have in their database. It does nothing to prevent or discourage the production of new material. If anything, I’d guess that it makes it more desirable, because if a pedophile can’t share his old photo collection with his online network then he’s going to want new images, isn’t he?

Make no mistake, child sexual abuse enabled by digital platforms is a massive problem. However, some 95% of the child sexual abuse material reported in 2020 was on Facebook, and Facebook already scans photos and reports illegal material to the NCMEC. How much attention do you think police gave each of those 20 million reports from Facebook? That’s about how much attention they’ll give anything flagged by Apple. With those numbers, it doesn’t seem like hashing known images has done much to solve the problem.

So we have a technology ripe for massive government abuse being rolled out as an ineffective fix to a societal problem. I’m sure some smart people at Apple pointed this out, but apparently management didn’t listen.

At this point, I think the only thing that might make Apple listen would be if everyone decided to boycott iOS 15. Just don’t upgrade. Apple always likes to boast about upgrade stats, so you can bet they’d notice. I’m not hopeful that a sizable percentage of people will take part in any such boycott, however.