The new system will detect images called Child Sexual Abuse Material (CSAM) using a process called hashing, where images are transformed into unique numbers that correspond to that image.
Apple started testing the system on Thursday, but most U.S. iPhone users won’t be part of it until an iOS 15 update later this year, Apple said.
The move brings Apple in line with other cloud services which already scan user files, often using hashing systems, for content that violates their terms of service, including child exploitation images.
It also represents a test for Apple, which says that its system is more private for users than previous approaches to eliminating illegal images of child sexual abuse, because it uses sophisticated cryptography on both Apple’s servers and user devices and doesn’t scan actual images, only hashes.
But many privacy-sensitive users still recoil from software that notifies governments about the contents on a device or in the cloud, and may react negatively to this announcement, especially since Apple has vociferously defended device encryption and operates in countries with fewer speech protections than the U.S.
Law enforcement officials around the world have also pressured Apple to weaken its encryption for iMessage and other software services like iCloud to investigate child exploitation or terrorism. Thursday’s announcement is a way for Apple to address some of those issues without giving up some of its engineering principles around user privacy.
How it works
Before an image is stored in Apple’s iCloud, Apple matches the image’s hash against a database of hashes provided by National Center for Missing and Exploited Children (NCMEC). That database will be distributed in the code of iOS beginning with an update to iOS 15. The matching process is done on the user’s iPhone, not in the cloud, Apple said.
If Apple then detects a certain number of violating files in an iCloud account, the system will upload a file that allows Apple to decrypt and see the images on that account. A person will manually review the images to confirm whether or not there’s a match.
Apple will only be able to review images that match content that’s already known and reported to these databases — it won’t be able to detect parents’ photos of their kids in the bath, for example, as these images won’t be part of the NCMEC database.
If the person doing the manual review concludes the system did not make an error, then Apple will disable the user’s iCloud account, and send a report to NCMEC or notify law enforcement if necessary. Users can file an appeal to Apple if they think their account was flagged by mistake, an Apple representative said.
The system only works on images uploaded to iCloud, which users can turn off, Apple said. Photos or other images on a device that haven’t been uploaded to Apple servers won’t be part of the system.