Report: Client-Side Scanning Is An Insecure Nightmare Just Waiting To Be Exploited By Governments

from the 14-out-of-14-cybersecurity-experts-agree dept

In August, Apple declared that combating the spread of CSAM (child sexual abuse material) was more important than protecting millions of users who've never used their devices to store or share illegal material. While encryption would still protect users' data and communications (in transit and at rest), Apple had given itself permission to inspect data residing on people's devices before allowing it to be sent to others.

This is not a backdoor in a traditional sense. But it can be exploited just like an encryption backdoor if government agencies want access to devices' contents or mandate companies like Apple do more to halt the spread of other content governments have declared troublesome or illegal.

Apple may have implemented its client-side scanning carefully after weighing the pros and cons of introducing a security flaw, but there's simply no way to engage in this sort of scanning without creating a very large and slippery slope capable of accommodating plenty of unwanted (and unwarranted) government intercession.

Apple has put this program on hold for the time being, citing concerns raised by pretty much everyone who knows anything about client-side scanning and encryption. The conclusions that prompted Apple to step away from the precipice of this slope (at least momentarily) have been compiled in a report [PDF] on the negative side effects of client-side scanning, written by a large group of cybersecurity and encryption experts (Hal Abelson, Ross Anderson, Steven M. Bellovin, Josh Benaloh, Matt Blaze, Jon Callas, Whitfield Diffie, Susan Landau, Peter G. Neumann, Ronald L. Rivest, Jeffrey I. Schiller, Bruce Schneier, Vanessa Teague, and Carmela Troncoso). (via The Register)

Here's how that slippery slope looks. Apple's client-side scanning may be targeted, utilizing hashes of known CSAM images, but once the process is in place, it can easily be repurposed.

Only policy decisions prevent the scanning expanding from illegal abuse images to other material of interest to governments; and only the lack of a software update prevents the scanning expanding from static images to content stored in other formats, such as voice, text, or video.

And if people don't think governments will demand more than Apple's proactive CSAM efforts, they haven't been paying attention. CSAM is only the beginning of the list of content governments would like to see tech companies target and control.

While the Five Eyes governments and Apple have been talking about child sex-abuse material (CSAM) —specifically images— in their push for CSS, the European Union has included terrorism and organized crime along with sex abuse. In the EU’s view, targeted content extends from still images through videos to text, as text can be used for both sexual solicitation and terrorist recruitment. We cannot talk merely of “illegal” content, because proposed UK laws would require the blocking online of speech that is legal but that some actors find upsetting.

Once capabilities are built, reasons will be found to make use of them. Once there are mechanisms to perform on-device censorship at scale, court orders may require blocking of nonconsensual intimate imagery, also known as revenge porn. Then copyright owners may bring suit to block allegedly infringing material.

That's just the policy and law side. And that's only a very brief overview of clearly foreseeable expansions of CSS to cover other content, which also brings with it concerns about it being used as a tool for government censorship. Apple has already made concessions to notoriously censorial governments like China's in order to continue to sell products and services there. Additional demands will obviously be made if Apple implements scanning that can be exploited to locate and censor critics of the government.

There's plenty of bad stuff on the technical side, too. CSS is pretty much malware, the report says:

CSS is at odds with the least-privilege principle. Even if it runs in middleware, its scope depends on multiple parties in the targeting chain, so it cannot be claimed to use least-privilege in terms of the scanning scope. If the CSS system is a component used by many apps, then this also violates the least-privilege principle in terms of scope. If it runs at the OS level, things are worse still, as it can completely compromise any user’s device, accessing all their data, performing live intercept, and even turning the device into a room bug.

CSS has difficulty meeting the open-design principle, particularly when the CSS is for CSAM, which has secrecy requirements for the targeted content. As a result, it is not possible to publicly establish what the system actually does, or to be sure that fixes done in response to attacks are comprehensive. Even a meaningful audit must trust that the targeted content is what it purports to be, and so cannot completely test the system and all its failure modes.

Finally, CSS breaks the psychological-acceptability principle by introducing a spy in the owner’s private digital space. A tool that they thought was theirs alone, an intimate device to guard and curate their private life, is suddenly doing surveillance on behalf of the police. At the very least, this takes the chilling effect of surveillance and brings it directly to the owner’s fingertips and very thoughts.

While the report does offer some suggestions on how to make scanning less exploitable, the downsides are too numerous to conclude this can somehow be done safely. Given how many intrusive surveillance programs have already been justified with concerns about terrorism or the spread of illicit material, CSS -- no matter how implemented -- with become a tempting tool for governments to exploit.

In a world where our personal information lies in bits carried on powerful communication and storage devices in our pockets, both technology and laws must be designed to protect our privacy and security, not intrude upon it. Robust protection requires technology and law to complement each other. Client-side scanning would gravely undermine this, making us all less safe and less secure.

Despite this comprehensive report warning against the implementation of client-side scanning, there's a chance Apple may still roll its version out. And once it does, the pressure will be on other companies to do at least as much as Apple is doing to combat CSAM. The only upside is that if governments decide scanning should be used for reasons other than Apple intends, it has the power to shut its system down.

Filed Under: client side scanning, csam, iphones, security
Companies: apple

Apple Recognizes It Jumped Too Quickly On Its CSAM Detection System; Delays Implementation

from the good dept

Sometimes speaking out works. A month ago, Apple announced a series of new offerings that it claimed would be useful in fighting back against CSAM (child sexual abuse material). This is a real problem, and it's commendable that Apple was exploring ways to fight it. However, the major concern was how Apple had decided to do this. Despite the fact that a ton of experts have been working on ways to deal with this extremely challenging problem, Apple (in Apple fashion) went it alone and just jumped right in the deep end, causing a lot more trouble than necessary -- both because their implementation had numerous serious risks that Apple didn't seem to account for, and (perhaps more importantly) because the plan could wipe away years of goodwill in conversations between technologists, security professionals, human rights advocates and more in trying to seek solutions that better balance the risks.

Thankfully, with much of the security community, the human rights community, and others calling attention to Apple's dangerous approach, the company has now announced a plan to delay the implementation, gather more information, and actually talk to experts before deciding how to move forward. Apple put (in tiny print...) an update on the page where it announced these features.

Update as of September 3, 2021: Previously we announced plans for features intended to help protect children from predators who use communication tools to recruit and exploit them and to help limit the spread of Child Sexual Abuse Material. Based on feedback from customers, advocacy groups, researchers, and others, we have decided to take additional time over the coming months to collect input and make improvements before releasing these critically important child safety features.

It's good that the company has finally realized that it moved too quickly on this and hadn't necessarily understood the ramifications of its decision. It remains to be seen whether the company will actually do more in terms of realizing how dangerous its approach was, or if it will simply look to make a few cosmetic changes to the system.

Notably, this announcement came out just as Scientific American released an interesting article that warns that one of the child safety features -- one that had received less concern than the others -- might harm children more than it helps. This is the "communication safety in messages" feature that would scan iMessages of kids under 13, blur messages that the system deemed sexually explicit, and alert parents if the kid sends or opens such a message.

There were some initial concerns about this -- especially regarding LGTBQ children whose parents might not be understanding. However, many of those initial concerns were quieted by the details of the program -- including the fact that it was opt-in and was designed to be transparent to both the kids and the parents what was happening (so no sneaky surveillance or surprise alerts). It's also specifically designed for child accounts that are set up in Family Sharing.

However, as the SciAm article notes, this system still should raise concerns, because it would make kids think they're always being watched:

In fact, even by having this feature, we are teaching young people that they do not have a right to privacy. Removing young people’s privacy and right to give consent is exactly the opposite of what UNICEF’s evidence-based guidelines for preventing online and offline child sexual exploitation and abuse suggest. Further, this feature not only risks causing harm, but it also opens the door for wider intrusions into our private conversations, including intrusions by government.

We need to do better when it comes to designing technology to keep the young safe online. This starts with involving the potential victims themselves in the design of safety systems. As a growing movement around design justice suggests, involving the people most impacted by a technology is an effective way to prevent harm and design more effective solutions. So far, youth haven’t been part of the conversations that technology companies or researchers are having. They need to be.

Again, there are real concerns here, and parents obviously want to protect their children. But over and over again we've seen the way you do that is by teaching them how to handle dangerous situations, rather than adding yet another layer of surveillance. The surveillance teaches them not only that they have no privacy, but similarly takes away their own agency in learning how to deal with difficult situations themselves.

Filed Under: children, client side scanning, csam, scanning, security
Companies: apple

Researchers Who Built Similar System Explain Why Apple's CSAM Scanning System Is Dangerous

from the it's-not-good dept

Jonathan Mayer, a Princeton University professor and former chief technologist at the FTC, is one of the smartest people I know. Every time I've spoken with him I feel like I learn something. He's now written a quite interesting article for the Washington Post noting how he, and a graduate researcher at Princeton, Anunay Kulshrestha, actually built a CSAM scanning system similar to the one that Apple recently announced, which has security experts up in arms over the risks inherent to the approach.

Mayer and Kulshretha note that while Apple is saying that people worried about their system are misunderstanding it, they are not. They know what they're talking about -- and they still say the system is dangerous.

We wrote the only peer-reviewed publication on how to build a system like Apple’s — and we concluded the technology was dangerous. We’re not concerned because we misunderstand how Apple’s system works. The problem is, we understand exactly how it works.

Mayer and Kulshretha are certainly not making light of the challenges and problems associated with stopping CSAM. It's why they started their project to see if there was an effective way to identify CSAM even in end-to-end encrypted systems. And they even built a system to do just that. And what they found is that the risks are simply too great.

We were so disturbed that we took a step we hadn’t seen before in computer science literature: We warned against our own system design, urging further research on how to mitigate the serious downsides. We’d planned to discuss paths forward at an academic conference this month.

That dialogue never happened. The week before our presentation, Apple announced it would deploy its nearly identical system on iCloud Photos, which exists on more than 1.5 billion devices. Apple’s motivation, like ours, was to protect children. And its system was technically more efficient and capable than ours. But we were baffled to see that Apple had few answers for the hard questions we’d surfaced.

The potential dangers that Mayer and Kulshretha are exactly what many had warned about when Apple announced its plans:

After many false starts, we built a working prototype. But we encountered a glaring problem.

Our system could be easily repurposed for surveillance and censorship. The design wasn’t restricted to a specific category of content; a service could simply swap in any content-matching database, and the person using that service would be none the wiser.

A foreign government could, for example, compel a service to out people sharing disfavored political speech. That’s no hypothetical: WeChat, the popular Chinese messaging app, already uses content matching to identify dissident material. India enacted rules this year that could require pre-screening content critical of government policy. Russia recently fined Google, Facebook and Twitter for not removing pro-democracy protest materials.

We spotted other shortcomings. The content-matching process could have false positives, and malicious users could game the system to subject innocent users to scrutiny.

That's already pretty damning. But there's some other even more damning information that has come out as well. As we noted in our earlier posts (and as Mayer and Kulshretha noted in their research), the risk of false positives is extremely high. And late last week, the hypothetical became a lot more real. Someone reverse engineered the NeuralHash algorithm that Apple is using and put it on Github.

It did not take long for people to point that, first of all, there are collisions with totally unrelated images:

Now, you might say that since that second image is not really an image, maybe it's not that big a deal? Well, about that...

And, of course, one of the issues with any hash-based system is the idea that subtle changes to the image would output a totally different hash, thus making it easy for some to simply route around the scanning. Apple had suggested its system could defeat that, but...

This is part of the reason that we highlighted earlier that security researchers are so up in arms about this. Apple seemingly ignored so much of the research and conversations that were happening about these approaches, and just barged right in announcing that it had a solution without exploring the tradeoffs and difficulties associated with it -- leaving that for security experts to point out afterwards.

Apple is trying to downplay these findings, saying that it expected the collisions at least, and that it's system would also do a separate server side hashing comparison which would stop the false collisions. Though, as Bruce Schneier points out, if this was "expected," then why wasn't it discussed in the initial details that were released? Similarly, I have yet to see a response to the flip side issue of changing the images in a way that fool NeuralHash while still looking the same.

I know Apple keeps wanting to insist that it's thought through all of this, but it doesn't seem to have thought through any of how the security community would see all of this, and it's after-the-fact scrambling is not exactly reassuring.

Filed Under: backdoors, client side scanning, csam, encryption, hash clash, jonathan mayer, surveillance
Companies: apple

Apple Undermines Its Famous Security 'For The Children'

from the this-is-dangerously-dumb dept

Apple is somewhat famous for its approach to security on its iPhones. Most famously, Apple went to court to fight the FBI's demand that they effectively insert a backdoor into its on-phone encryption (by being able to force an update to the phone). Apple has tons of goodwill in the security community (and the public) because of that, though not in the law enforcement community. Unfortunately, it appears that Apple is throwing away much of that good will and has decided to undermine the security of its phone... "for the children" (of course).

This week, Apple announced what it refers to as "expanded protections for children." The company has been receiving lots of pressure from certain corners (including law enforcement groups who hate encryption), claiming that its encryption was helping hide child sexual abuse material (CSAM) on phones (and in iCloud accounts). So Apple's plan is to introduce what's generally called "client-side scanning" to search for CSAM on phones as well as a system that scans iCloud content for potentially problematic content. Apple claims that it's doing this in a manner that is protective of privacy. And, to be fair, this clearly isn't something that Apple rolled out willy-nilly without considering the trade-offs. It's clear from Apple's detailed explanations of the new "safety" features, that it is trying to balance the competing interests at play here. And, obviously, stopping the abuse of children is an important goal.

The problem is that, even with all of the balancing Apple has done here, it's definitely moved down a very dangerous, and very slippery slope towards using this approach for other things.

Apple's brief description of its new offerings are as follows:

Apple is introducing new child safety features in three areas, developed in collaboration with child safety experts. First, new communication tools will enable parents to play a more informed role in helping their children navigate communication online. The Messages app will use on-device machine learning to warn about sensitive content, while keeping private communications unreadable by Apple.

Next, iOS and iPadOS will use new applications of cryptography to help limit the spread of CSAM online, while designing for user privacy. CSAM detection will help Apple provide valuable information to law enforcement on collections of CSAM in iCloud Photos.

Finally, updates to Siri and Search provide parents and children expanded information and help if they encounter unsafe situations. Siri and Search will also intervene when users try to search for CSAM-related topics.

Some of the initial concerns about these descriptions -- including fears that, say, LGBTQ+ children might be outed to their parents -- have been somewhat (though not entirely) alleviated with the more detailed explanation. But that doesn't mean there aren't still very serious concerns about how this plays out in practice and what this means for Apple's security.

First, there's the issue of client-side scanning. As an EFF post from 2019 explains, client-side scanning breaks end-to-end encryption. In the EFF's latest post about Apple's announcement, it includes a quick description of how this introduces a backdoor:

We’ve said it before, and we’ll say it again now: it’s impossible to build a client-side scanning system that can only be used for sexually explicit images sent or received by children. As a consequence, even a well-intentioned effort to build such a system will break key promises of the messenger’s encryption itself and open the door to broader abuses.

All it would take to widen the narrow backdoor that Apple is building is an expansion of the machine learning parameters to look for additional types of content, or a tweak of the configuration flags to scan, not just children’s, but anyone’s accounts. That’s not a slippery slope; that’s a fully built system just waiting for external pressure to make the slightest change. Take the example of India, where recently passed rules include dangerous requirements for platforms to identify the origins of messages and pre-screen content. New laws in Ethiopia requiring content takedowns of “misinformation” in 24 hours may apply to messaging services. And many other countries—often those with authoritarian governments—have passed similar laws. Apple’s changes would enable such screening, takedown, and reporting in its end-to-end messaging. The abuse cases are easy to imagine: governments that outlaw homosexuality might require the classifier to be trained to restrict apparent LGBTQ+ content, or an authoritarian regime might demand the classifier be able to spot popular satirical images or protest flyers.

We’ve already seen this mission creep in action. One of the technologies originally built to scan and hash child sexual abuse imagery has been repurposed to create a database of “terrorist” content that companies can contribute to and access for the purpose of banning such content. The database, managed by the Global Internet Forum to Counter Terrorism (GIFCT), is troublingly without external oversight, despite calls from civil society. While it’s therefore impossible to know whether the database has overreached, we do know that platforms regularly flag critical content as “terrorism,” including documentation of violence and repression, counterspeech, art, and satire.

It's actually difficult to find any security experts who support Apple's approach here. Alec Muffett sums it up in a single tweet:

This is the very slippery slope. If we somehow believe that governments won't demand Apple cave on a wide variety of other types of content, you haven't been paying attention. Of course, Apple can claim that it will stand strong against such demands, but now we're back to being entirely dependent on trusting Apple.

As noted above, there were some initial concerns about the parent notifications, but as EFF's description notes, the rollout here does include some level of consent by users before their parents are notified, but it's still quite problematic:

In these new processes, if an account held by a child under 13 wishes to send an image that the on-device machine learning classifier determines is a sexually explicit image, a notification will pop up, telling the under-13 child that their parent will be notified of this content. If the under-13 child still chooses to send the content, they have to accept that the “parent” will be notified, and the image will be irrevocably saved to the parental controls section of their phone for the parent to view later. For users between the ages of 13 and 17, a similar warning notification will pop up, though without the parental notification.

Similarly, if the under-13 child receives an image that iMessage deems to be “sexually explicit”, before being allowed to view the photo, a notification will pop up that tells the under-13 child that their parent will be notified that they are receiving a sexually explicit image. Again, if the under-13 user accepts the image, the parent is notified and the image is saved to the phone. Users between 13 and 17 years old will similarly receive a warning notification, but a notification about this action will not be sent to their parent’s device.

This means that if—for instance—a minor using an iPhone without these features turned on sends a photo to another minor who does have the features enabled, they do not receive a notification that iMessage considers their image to be “explicit” or that the recipient’s parent will be notified. The recipient’s parents will be informed of the content without the sender consenting to their involvement. Additionally, once sent or received, the “sexually explicit image” cannot be deleted from the under-13 user’s device.

Whether sending or receiving such content, the under-13 user has the option to decline without the parent being notified. Nevertheless, these notifications give the sense that Apple is watching over the user’s shoulder—and in the case of under-13s, that’s essentially what Apple has given parents the ability to do.

It is also important to note that Apple has chosen to use the notoriously difficult-to-audit technology of machine learning classifiers to determine what constitutes a sexually explicit image. We know from years of documentation and research that machine-learning technologies, used without human oversight, have a habit of wrongfully classifying content, including supposedly “sexually explicit” content. When blogging platform Tumblr instituted a filter for sexual content in 2018, it famously caught all sorts of other imagery in the net, including pictures of Pomeranian puppies, selfies of fully-clothed individuals, and more. Facebook’s attempts to police nudity have resulted in the removal of pictures of famous statues such as Copenhagen’s Little Mermaid. These filters have a history of chilling expression, and there’s plenty of reason to believe that Apple’s will do the same.

There remains a real risk of false positives in this kind of system. There's a very worth reading blog post explaining how automated matching technologies fail, often in catastrophic ways. You really need to read that entire post as brief excerpts wouldn't do it justice -- but as it notes, the risk of false positives here are very high, and the cost of such false positives can be catastrophic. Obviously, CSAM is also catastrophic, so you can see how there is a real challenge in balancing those interests, but there are legitimate concerns that with this approach it's unclear if the balance is properly calibrated.

Obviously, Apple is trying to walk a fine line here. No one wants to be supporting CSAM distribution. But, once again, all the pressure on Apple feels like people blaming the tool for (serious) abuses by its users, and in demanding a "solution," opening up a very dangerous situation. If there were some way to guarantee that these technologies wouldn't be abused, or mess up, you could kind of see how this makes sense. But history has shown time and time again that neither of these things is really possible. Opening up this hole in Apple's famous security means more demands are coming.

Filed Under: client side scanning, csam, encryption, for the children, parent notification, privacy, security
Companies: apple