Researchers Who Built Similar System Explain Why Apple's CSAM Scanning System Is Dangerous
from the it's-not-good dept
Jonathan Mayer, a Princeton University professor and former chief technologist at the FTC, is one of the smartest people I know. Every time I've spoken with him I feel like I learn something. He's now written a quite interesting article for the Washington Post noting how he, and a graduate researcher at Princeton, Anunay Kulshrestha, actually built a CSAM scanning system similar to the one that Apple recently announced, which has security experts up in arms over the risks inherent to the approach.
Mayer and Kulshretha note that while Apple is saying that people worried about their system are misunderstanding it, they are not. They know what they're talking about -- and they still say the system is dangerous.
We wrote the only peer-reviewed publication on how to build a system like Apple’s — and we concluded the technology was dangerous. We’re not concerned because we misunderstand how Apple’s system works. The problem is, we understand exactly how it works.
Mayer and Kulshretha are certainly not making light of the challenges and problems associated with stopping CSAM. It's why they started their project to see if there was an effective way to identify CSAM even in end-to-end encrypted systems. And they even built a system to do just that. And what they found is that the risks are simply too great.
We were so disturbed that we took a step we hadn’t seen before in computer science literature: We warned against our own system design, urging further research on how to mitigate the serious downsides. We’d planned to discuss paths forward at an academic conference this month.
That dialogue never happened. The week before our presentation, Apple announced it would deploy its nearly identical system on iCloud Photos, which exists on more than 1.5 billion devices. Apple’s motivation, like ours, was to protect children. And its system was technically more efficient and capable than ours. But we were baffled to see that Apple had few answers for the hard questions we’d surfaced.
The potential dangers that Mayer and Kulshretha are exactly what many had warned about when Apple announced its plans:
After many false starts, we built a working prototype. But we encountered a glaring problem.
Our system could be easily repurposed for surveillance and censorship. The design wasn’t restricted to a specific category of content; a service could simply swap in any content-matching database, and the person using that service would be none the wiser.
A foreign government could, for example, compel a service to out people sharing disfavored political speech. That’s no hypothetical: WeChat, the popular Chinese messaging app, already uses content matching to identify dissident material. India enacted rules this year that could require pre-screening content critical of government policy. Russia recently fined Google, Facebook and Twitter for not removing pro-democracy protest materials.
We spotted other shortcomings. The content-matching process could have false positives, and malicious users could game the system to subject innocent users to scrutiny.
That's already pretty damning. But there's some other even more damning information that has come out as well. As we noted in our earlier posts (and as Mayer and Kulshretha noted in their research), the risk of false positives is extremely high. And late last week, the hypothetical became a lot more real. Someone reverse engineered the NeuralHash algorithm that Apple is using and put it on Github.
It did not take long for people to point that, first of all, there are collisions with totally unrelated images:
Well that didn't take long.
“Can you verify that these two images collide?“
“Yes! I can confirm that both images generate the exact same [NeuralHash] hashes on my iPhone. And they are identical to what you generated here.”
“59a34eabe31910abfb06f308”https://t.co/cXS4wvBMG9 pic.twitter.com/FFIhNuGIBo
— Kenn White (@kennwhite) August 18, 2021
Now, you might say that since that second image is not really an image, maybe it's not that big a deal? Well, about that...
Make your own NeuralHash collisions with https://t.co/sV1dF4Oulb pic.twitter.com/0jGPFSDFi6
— Anish Athalye (@anishathalye) August 19, 2021
And, of course, one of the issues with any hash-based system is the idea that subtle changes to the image would output a totally different hash, thus making it easy for some to simply route around the scanning. Apple had suggested its system could defeat that, but...
Do you see a difference between these two images? Apple's NeuralHash thinks they differ on 52 of 96 bits (ed570844756690de887ceec3 vs 2d574044756690de887cfe43).
No I won't tell you how I did it, but it runs on CPU in about 10 seconds. NeuralHash is trivial to evade. pic.twitter.com/0D2nsjvUVJ
— Rich Harang (@rharang) August 19, 2021
This is part of the reason that we highlighted earlier that security researchers are so up in arms about this. Apple seemingly ignored so much of the research and conversations that were happening about these approaches, and just barged right in announcing that it had a solution without exploring the tradeoffs and difficulties associated with it -- leaving that for security experts to point out afterwards.
Apple is trying to downplay these findings, saying that it expected the collisions at least, and that it's system would also do a separate server side hashing comparison which would stop the false collisions. Though, as Bruce Schneier points out, if this was "expected," then why wasn't it discussed in the initial details that were released? Similarly, I have yet to see a response to the flip side issue of changing the images in a way that fool NeuralHash while still looking the same.
I know Apple keeps wanting to insist that it's thought through all of this, but it doesn't seem to have thought through any of how the security community would see all of this, and it's after-the-fact scrambling is not exactly reassuring.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: backdoors, client side scanning, csam, encryption, hash clash, jonathan mayer, surveillance
Companies: apple
Reader Comments
Subscribe: RSS
View by: Time | Thread
They shouldn't avoid our email scanning
From 9to5 Mac
There is more in the article...
[ link to this | view in chronology ]
The big problem is that they have implemented a system, that has root access, and can exfiltrate anything they want from the phone. Scanning on the phone saves Apple CPU power, but is incidental to the fact that they have implemented a system that gives them full access and export for anything on the phone. That is not a backdoor, but rather full government access to phone contents, including keys as they are entered, as soon as a government can compel Apple to comply with their demands.
[ link to this | view in chronology ]
Re:
Quite the contrary. It is a backdoor, just one surrounded with neon lights, and illuminating arrows pointing right to it.
Full government access is one thing, but this goes much further than that. Apple will happily give access to anyone who's willing to pay, so don't be surprised it you're called in front of your Employer's HR because you have something on your iPhone they didn't like.
Don't count on "lawful-intercept" or wire-tapping laws to protect you either. If Apple hasn't already, you can be sure they'll updated iPhone terms and conditions, so they can claim that you have "voluntarily agreed" to this snooping.
[ link to this | view in chronology ]
Re: Re:
It's not a backdoor, Apple have wedged the front door open.
[ link to this | view in chronology ]
Apple says it won't give in to GOVERNMENT demands.
Apple has already set up its own shell company. It plans to "sell" the data to the shell company, which will sell the data to various governments around the world.
Thus keeping "the literal letter of the promise, but actually making a lot of money".
The internal plans specify child porn as "the ice breaker" to get people used to being spied on, then they will start on emails, documents, passwords, browsing history one by one. A slow push back of privacy in the hopes the majority of customers won't notice.
[ link to this | view in chronology ]
Re:
... and if official/government access was all we had to worry about that would be tolerable ... possibly.
That's not all we have to worry about though. A system like this is 100% going to be targeted by adversarial intelligence services, criminal, hackers, etc. One way or another people that have no legitimate or court authorised right to the system will make use of it. Whether it's old fashioned bribery/blackmail, hacking, exploiting software flaws, etc. they will get access to it.
[ link to this | view in chronology ]
Re:
Is this speculative fiction, or a thing that's actually happened? If the latter, where can we read about it?
[ link to this | view in chronology ]
old idea
There was an interesting thought long ago, and a debate that didnt last to long.
If we all could read each others minds, wouldnt it be a great thing?
We could understand each other fully and with meaning.
And know every crook and thief around us.
Learning to control ourselves and our minds would be the first challenge.
We would know WHO needs help.
We could understand and KNow how others feel about Special circumstances, maybe find out how to Adjust them to be abit different. But we would know what created things in their minds.
But those most scared are the ones hiding truth.
[ link to this | view in chronology ]
Hash
I think the folks at Apple may be enjoying a different kind of Hash if they can claim that they actually knew the problems pointed out and still went forward with it.
It's a lot like replacing a bridge with nothing and pointing out the the cars were able to drive off a cliff without any problems and it was fully expected.
They should share that Hash.
[ link to this | view in chronology ]
'You had One Job'
A system for catching certain content that's trivial to trick/bypass, will result in a significant number of false positives and can be easily repurposed to go after any number of other types of content without the user's knowledge with the only thing stopping that being Apple pinky-promising not to do that and/or other groups(government or not) doing so themselves.
Oh yeah, they really thought this one thought before rolling it out.
[ link to this | view in chronology ]
Considering it can be weaponized in many ways other than governments using it for censorship and the "for the children" mantra is often used by people that couldn't care less about the children or civil rights (ie: the Trump horde, dictators around the world etc) this will be very ugly to watch. Thanks Apple. And if I didn't already have enough reasons not to use Apple products now I have one that's probably worth all possible reasons multiplied by a few orders of magnitude.
[ link to this | view in chronology ]
A statement from Tim Cook
A statement from Tim Cook found in a interview with the Australian Financial Review about tech companies and privacy.
"Technology will only work if it has people's trust."
Guess you just through that in the Apple's dust bin.
[ link to this | view in chronology ]
Just another example of why policies don't mean anything if they have your data, or access to it.
[ link to this | view in chronology ]
Re:
This technology is a policy, and as subject to changes and gaming as any other policy -- it's just that this policy in now implemented on the "user's" (we used to call him/her the "owner") own phone.
[ link to this | view in chronology ]
Data center power usage model
Google showed up at conference and presented their whizzy solution. Reduce the transformation of DC to AC to DC, and some other bits.
Some got up in their face because there was an industry wide panel working on just that and they appeared to further down the road than Google.
If you are absolutely convinced you are the best, then checking your results against the work of others will never occur to you.
[ link to this | view in chronology ]
From the company that told you that you were holding their phone wrong rather than admit a massive design failure...
You expect them to admit their idea has any possible flaws?
Something something how ARE humans still alive.
[ link to this | view in chronology ]