Researchers Who Built Similar System Explain Why Apple's CSAM Scanning System Is Dangerous

from the it's-not-good dept

Jonathan Mayer, a Princeton University professor and former chief technologist at the FTC, is one of the smartest people I know. Every time I've spoken with him I feel like I learn something. He's now written a quite interesting article for the Washington Post noting how he, and a graduate researcher at Princeton, Anunay Kulshrestha, actually built a CSAM scanning system similar to the one that Apple recently announced, which has security experts up in arms over the risks inherent to the approach.

Mayer and Kulshretha note that while Apple is saying that people worried about their system are misunderstanding it, they are not. They know what they're talking about -- and they still say the system is dangerous.

We wrote the only peer-reviewed publication on how to build a system like Apple’s — and we concluded the technology was dangerous. We’re not concerned because we misunderstand how Apple’s system works. The problem is, we understand exactly how it works.

Mayer and Kulshretha are certainly not making light of the challenges and problems associated with stopping CSAM. It's why they started their project to see if there was an effective way to identify CSAM even in end-to-end encrypted systems. And they even built a system to do just that. And what they found is that the risks are simply too great.

We were so disturbed that we took a step we hadn’t seen before in computer science literature: We warned against our own system design, urging further research on how to mitigate the serious downsides. We’d planned to discuss paths forward at an academic conference this month.

That dialogue never happened. The week before our presentation, Apple announced it would deploy its nearly identical system on iCloud Photos, which exists on more than 1.5 billion devices. Apple’s motivation, like ours, was to protect children. And its system was technically more efficient and capable than ours. But we were baffled to see that Apple had few answers for the hard questions we’d surfaced.

The potential dangers that Mayer and Kulshretha are exactly what many had warned about when Apple announced its plans:

After many false starts, we built a working prototype. But we encountered a glaring problem.

Our system could be easily repurposed for surveillance and censorship. The design wasn’t restricted to a specific category of content; a service could simply swap in any content-matching database, and the person using that service would be none the wiser.

A foreign government could, for example, compel a service to out people sharing disfavored political speech. That’s no hypothetical: WeChat, the popular Chinese messaging app, already uses content matching to identify dissident material. India enacted rules this year that could require pre-screening content critical of government policy. Russia recently fined Google, Facebook and Twitter for not removing pro-democracy protest materials.

We spotted other shortcomings. The content-matching process could have false positives, and malicious users could game the system to subject innocent users to scrutiny.

That's already pretty damning. But there's some other even more damning information that has come out as well. As we noted in our earlier posts (and as Mayer and Kulshretha noted in their research), the risk of false positives is extremely high. And late last week, the hypothetical became a lot more real. Someone reverse engineered the NeuralHash algorithm that Apple is using and put it on Github.

It did not take long for people to point that, first of all, there are collisions with totally unrelated images:

Now, you might say that since that second image is not really an image, maybe it's not that big a deal? Well, about that...

And, of course, one of the issues with any hash-based system is the idea that subtle changes to the image would output a totally different hash, thus making it easy for some to simply route around the scanning. Apple had suggested its system could defeat that, but...

This is part of the reason that we highlighted earlier that security researchers are so up in arms about this. Apple seemingly ignored so much of the research and conversations that were happening about these approaches, and just barged right in announcing that it had a solution without exploring the tradeoffs and difficulties associated with it -- leaving that for security experts to point out afterwards.

Apple is trying to downplay these findings, saying that it expected the collisions at least, and that it's system would also do a separate server side hashing comparison which would stop the false collisions. Though, as Bruce Schneier points out, if this was "expected," then why wasn't it discussed in the initial details that were released? Similarly, I have yet to see a response to the flip side issue of changing the images in a way that fool NeuralHash while still looking the same.

I know Apple keeps wanting to insist that it's thought through all of this, but it doesn't seem to have thought through any of how the security community would see all of this, and it's after-the-fact scrambling is not exactly reassuring.

Filed Under: backdoors, client side scanning, csam, encryption, hash clash, jonathan mayer, surveillance
Companies: apple

The DOJ Is Conflating The Content Moderation Debate With The Encryption Debate: Don't Let Them

from the it's-not-the-same dept

As we've detailed a lot over the last week, the DOJ has decided that after years of failing to get backdoors mandated by warning about the "terrorism" bogeyman, it's decided to pick up the FOSTA playbook, and instead start focusing on child porn -- or what "serious people" now refer to as Child Sexual Abuse Material (CSAM). It did this last week with an assist from the NY Times, who published an article with (legitimately) scary stories, but somehow blaming the internet companies... because they actually report it when they find such content on their networks. I've seen more than a few people, even those who generally have been strong voices on the encryption debate and against backdoors, waver a bit on this particular subject, and note that maybe there shouldn't be encryption on social media networks, because it might (as the narrative says) help awful people hide their child porn.

Except... that's confusing a few different things. Mainly, it's mixing up the content moderation debate with the "lawful access" or "backdoors" debate. Yes, encryption makes it harder for the police to get in and see certain things, but that's by design. We live in a country with the 4th Amendment, in which we believe that it should be difficult for law enforcement to snoop deeply into our lives -- and that's always meant that some people will do and plot bad stuff out of the sight and hearing of law enforcement. Yet, if you were to look at law enforcement over the past 100 years, you can bet that they have many times more access to information about people today than they have in the past. The claim of "going dark" is laughable when you compare the information that law enforcement can get today even to what it could get 15 or 30 years ago.

But, importantly, bringing CSAM into the debate muddies the water by pretending -- incorrectly -- that in an end-to-end encrypted world you can't do any content moderation, and there's simply no way for platforms to block or report certain kinds of content. Yet, as Princeton professor Jonathan Mayer highlights in a new paper, content moderation is not impossible in an encrypted system. It may be different than it is today, but it's still very much possible:

Much of the public discussion about content moderation and end-to-end encryption over the past week has appeared to reflect two common technical assumptions:

1. Content moderation is fundamentally incompatible with end-to-end encrypted messaging.
   
2. Enabling content moderation for end-to-end encrypted messaging fundamentally poses the same challenges as enabling law enforcement access to message content.

In a new discussion paper, I provide a technical clarification for each of these points.

1. Forms of content moderation may be compatible with end-to-end encrypted messaging, without compromising important security principles or undermining policy values.
   
2. Enabling content moderation for end-to-end encrypted messaging is a different problem from enabling law enforcement access to message content. The problems involve different technical properties, different spaces of possible designs, and different information security and public policy implications.

You can read the whole thing, but as the paper notes, user reporting of such content still works in an end-to-end encrypted world, as does hash matching if done at the client end. There's a lot more in there as well, but what you realize in reading the paper is that while law enforcement has now latched onto the CSAM issue as its hook to break encryption (in part, I've been told by someone working with the DOJ, because they found it "polled well"), it's an entirely different problem. This is yet another "but think of the children" argument, which ignores the technical and societal realities.

Filed Under: chris wray, content moderation, doj, encryption, going dark, jonathan mayer, william barr
Companies: facebook

How The NSA's 'Cybersecurity' Surveillance Should Completely Change The Debate On Cybersecurity Bills

from the they're-about-surveillance dept

For quite some time now, we've been warning about the government's questionable attempts to pass "cybersecurity" bills that focus on "information sharing" with names like CISA and CISPA. Defenders of these bills insist that they're "just voluntary" and are necessary because it would enable private companies to share threat information with the US government, so that the US government could help stop attacks. Of course, we've been asking for years (1) why, if this is so useful, companies can't already share this information and (2) what attacks these bills would have actually stopped? No one ever seems to have any answers.

Defenders of the bill also insist that there really shouldn't be any privacy concerns because companies can just hand over the limited information on the attacks, not any personal user info. However, with the recent revelations from Pro Publica and the NY Times (via Snowden documents) about how the NSA uses "cyber signatures" in sniffing through the upstream collection (i.e., sniffing through all internet traffic by tapping into fiber backbones) computer security expert Jonathan Mayer notes that this completely changes the equation on just how bad these "information sharing" cybersecurity bills really are.

Before it was known that the NSA could do this, the argument was that sharing details of a cybersecurity threat would just lead to DHS and NSA taking that "threat" information, and then seeing if it can help figure out ways to prevent the threat. But, now that we know the NSA can sniff the entire upstream collection using such "cyber signatures" and then is allowed to collect and keep whatever it finds as an incidental collection, this becomes very clearly a surveillance bill -- just as Senator Ron Wyden warned.

That's because the new documents make it clear that the NSA not only wants to search based on these broad "cyber signatures" but then claims it gets to keep that data and can search through whatever it collects. These are the infamous "backdoor searches" that Senator Wyden has been warning about for ages.

So, these "information sharing" bills don't just give the NSA access to private information from companies, but really give the NSA the "cyber signatures" it needs to then snarf up a ton of other private information that it has long wanted access to. This is why closing the "backdoor search" loophole is so important as well -- and not letting any of these "information sharing" bills pass is also of utmost importance.

Oh, and one other sneaky thing in all of this that Mayer highlights: defenders of these information sharing bills insist that they're not surveillance bills because, as Rep. Adam Schiff noted: "this bill makes clear in black and white legislative text that nothing authorizes government surveillance in this act." But, as Mayer points out that's incredibly misleading because the government already has the authorization it needs, under the secret program that was just revealed. What the information sharing does is make that authorization much more powerful by making it easier for the NSA to collect the information it then can slide into the program in order to snarf up much more important private information.

Filed Under: cisa, cispa, cybersecurity, information sharing, jonathan mayer, nsa, surveillance

Encryption: What The FBI Wants It Can Only Have By Destroying Computing And Censoring The Internet

from the and-it-doesn't-seem-to-understand-this dept

The FBI -- and by extension, every law enforcement agency it partners with -- wants holes carved in cellphone encryption. The problem is that it doesn't even know what specifically it wants.

When asked directly if the FBI wants a backdoor, [Amy] Hess [Asst. Director of FBI's Science & Technology branch] dodged the question and did not describe in detail what actual solution the FBI is seeking.

“We are simply asking for information that we seek in response to a lawful order in a readable format,” Hess responded, while also repeating that the Bureau supports strong encryption. “But how that actually happens should be the decision of the provider.”

When pressed again, Hess said that it would be okay for the FBI not to have a key to decrypt data, if the provider “can get us that information by maintaining the key themselves.”

That's asking the impossible -- for a great many reasons. First and foremost, compromised encryption is compromised encryption. It can be exploited by criminals and other unwanted entities just as certainly as it can assist law enforcement agencies in obtaining the information they're seeking. There's no way around this fact. You cannot have "good guys only" encryption.

But beyond that, even if the FBI manages to get what it wants, it will do so at the expense of general computing. Requiring built-in backdoors or key escrow will dismantle the very systems it's meant to access. Computer scientist Jonathan Mayer delivers a long, detailed hypothetical involving the Android platform and how the FBI's desired access would fail -- and do severe collateral damage -- every step of the way. (via Boing Boing)

First off, if Google gives the FBI the backdoors it wants, that only nails down Google. But Google also distributes thousands of third-party apps through its Play store. And these apps may not contain the subverted encryption the FBI is looking for. Now, Google has to be in the business of regulating third-party apps to ensure they meet the government's standard for compromised encryption.

The obvious answer is that Google can’t stop with just backdooring disk encryption. It has to backdoor the entire Android cryptography library. Whenever a third-party app generates an encrypted blob of data, for any purpose, that blob has to include a backdoor.

This move may work, but it only affects apps using Google's encryption. Other offerings may rely on other encryption methods. Then what? It has a few options, all of them carrying horrendous implications.

One option: require Google to police its app store for strong cryptography. Another option: mandate a notice-and-takedown system, where the government is responsible for spotting secure apps, and Google has a grace period to remove them. Either alternative would, of course, be entirely unacceptable to the technology sector—the DMCA’s notice-and-takedown system is widely reviled, and present federal law (CDA 230) disfavors intermediary liability.

At this point, Mayer suggests the "solution" is already outside the realm of political feasibility. Would the FBI really push this far to obtain encryption backdoors? The FBI itself seems unsure of how far it's willing to go, and many officials quoted (like the one above) seem to think all the FBI really needs to do is be very insistent on this point, and techies will come up with some magical computing solution that maintains the protective qualities of encryption while simultaneously allowing the government to open the door and have a look around any time it wants to.

So, if the FBI is willing to travel this very dark road littered with an untold amount of collateral damage, it still hasn't managed to ensure the phones it encounters will open at its command. Considering phone users could still acquire apps from other sources, the government's reach would only extend as far as the heavily-policed official app store (and other large competitors' app stores). Now what? More government power and less operational stability.

The only solution is an app kill switch. (Google’s euphemism is “Remote Application Removal.”) Whenever the government discovers a strong encryption app, it would compel Google to nuke the app from Android phones worldwide. That level of government intrusion—reaching into personal devices to remove security software—certainly would not be well received. It raises serious Fourth Amendment issues, since it could be construed as a search of the device or a seizure of device functionality and app data. What’s more, the collateral damage would be extensive; innocent users of the app would lose their data.

Even if the government were willing to take it this far, it still doesn't eradicate apps that it can't crack. (But it may be sufficient to only backdoor the most used apps, which may be all it's looking to achieve...) App creators could decide to avoid Google's government-walled garden and mandated kill switch by assigning random identifiers and handling a majority of the app's services (like a messaging service, etc.) via a website, out of reach of app removal tools and government intervention. To stop this, the US government would need to do the previously unimaginable:

In order to prevent secure data storage and end-to-end secure messaging, the government would have to block these web apps. The United States would have to engage in Internet censorship.

Robert Graham at Errata Security makes similar points in his post on the subject, but raises a couple of other interesting (in the horrific train wreck meaning of the word) points. While the government may try to regulate the internet, it can't (theoretically) touch services hosted in foreign countries. (Although it may soon be able to hack away at them with zero legal repercussions…)

Such services could be located in another country, because there are no real national borders in cyberspace. In any event, such services aren't "phone" services, but instead just "contact" services. They let people find each other, but they don't control the phone call. It's possible to bypass such services anyway, by either using a peer-to-peer contact system, or overloading something completely different, like DNS.

Like crypto, the entire Internet is based on the concept of end-to-end, where there is nothing special inside the network that provides a service you can regulate.

The FBI likely has no desire to take its fight against encryption this far. The problem is that it thinks its "solution" to encryption is "reasonable." But it isn't.

The point is this. Forcing Apple to insert a "Golden Key" into the iPhone looks reasonable, but the truth is the problem explodes to something far outside of any sort of reasonableness. It would mean outlawing certain kinds of code -- which is probably not possible in our legal system.

The biggest problem here is that no one arguing for "golden keys," key escrow, "good guy" backdoors, etc. seems to have any idea what implementing this could actually result in. They think it's just tech companies sticking it to The Man, possibly because a former NSA sysadmin went halfway around the world with a pile of documents and a suitcase of whistles with "BLOW ME" printed on the side.

But it isn't. And their continual shrugged assertion that the "smart guys" at tech companies will figure this all out for them is not only lazy, it's colossally ignorant. There isn't a solution. The government can't demand that companies not provide encryption. It's not willing to ban encryption, nor is it in any position to make that ban stick. It doesn't know what it needs. It only knows what it wants. And it can't have what it wants -- not because no one wants to give it to them -- but because no one can give it to them.

Yes, many tech companies are far more wary of collaborating with the government in this post-Snowden era, but in this case, the tech world cannot give the FBI what it wants without destroying nearly everything surrounding the "back door." And continually trotting out kidnappers, child porn enthusiasts and upskirt photographers as reasons for breaking cell phone platforms doesn't change the fact that it cannot be done without potentially harming every non-criminal phone owner and the services they use.

Filed Under: backdoors, computer security, cybersecurity, encryption, fbi, jonathan mayer, security