Be Careful What You Wish For: Demanding Platforms Delete Disinformation May Make It Harder To Understand What Happened

from the watch-out dept

Here's another one in the "be careful what you wish for" category. Over the last few years, under tons of pressure from politicians and many users, various internet platforms have gotten more and more aggressive in removing content and accounts that were credibly accused of spreading disinformation and propaganda. Most people cheered over this, and you can completely understand why. But, that doesn't mean it doesn't create some consequences that might not all be good. J.A. Guerrero-Saade points out that all of this content removal can make things harder for researchers and investigators.

I'm reminded, somewhat, of all the demands in the past (and present) that these platforms need to take down "terrorist" content as quickly as possible. That ignores the fact that "terrorist" content can actually also be evidence of war crimes and atrocities that it might be useful for certain people to see. It also makes it that much more difficult for investigators -- whether government actors or open source investigative reporters -- to track down the perpetrators.

In response to this thread, Facebook's former Chief Security Officer, Alex Stamos, warned that he didn't think the research Facebook conducted over the past few years to find Russian disinfo spreaders would even be allowable under the GDPR:

This, as always, is the kind of thing we've been concerned about from the very beginning with the GDPR. There may be the best of intentions behind it, and "protecting privacy" always sounds good. But there are significant consequences to this, and demanding that private data be locked up, or that "bad" content be deleted as fast as possible, can have significant real world consequences that we might not like very much over the long haul.

Filed Under: alex stamos, deleting information, disinformation, gdpr, historical record, privacy, propaganda

The Scale Of Moderating Facebook: It Turns Off 1 Million Accounts Every Single Day

from the not-an-easy-issue dept

For years now, we've discussed why it's problematic that people are demanding internet platforms moderate more and more speech. We should be quite wary of internet platforms taking on the role of the internet's police. First, they're really bad at it. As we noted in a recent post, platforms are horrendously bad at distinguishing abusive content from those documenting abusive content and that creates all sorts of unfortunate and bizarre results, with those targeted by harassing content often having their own accounts shut down. On top of that, the only way to actually moderate content at scale requires a set of rules, and any such set of rules, as applied, will create hysterically bad results. And that's because the scale of the problem is so massive. It's difficult for most people to comprehend even slightly the scale involved here. As a former Facebook employee who worked on this stuff once told me, "Facebook needs to make one million decisions each day -- one million today, one million tomorrow, one million the next day." The idea that they won't make errors (both of the Type 1 and Type 2 category) is laughable.

And it appears that the scale is only growing. Facebook has now admitted that it shuts off 1 million accounts every single day -- which means that earlier number I heard is way low. If it's killing one million accounts every day, that means it's making decisions on way more accounts than that. And, the company knows that it gets things wrong:

Still, the sheer number of interactions among its 2 billion global users means it can't catch all "threat actors," and it sometimes removes text posts and videos that it later finds didn't break Facebook rules, says Alex Stamos.

"When you're dealing with millions and millions of interactions, you can't create these rules and enforce them without (getting some) false positives," Stamos said during an onstage discussion at an event in San Francisco on Wednesday evening.

That should be obvious, but too many people think that the answer is to just put even more pressure on Facebook -- often through laws requiring it to moderate content, takedown content and kill accounts. And, when you do that, you actually make the false positive problem that much worse. Assuming, for the sake of argument, that Facebook has to kill 10% of all the accounts it reviews, that's 10 million accounts every day. If the punishment for taking down content that should have been left up is public shame/ridicule, that acts as at least some defense to get Facebook to be somewhat careful about not taking down stuff that it shouldn't. But, on the flip side, if you add a law (such as the new one in Germany) that puts criminal penalties on social media companies for leaving up content that it wants taken down, you've changed the equation.

Now, the choice isn't between "public ridicule vs. bad person on our platform" it's "public ridicule v. criminal charges and massive fines." So the incentive for Facebook, and other platforms changes such that it's now encouraged to kill a hell of a lot more accounts, just in case. So suddenly the number of "false positives" is going to sky rocket. That's not a very good solution -- especially if you want platforms to support free speech. Again, platforms have every right to moderate content on their platforms, but we should be greatly concerned when governments are forcing them to moderate in a way that may have widespread consequences on how people speak, and where those policies can tilt the scales in often dangerous ways.

Filed Under: alex stamos, choices, intermediary liability, moderation, scale
Companies: facebook

Yahoo Secretly Built Software To Scan All Emails Under Pressure From NSA Or FBI

from the uh-wait-a-second dept

So Reuters had a big exclusive report this morning about Yahoo creating "custom software to search all of its customers' incoming emails for specific information" at the behest of the NSA or FBI. This was built last year -- which came well after the Snowden disclosures, and after Yahoo had been revealed to have legally challenged earlier NSA dragnet attempts -- and after it had rolled out end-to-end encryption on email.

Apparently, this was a decision made at the top by Marissa Mayer, and pissed off the company's top security guy, Alex Stamos (who is awesome and a big supporter of end-to-end encryption) leading him to leave the company (and move to Facebook, where he is currently).

According to the two former employees, Yahoo Chief Executive Marissa Mayer's decision to obey the directive roiled some senior executives and led to the June 2015 departure of Chief Information Security Officer Alex Stamos, who now holds the top security job at Facebook Inc."Yahoo is a law abiding company, and complies with the laws of the United States," the company said in a brief statement in response to Reuters questions about the demand. Yahoo declined any further comment.

Of course, this comes out less than a week after the NY Times had a big report on how Mayer de-prioritized security, despite having built up a great team of computer security experts called "The Paranoids." But, Mayer apparently downplayed or blocked their efforts, leading many to go elsewhere. And now we find out that Yahoo agreed to create this special software for scanning all emails for certain phrases or keywords. Bizarrely, this new report notes that Mayer gave the task of writing this software not to the security team, but to email engineers, leaving the security team in the dark, until they discovered it, thinking it was malware:

They were also upset that Mayer and Yahoo General Counsel Ron Bell did not involve the company's security team in the process, instead asking Yahoo's email engineers to write a program to siphon off messages containing the character string the spies sought and store them for remote retrieval, according to the sources.

The sources said the program was discovered by Yahoo's security team in May 2015, within weeks of its installation. The security team initially thought hackers had broken in.

When Stamos found out that Mayer had authorized the program, he resigned as chief information security officer and told his subordinates that he had been left out of a decision that hurt users' security, the sources said. Due to a programming flaw, he told them hackers could have accessed the stored emails.

Now, there are still a number of open questions about this: chief among them if others, such as Google, Microsoft, Facebook, and Twitter were similarly compelled to create similar software. This may not be that meaningful, but the article does not say that it was a FISA Court "order" but rather a "directive" that compelled this:

The company complied with a classified U.S. government directive, scanning hundreds of millions of Yahoo Mail accounts at the behest of the National Security Agency or FBI, said two former employees and a third person apprised of the events.

The question then is what secret "directive" does the government have that allows such broad scanning? The most likely (but certainly not the only) possibility is a stretched interpretation of Section 702 of the FISA Amendments Act. That Section is responsible for two known programs for the NSA to collect info: PRISM, which had big tech companies sharing specific information with the NSA, and "upstream" collection in which broadband providers like AT&T would scan all traffic for certain information. Without more detail, it's a little difficult to know what happened here, but it sounds like something in between PRISM and upstream -- in which online service providers were similarly asked to scan all content for certain information.

It seems clear that Yahoo either didn't think it could win a legal fight over this (certainly a possibility), or that it just didn't want to. At the very least, this seems like yet another example of totally secretive rulemaking by the US government on what surveillance capabilities are legal, without any public review or adversarial process designed to make sure that civil liberties are protected. I know that many of the more paranoid folks out there think that the NSA already had deals with the big companies to scan all content, but they weren't supposed to, and as far as we knew they did not as of a few years ago. But if that changed last year, that's a big, big deal, and much more information needs to become public on this.

Filed Under: alex stamos, bulk collection, email, marissa mayer, mass surveillance, nsa, privacy, scanning, section 702
Companies: yahoo

NSA Director: If I Say 'Legal Framework' Enough, Will It Convince You Security People To Shut Up About Our Plan To Backdoor Encryption?

from the wanna-try-that-again dept

Admiral Mike Rogers, the NSA Director, has barely been on the job for a year, and so far he'd mostly avoided making the same kinds of absolutely ridiculous statements that his predecessor General Keith Alexander was known for. Rogers had, at the very least, appeared slightly more thoughtful in his discussions about the surveillance state and his own role in it. However, Rogers ran into a bit of trouble at New America's big cybersecurity event on Monday -- in that there were actual cybersecurity folks in the audience and they weren't accepting any of Rogers' bullshit answers. The most notable exchange was clearly between Rogers and Alex Stamos, Yahoo's chief security officer, and a well known privacy/cybersecurity advocate.

Alex Stamos (AS): “Thank you, Admiral. My name is Alex Stamos, I’m the CISO for Yahoo!. … So it sounds like you agree with Director Comey that we should be building defects into the encryption in our products so that the US government can decrypt…

Mike Rogers (MR): That would be your characterization. [laughing]

AS: No, I think Bruce Schneier and Ed Felton and all of the best public cryptographers in the world would agree that you can’t really build backdoors in crypto. That it’s like drilling a hole in the windshield.

MR: I’ve got a lot of world-class cryptographers at the National Security Agency.

AS: I’ve talked to some of those folks and some of them agree too, but…

MR: Oh, we agree that we don’t accept each others’ premise. [laughing]

AS: We’ll agree to disagree on that. So, if we’re going to build defects/backdoors or golden master keys for the US government, do you believe we should do so — we have about 1.3 billion users around the world — should we do for the Chinese government, the Russian government, the Saudi Arabian government, the Israeli government, the French government? Which of those countries should we give backdoors to?

MR: So, I’m not gonna… I mean, the way you framed the question isn’t designed to elicit a response.

AS: Well, do you believe we should build backdoors for other countries?

MR: My position is — hey look, I think that we’re lying that this isn’t technically feasible. Now, it needs to be done within a framework. I’m the first to acknowledge that. You don’t want the FBI and you don’t want the NSA unilaterally deciding, so, what are we going to access and what are we not going to access? That shouldn’t be for us. I just believe that this is achievable. We’ll have to work our way through it. And I’m the first to acknowledge there are international implications. I think we can work our way through this.

AS: So you do believe then, that we should build those for other countries if they pass laws?

MR: I think we can work our way through this.

AS: I’m sure the Chinese and Russians are going to have the same opinion.

MR: I said I think we can work through this.

AS: Okay, nice to meet you. Thanks.

[laughter]

MR: Thank you for asking the question. I mean, there are going to be some areas where we’re going to have different perspectives. That doesn’t bother me at all. One of the reasons why, quite frankly, I believe in doing things like this is that when I do that, I say, “Look, there are no restrictions on questions. You can ask me anything.” Because we have got to be willing as a nation to have a dialogue. This simplistic characterization of one-side-is-good and one-side-is-bad is a terrible place for us to be as a nation. We have got to come to grips with some really hard, fundamental questions. I’m watching risk and threat do this, while trust has done that. No matter what your view on the issue is, or issues, my only counter would be that that’s a terrible place for us to be as a country. We’ve got to figure out how we’re going to change that.

[Moderator Jim Sciutto]: For the less technologically knowledgeable, which would describe only me in this room today, just so we’re clear: You’re saying it’s your position that in encryption programs, there should be a backdoor to allow, within a legal framework approved by the Congress or some civilian body, the ability to go in a backdoor?

MR: So “backdoor” is not the context I would use. When I hear the phrase “backdoor,” I think, “well, this is kind of shady. Why would you want to go in the backdoor? It would be very public.” Again, my view is: We can create a legal framework for how we do this. It isn’t something we have to hide, per se. You don’t want us unilaterally making that decision, but I think we can do this.

As you read it, you realize that Rogers keeps thinking that if he says "legal framework" enough times, he can pretend he's not really talking about undermining encryption entirely. Well known cybersecurity guy Bruce Schneier pushed back, pointing out that:

It’s not the legal framework that’s hard, it’s the technical framework. That’s why it’s all or nothing.

No matter what anyone said, however, Rogers appears to keep going back to the "legal framework" well, over and over again, as if that magic phrase would change magical thinking into reality:

“If these are the paths that criminals, foreign actors, terrorist are going to use to communicate, how do we access that?” he asked, citing the need for a “formalized process” to break through encrypted technology.

Rogers pointed toward cooperation between tech companies and law enforcement to combat child pornography. “We have shown in other areas that through both technology, a legal framework, and social compact that we have been able to take on tough issues. I think we can do the same thing here.”

Yes, but that's very different, even as anyone looking to rip apart important privacy and free speech tools loves to shout "child porn," the examples are not even remotely comparable. And no one's looking to backdoor everything just to get at people passing around child porn. But the larger point stands. Rogers seems to think that there is a magic bullet/golden key that will magically only let the good guys through if only the tech industry is willing to work with him on this.

“You don’t want the FBI and you don’t want the NSA unilaterally deciding what” is permissible, Mr. Rogers said.

Except that presumes that if only the surveillance community and the tech industry got together they could come up with such a safe system, and as everyone else is telling him, that's impossible. And for a guy who is supposed to be running an agency that understand cryptography better than anyone else, that's really troubling:

Filed Under: admiral mike rogers, alex stamos, bruce schneier, encryption, fbi, legal framework, mobile encryption, nsa, privacy, security