Twitter Admits It Messed Up In Suspending Accounts Under Its New Policy, But Policies Like This Will ALWAYS Lead To Overblocking

from the who-didn't-see-that-coming dept

Last week, we called out a questionable move by Twitter to put in place a new policy that banned images or videos that included someone without their permission. The company claimed that this was to prevent harassment and also (for reasons I still don't understand) buried the fact that this was not supposed to apply to public figures or newsworthy media. However, as we pointed out at the time, the standard here was incredibly subjective, and wide open to abuse. Indeed, we pointed out examples in that article of the policy clearly being abused. And even as Fox News talking heads insisted the policy would only be used against conservatives, in actuality, a bunch of alt right/white nationalists/white supremacists immediately saw this as the perfect way to get back at activists who had been calling out their behavior, leading to a mass brigading effort to "report" those activists for taking pictures and videos at white nationalist rallies and events.

In other words, exactly what we and tons of others expected.

And, a few days later, Twitter admitted it messed up the implementation of the policy -- though it doesn't appear to be rolling back the policy itself.

Twitter said Friday that it had mistakenly suspended accounts under a new policy following a flood of “coordinated and malicious reports” targeting anti-extremism researchers and journalists first reported Thursday by The Washington Post.

The company said it had corrected the errors and launched an internal review to ensure that its new rule — which allows someone whose photo or video was tweeted without their consent to request its removal — was “used as intended.”

[....]

In a statement Friday, Twitter spokesman Trenton Kennedy said that the company had been overwhelmed with a “significant amount” of malicious reports and that its “enforcement teams made several errors” in the aftermath.

What I am perplexed about, however, is that I know for a fact that Twitter has a bunch of smart and thoughtful people who work on trust and safety issues and who clearly would know how this policy would play out in practice, and yet for whatever reason the policy was still rolled out the way it was. I don't know if this is because people underestimated how quickly and wildly it would be abused, or if others at the company simply overruled concerns raised by experts, or if it was something else entirely. Either way... it's a weird and surprising misstep for Twitter.

That said, it's also illustrative of a really important point that we've been trying to raise for ages, going back to the DMCA takedown process, and covering all sorts of other policy debates regarding content moderation: if you give people tools to take down some content, they will be abused. Always. That's not to say you should never moderate or never take down content, because that's impossible. There is always going to be some content that sites need to take down, whether for legal purposes or because it's harming the integrity of the site (things like spam or harassment).

But any such policy always opens itself up to abuse and dishonest reporting. And any company that has such policies (i.e., every company with third party content) needs to have a plan in place not just to deal with the abusive/problematic content on the site, but also the abuse of the moderation process to silence voices that shouldn't actually be silenced.

Again, from my interactions with Twitter trust & safety people, I know they know this. And this is part of why I find the rollout of this policy so perplexing.

However, it's also an important lesson for policymakers in various state legislatures, in DC, and around the globe. There is so much effort these days to pressure (or require!) internet companies to remove "bad" content, but almost none of those policy plans take into account the ways in which those policies will be abused to silence reporting, silence marginalized voices, and silence those calling out abuses of power. Twitter's rollout of this policy has been a disaster (and one that could have been prevented) but at the very least it should be a warning to policymakers who seem to think that they can design policy requirements to moderate certain content, without bothering to explore the likelihood that those mandates will be abused to silence important speech.

Filed Under: content moderation, overblocking, photos, private information, private media, videos
Companies: twitter

Twitter's New 'Private Information' Policy Takes Impossible Content Moderation Challenges To New, Ridiculous Levels

from the the-end-of-everything dept

I've been working on a post about a whole bunch of interesting (and good!) things that Twitter has been doing over the last few months. I'm not sure when that post will be ready, but I need to interrupt that process to note a truly head-scratching change in Twitter's moderation policy announced this week. Specifically, Twitter announced that its "private information policy" has now been expanded to include "media." Specifically, Twitter says that it will remove photos and videos that are posted "without the permission of the person(s) depicted." In other words, Twitter has taken the old, meme-ified, "I'm in this photo and I don't like it" into official policy for taking down content.

Buried deeper in the rules is a very subjective conditional:

This policy is not applicable to media featuring public figures or individuals when media and accompanying text are shared in the public interest or add value to public discourse.

But that's going to lead to some very, very big judgment calls about (a) who is a "public figure" and (b) what is "in the public interest." And early examples suggest that Twitter's Trust & Safety team are failing this test.

I can understand the high level, first pass thinking that leads to this policy: there are situations in which photos or videos that are taken surreptitiously and then are used to mock or harass someone can certainly raise questions and there are perfectly reasonable policy choices to be made on how to handle those scenarios. But how do you distinguish those rare circumstances with a much wider series of cases where people may not have given permission to be in photos or videos, but keeping that content online is clearly beneficial. This can range from the obvious incidental background images of people walking by in a crowded place to -- much more concerning -- situations of journalism being done by individuals, recording important events. Twitter's insistence that it won't apply to "public interest" issues is hardly reassuring. We've seen those claims before and they rarely work in practice.

The most obvious example of this is one of the biggest stories of 2020: the video taping of police officer Derek Chauvin kneeling on George Floyd's neck until he died. In theory, under the broadness of this policy, that video would be taken down off of Twitter. There are lots of other situations as well, including things like Amy Cooper, who was filmed in Central Park calling the police on Christian Cooper (no relation to Amy) who was in the park bird-watching. There are plenty of other examples where people are filmed in public, without their permission, but it's done to reveal important things that have happened in the world. For example, law enforcement relied on help from social media to help identify people who stormed the Capital on January 6th. It seems that under this new policy, all those photos of January 6th insurrectionists could be removed from Twitter. Is sharing them in the public interest? Depends on who you ask, I imagine...

For years we've seen tons of people abusing other systems to take down content they didn't like. For example, there was the part owner of the Miami Heat who literally sued over an unflattering photo by first obtaining the copyright for it. Or the revenge porn extortionist who tried to force stories about him offline with copyright notices. In Europe, we've seen something similar with abuses of the "right to be forgotten" to memory hole news stories.

And here, Twitter is setting up to just take down any such photo or video upon request? This seems wide open for massive levels of abuse. Indeed, there are already a number of reports about the policy being used to silence activists and researchers:

Yes, even some of the examples above may be considered edge cases with more nuance than is presented by the people posting them, but as we've seen with copyright and the right to be forgotten, give people a tool to get any information removed from social media, and it will be massively and widely abused to try to hide bad behavior.

I'm honestly perplexed at why Twitter implemented such a broad policy, so difficult to enforce, and so open to abuse. It seems extremely unlike the more thoughtful trust & safety moves the company has made over the past few years.

Filed Under: content moderation, media, newsworthy, photos, private information, public figure, trust & safety, videos
Companies: twitter

Journalists In St. Louis Discover State Agency Is Revealing Teacher Social Security Numbers; Governors Vows To Prosecute Journalists As Hackers

from the wtf-missouri? dept

Last Friday, Missouri's Chief Information Security Officer Stephen Meyer stepped down after 21 years working for the state to go into the private sector. His timing is noteworthy because it seems like Missouri really could use someone in their government who understands basic cybersecurity right now.

We've seen plenty of stupid stories over the years about people who alert authorities to security vulnerabilities then being threatened for hacking, but this story may be the most ridiculous one we've seen. Journalists for the St. Louis Post-Dispatch discovered a pretty embarrassing leak of private information for teachers and school administrators. The state's Department of Elementary and Secondary Education (DESE) website included a flaw that allowed the journalists to find social security numbers of the teachers and administrators:

Though no private information was clearly visible nor searchable on any of the web pages, the newspaper found that teachers’ Social Security numbers were contained in the HTML source code of the pages involved.

The newspaper asked Shaji Khan, a cybersecurity professor at the University of Missouri-St. Louis, to confirm the findings. He called the vulnerability “a serious flaw.”

“We have known about this type of flaw for at least 10-12 years, if not more,” Khan wrote in an email. “The fact that this type of vulnerability is still present in the DESE web application is mind boggling!”

In the HTML source code means that it sent that information to the computers/browsers of those who knew what pages to go to. It also appears that the journalists used proper disclosure procedures, alerting the state and waiting until it had been patched before publishing their article:

The Post-Dispatch discovered the vulnerability in a web application that allowed the public to search teacher certifications and credentials. The department removed the affected pages from its website Tuesday after being notified of the problem by the Post-Dispatch.

Based on state pay records and other data, more than 100,000 Social Security numbers were vulnerable.

The newspaper delayed publishing this report to give the department time to take steps to protect teachers’ private information, and to allow the state to ensure no other agencies’ web applications contained similar vulnerabilities.

Also, it appears that the problems here go back a long ways, and the state should have been well aware that this problem existed:

The state auditor’s office has previously sounded warning bells about education-related data collection practices, with audits of DESE in 2015 and of school districts in 2016.

The 2015 audit found that DESE was unnecessarily storing students’ Social Security numbers and other personally identifiable information in its Missouri Student Information System. The audit urged the department to stop that practice and to create a comprehensive policy for responding to data breaches, among other recommendations. The department complied, but clearly at least one other system contained an undetected vulnerability.

This is where a competent and responsible government would thank the journalists for finding the vulnerability and disclosing it in an ethical manner designed to protect the info of the people the state failed to properly protect.

But that's not what happened.

Instead, first the Education Commissioner tried to make viewing the HTML source code nefarious:

In the letter to teachers, Education Commissioner Margie Vandeven said “an individual took the records of at least three educators, unencrypted the source code from the webpage, and viewed the social security number (SSN) of those specific educators.”

It was never "encrypted," Commissioner, if the journalists could simply look at the source code and get the info.

Then DESE took it up a notch and referred to the journalists as "hackers."

But in the press release, DESE called the person who discovered the vulnerability a “hacker” and said that individual “took the records of at least three educators” — instead of acknowledging that more than 100,000 numbers had been at risk, and that they had been available to anyone through DESE’s own search engine.

And then, it got even worse. Missouri Governor Mike Parson called a press conference in which he again called the journalists hackers and said he had notified prosecutors and the Highway Patrol's Digital Forensic Unit to investigate. Highway Patrol? He also claimed (again) that they had "decoded the HTML source code." That's... not difficult. It's called "view source" and it's built into every damn browser, Governor. It's not hacking. It's not unauthorized.

It gets worse. Governor Parson claims that this "hack" could cost $50 million. I only wish I was joking.

This incident alone may cost Missouri taxpayers up to $50 million and divert workers and resources from other state agencies. This matter is serious.

The state is committing to bring to justice anyone who hacked our system and anyone who aided or encouraged them — In accordance with what Missouri law allows AND requires.

A hacker is someone who gains unauthorized access to information or content. This individual did not have permission to do what they did. They had no authorization to convert and decode the code. This was clearly a hack.

We must address any wrongdoing committed by bad actors.

If it costs $50 million to properly secure the data on your website that previous audits had already alerted you as a problem, then that's on the incompetent government who failed to properly secure the data in the first place. Not on journalists ethically alerting you to fix the vulnerability. And, there's no "unauthorized access." Your system put that info into people's browsers. There's no "decoding" to view the source. That's not how any of this works.

As people started loudly mocking Governor Parson, he decided to double down, insisting that it was more than a simple "right click" and repeating that journalists had to "convert and decode the data."

Again, even if it took a few steps, that's still not hacking. It's still a case where the state agency made that info available. That's not on the journalists who responsibly disclosed it. It's on the state for failing to protect the data properly (and for collecting and storing too much data in the first place).

Indeed, in doing this ridiculous show of calling them hackers and threatening prosecution, all the state of Missouri has done is make damn sure that the next responsible/ethical journalists and/or security researchers will not alert the state to their stupidly bad security. Why take the risk?

Filed Under: blame the messenger, dese, disclosure, ethical disclosure, hacking, mike parson, private information, schools, social security numbers, st. louis, teachers, vulnerabilities
Companies: st. louis post-dispatch

Brazilian Court Agrees Wikipedia Can Use Publicly-Available Personal Information For An Article

from the wow,-you-don't-say dept

A couple of weeks ago, we wrote about a victory in the courts for Creative Commons licenses, noting that such judgments were still rather few and far between. That's unfortunate, in the sense that some people still think CC licensing is weird, rarely-used or even invalid. The situation regarding Wikipedia is similar. Even though it has been around for 15 years -- just like Creative Commons -- it too suffers from continuing doubts about its aims and methods, and a relative dearth of legal cases helping to clarify the status of both.

Here's one from Brazil, which has recently been settled in favour of Wikipedia's parent organization, the Wikimedia Foundation. It concerns the Brazilian musician Rosanah Fienngo, who had brought a lawsuit objecting to information about her personal life being included on her Portuguese Wikipedia page. Wikimedia pointed out:

The Portuguese Wikipedia article about Ms. Fienngo contained information about her as a notable public figure in Brazil. This information included some details of her personal life, but this information was derived from public sources, most of which Ms. Fienngo had provided herself, such as an interview Ms. Fienngo gave to the gossip website O Fuxico.

You would have thought that someone who had provided details about her personal life to a gossip website would (a) realize that people might pass on that public information -- that's how gossip works -- and (b) be grateful to those who spread details she herself had chosen to make public. Fortunately, the judge seems to have understood the situation:

The court stated that although the information available on her Wikipedia page concerned her private life, Ms. Fienngo had already disclosed that information to the media herself, so its inclusion on Wikipedia was not an invasion of her privacy.

It's ridiculous that it required a court case to establish that, but the good news is the judgment should help to discourage others from bringing more such suits. Well, probably. Unfortunately, another similarity between the Brazilian Wikipedia case and the earlier Creative Commons one is that Ms. Fienngo could make an appeal, although Wikimedia notes:

We believe that the decision was strong enough that community members should feel free to make editorial decisions to write articles like the one about Ms. Fienngo.

Let's hope they're right.

Follow me @glynmoody on Twitter or identi.ca, and +glynmoody on Google+

Filed Under: brazil, private information, public information, rosanah fienngo
Companies: wikimedia, wikipedia