The Impossibility Of Content Moderation: YouTube's New Ban On Nazis Hits Reporter Who Documents Extremism, Professor Teaching About Hitler

from the so-that's-all-working-well dept

So just as the recent big content moderation mess was happening on YouTube, the company announced that it had changed its policies to better deal with violent extremism and supremacism on the platform:

Today, we're taking another step in our hate speech policy by specifically prohibiting videos alleging that a group is superior in order to justify discrimination, segregation or exclusion based on qualities like age, gender, race, caste, religion, sexual orientation or veteran status. This would include, for example, videos that promote or glorify Nazi ideology, which is inherently discriminatory. Finally, we will remove content denying that well-documented violent events, like the Holocaust or the shooting at Sandy Hook Elementary, took place.

The timing of this announcement was seen as curious (or, at the very least, poorly timed) as it came basically hours after they had refused to take down Steven Crowder's account (see the earlier post linked above), even though that wasn't an identical situation -- though analogous enough that tons of people commented on it.

In making the announcement, YouTube correctly noted that this new bit of line drawing could represent some problems, including among those tracking hate and extremism:

We recognize some of this content has value to researchers and NGOs looking to understand hate in order to combat it, and we are exploring options to make it available to them in the future. And as always, context matters, so some videos could remain up because they discuss topics like pending legislation, aim to condemn or expose hate, or provide analysis of current events. We will begin enforcing this updated policy today; however, it will take time for our systems to fully ramp up and we’ll be gradually expanding coverage over the next several months.

But within hours of the new policy rolling out, we were already seeing how difficult it is to implement without taking down content that probably deserves to remain up. Ford Fischer, a reporter who tracks extremist and hate groups, and whose work is regularly cited, noted that his own channel had been demonetized.

Fischer than discusses the specific videos that YouTube says is the reason for this -- and it does include holocaust denialism, but for the sake of documenting it, not promoting it:

And this gets, once again, to the very problem of expecting platforms to police this kind of speech. The exact same content can mean very different things in different contexts. In some cases, it may be used to promote odious ideology. In other cases, it's used to document and expose that ideology and the ignorance and problems associated with it.

But how do you craft a policy that can determine one from the other? As YouTube is discovering (truth is, they probably already knew this), the answer is that you don't. Any policy ends up creating some sort of collateral damage, and the demands from well meaning people mean that the direction this tends to go in leads to greater and greater takedowns. But, if in the process of doing this we end up sweeping the documentation under the rug, that's a problem as well.

Here's another example: right after YouTube's new policy was put in place, a history teacher found that his own YouTube channel was banned. Why? Because he hosted archival footage of Hitler:

“My stomach fell,” Allsop told BuzzFeed News via email. “I’m a history teacher, not someone who promotes hatred. I share archive footage and study materials to help students learn about the past.”

Once again, it often sounds easy to say something like "well, let's ban the Nazis." I'd even argue it's a reasonable goal for a platform to have a blanket "no Nazis" policy. But the reality is that the implementation is not nearly as easy as many people believe. And the end result can be that archival and documentary footage gets blocked. And that could have serious long term consequences if part of our goal is to educate people about why Nazis are bad.

Of course, none of this should come as a surprise to anyone who's been dealing with these issues over the past couple of decades. Early attempts to ban "porn" also took down information on breast cancer. Attempts to block "terrorist content" have repeatedly taken down people documenting war crimes. This kind of thing happens over and over and over again and believing that this time will magically be different is a fool's errand.

Filed Under: content moderation, content moderation at scale, documenting, learning, nazis, policing speech, reporting, research
Companies: youtube

Twitter Permanently Suspends (Then Unsuspends) Lawyer For Telling NRA Supporters To 'Fuck Off' And 'Own The Death'

from the seems-a-bit-harsh dept

For all the talk of how Twitter supposedly is banning conservatives left and right (it's not), the company is actually dealing with the impossibility of handling content moderation at scale. Adequately determining which content is "good" and which is "bad" is an impossible task.

Let's take an example case, which -- at the very least -- should show that Twitter isn't just banning conservatives. The pseudonymous defense lawyer known as "bmaz" is probably known to many readers here. He's been a long-time co-author of the Empty Wheel blog with Marcy Wheeler. He's also a prolific and emotional tweeter. Sometimes I agree with him and sometimes I do not. When I do not, he doesn't shy away from letting me hear about it, often expressing his opinions strongly -- which is something I've always appreciated about bmaz. He may not be polite, but he's direct and doesn't hide his true feelings.

And as of last week he was gone from Twitter. According to Scott Greenfield, bmaz has been "permanently" banned from Twitter. Why? Because he told a bunch of NRA supporters his general feelings towards their position, with two of them being "Own the death you piece of crap" and "You too are a fucking idiot asshole. As are your whole 62 followers. Fuck off."

If you can't see that, it shows the two tweets quoted above (both directed at a short list of NRA spokespeople or supporters, as part of what appears to be an ongoing thread -- though there's no way to check that now). It also shows Twitter giving its reasons for cutting bmaz off.

You account, bmaz has been suspended for violating the Twitter Rules.

Specifically, for:

Violating our rules against abusive behavior. You may not engage in the targeted harassment of someone, or incite other people to do so. We consider abusive behavior an attempt to harass, intimidate, or silence someone else's voice.

It also warns him against trying to set up an alternative account.

Now, if you were to believe the prevailing narrative among some, Twitter only bans conservatives, not people angrily tweeting at a bunch of NRA folks. But here, it did the opposite. I have trouble seeing how the two tweets in question constitute "targeted harassment" or any sort of "attempt to harass, intimidate, or silence someone else's voice." I mean, among the people the tweet was directed at was Dana Loesch who famously starred in some trolling NRA ads warning the media, athletes, and liberal politicians that their "time is running out." In another ad, Loesch warns that liberals will "perish in political fire."

Those seem a bit more harassing and intimidating than "own the death" and "fuck off," but what do I know?

Anyway, the point here isn't to say that Twitter is bad at this (though, it is). Rather it's to point out that any large social media platform is going to be bad at this. There is no good way to moderate content at scale without making lots of mistakes. In isolation, these tweets certainly appear angry and rude, but without much trouble I could find a dozen similar tweets in various flamewars that are happening on Twitter as we speak. I'm pretty sure I've lost my cool and told folks on Twitter to "fuck off" as well. It happens.

bmaz appealed his suspension, and that's already been rejected as well, with Twitter now giving him a totally different response as to why he was banned:

Hello,
Your account has been suspended and will not be restored because it was found to be violating Twitter’s Terms of Service, specifically the Twitter Rules against hateful conduct.

It is against our rules to promote violence against or directly attack or threaten other people on the basis of race, ethnicity, national origin, sexual orientation, gender, gender identity, religious affiliation, age, disability, or disease.

Again, nothing he did appears to involve promoting violence against anyone or threatening people on any of those bases. Instead, he was just venting at people who he believed were idiots. As Scott Greenfield notes:

Disagree with what @bmaz thinks of guns all you want. Disagree with @bmaz’s way of expressing himself. Or agree with him. It doesn’t matter. There was no violation of the terms of service, no targeted harassment, no “attempt to silence” someone else’s voice. There was certainly no violation on the basis of race, etc., unless the mere hint that a woman (and just because the handle says it’s a woman, does that make it so?) was on the receiving end converts all make speech into a sexual assault.

Greenfield asks people to go on Twitter and speak up for bmaz, and ask Jack Dorsey to reverse the ban -- which is probably a good idea (though it's not Jack personally making these calls). And it appears to have worked. As of last night bmaz was back on Twitter, though he notes he has no idea how or why. It seems likely that a bunch of people asking Twitter "what the fuck" probably helped.

But there's a larger point here. When people keep demanding that Twitter "do something" to handle some of the very real problems of abuse and harassment on its site, they have to recognize that "doing something" that doesn't involve some really bad mistakes is effectively impossible at the size and scale that Twitter operates at. It's going to make big mistakes, and it's going to make them frequently. That's not because Twitter is bad or that the people who work there are bad. It's not that there's a "conservative bias" at Twitter (which should be evidence given the players in this particular story). It's that this is an impossible task. Twitter, I'm sure, has some technology tools and some humans trying to review a ton of content, and rather than lots of obvious cases of black and white, the vast majority of them will land in a gray zone. And there's literally no way that anyone can build in "context" into reviewing these kinds of moderation choices, because understanding the context is next to impossible, not to mention incredibly time consuming (especially when that context may involve things totally outside of Twitter).

Twitter almost certainly made a mistake here, but the point isn't to scream that Twitter is awful because it made a mistake. It's to recognize that when we suddenly demand that private companies be in charge of regulating speech, don't be surprised when they do it badly -- because that's the only way to moderate content at scale. Mistakes will be made -- both false positives and false negatives -- and at the scale of the users and content on a platform like Twitter, it's going to impact lots and lots of people.

So, sure, when someone who shouldn't have been gets suspended, feel free to call out Twitter for its mistakes. But don't expect that there's some magic wand that will suddenly make anyone (Twitter or any other company) suddenly able to actually do a good job at content moderation at scale.

Filed Under: bmaz, content moderation, content moderation at scale, policing speech, social media
Companies: nra, twitter