It's Not Personal: Content Moderation Always Involves Mistakes, Including Suspending Experts Sharing Knowledge
from the it-happens dept
I keep pointing out that content moderation at scale is impossible to do well. There are always going to be mistakes. And lots of them. We've spent years highlighting the many obvious mistakes that websites trying to make moderation decisions on thousands, hundreds of thousands, or even millions of pieces of content are going to make every day. It's completely natural for those who are on the receiving end of obviously bogus suspensions to take it personally -- though there does seem to be one group of people who have built an entire grievance complex on the false belief that the internet companies are targeting them specifically.
But if you look around, you can see examples of content moderation "mistakes" on a daily basis. Here's a perfect example. Dr. Matthew Knight, a respiratory physician in the UK, last week tweeted out a fairly uncontroversial statement about making sure there was adequate ventilation in the hospitality industry in order to help restart the economy. At this point, the scientific consensus is very much that good ventilation is absolutely key in preventing COVID transmission, and that the largest vector of super spreader events are indoor gatherings with inadequate ventilation. As such this tweet should be wholly uncontroversial:
To get hospitality working safely in the U.K. a significant investment is required in ventilation systems. The standards should be set ASAP and funding made available. Covid is airborne (as are other infections) and ventilation vital part of prevention. Good air quality vital
— DR MATTHEW KNIGHT MBE (@drmknight) May 19, 2021
And yet... despite this perfectly reasonable tweet from a clearly established expert, Twitter suspended his account for "spreading misleading and potentially harmful information related to COVID-19." It then rejected Dr. Knight's appeal.
There’s something wrong with Twitter’s censorship function. An Aerosol Scientist and a Respiratory Physician have both been blocked from accessing their accounts for “spreading misleading & potentially harmful information related to COVID-19”. How is this misleading/harmful? pic.twitter.com/wodknwKUKw
— Kristen K. Coleman (@drkristenkc) May 27, 2021
Thankfully it appears that Twitter eventually realized its mistake and gave Dr. Knight his account back. Lots of people are (understandably) asking why Twitter is so bad at this, and it's a fair enough question. But the simple fact is that the companies are all put in an impossible spot. When they weren't removing blatant mis- and disinfo about COVID-19, they were getting slammed from plenty of people (also for good reason). So they ramped up the efforts, and it still involves a large group of (usually non-experts) having to make a huge number of decisions very quickly.
There are always going to be mistakes. As Harvard's Evelyn Douek likes to note, content moderation is all about error rates. Each choice you make is going to have error rates. The biggest questions are what kinds of errors are preferable, and how many are you willing to deal with. Should the focus be on minimizing false positives? Or false negatives? Or somehow trying to balance the two? And the answers to that may vary given the circumstances and may change over time. But one thing that is clear is that no matter what choices are made, mistakes inevitably come with them, because content moderation at scale is simply impossible to do well.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: content moderation, covid, matthew knight
Companies: twitter
Reader Comments
Subscribe: RSS
View by: Time | Thread
The appeal rejection is the big problem
[ link to this | view in chronology ]
Re: The appeal rejection is the big problem
Nice idea, bur difficult to do because of the number of appeals. You can't allow a backlog to start as it will grow without limit and when the backlog starts it is too late to hire more people because it will continue to grow while you hire and train them..
[ link to this | view in chronology ]
You still can't win, but that isn't the reason.
Point of order: You don't just hire/train enough to handle the current rate, you hire enough to also dispose of the backlog.
The "excess judges" will eventually be absorbed as the appeals count increases. So yes, it can be done.
The issue isn't "you can't hire enough judges". The issue is "you can't afford to hire enough judges".
[ link to this | view in chronology ]
Re: You still can't win, but that isn't the reason.
The problem is not as simple as hire more moderators, you also need the buildings, managers, equipment, support personnel to support them. Meanwhile it is taking longer and longer for an appeal to reach moderation. At the scale of Twitter and Facebook et al. it is not a case of hiring a few more people, but more hire several thousand more people and try and train them before your problem grows even larger.
Overall, the problem is not how does Facebook moderate its users, but rather how do you moderate the whole of humanity.
[ link to this | view in chronology ]
Re: Re: You still can't win, but that isn't the reason.
And more and more people appeal, hoping to crash the system.
[ link to this | view in chronology ]
They Hope You Will Forget
It may be impossible to moderate at scale, but there will not be any improvement or accountability until the system stops being so opaque. Publish the algorithm, and explain why this one got censored.
[ link to this | view in chronology ]
Re: They Hope You Will Forget
Publish the algorithm, and then you'll have people manipulating it to the point of worthlessness.
Don't publish it, and then you have people claiming it's biased in some way.
There's no perfect solution.
[ link to this | view in chronology ]
Rigid and specific rules, a troll's wet dream
'As the specific rules that were forced upon platforms in the name of 'clarity' note saying the word 'Green' will get a comment flagged and removed, however I very clearly did not say green I said 'the primary color of plants on the planet earth' which is not prohibited and therefore the removal of my comment is against the rules and it deserves to be put back in place until they are further clarified to cover my comment.'
[ link to this | view in chronology ]
Always trust a troll to break a rules system by any means necessary.
[ link to this | view in chronology ]
content moderation is only working in one direction, the one that allows dire consequences on to whoever has offended someone or the software concerned
[ link to this | view in chronology ]
Scaling
Who is moderating the moderators?
Who is watching the watchers?
[ link to this | view in chronology ]
Re: Scaling
Quis moderabat ipsos moderatores?
I think that's how you say it in Latin…
[ link to this | view in chronology ]
Re: Re: Scaling
Comes from a Roman senator complaining that hiring a virile man to guard your wife while you are away on business is in itself a problem.
[ link to this | view in chronology ]
Strict liability
In cases such as Dr. Knight's, there should be strict liability for libel. Claiming untruthfully that a doctor of his credentials is "spreading misleading and potentially harmful information related to COVID-19" is libel per se, and considering the source of the libel, should be cause for substantial punitive damages as well.
[ link to this | view in chronology ]
Re: Strict liability
There's a difference when it's done with intentional malice and when it's not. This was not.
Swing and a miss.
[ link to this | view in chronology ]
Re: Re: Strict liability
The malice standard is NYT v. Sullivan and only applies to opinion. If you do a FACT check and say Dr. X is lying that is not an opinion. Its a statement of fact. It says so in the term fact check.
[ link to this | view in chronology ]
Re: Strict liability
Lol, that's not how defamation works. And it is absolutely not how "defamation per se" works.
[ link to this | view in chronology ]
Re: Re: Strict liability
Mike, is that the US law take on it or the UK?
[ link to this | view in chronology ]
Re: Re: Re: Strict liability
US Law.
[ link to this | view in chronology ]
Re: Re: Re: Strict liability
Relevant law.
[ link to this | view in chronology ]
Re: Strict liability
If that were true you would have been sued to the poorhouse for your bad legal takes.
[ link to this | view in chronology ]
The problem *is* the scale...
In any large ecosystem that supports any level of controversy, what to moderate out starts to depend heavily on which audience member happens to be looking.
For example, the mythical average techdirt commenter hates what kobe or OOTB says, flags it for boring and doesn't want to hear it .... but I'm pretty sure one of Mike Masnick's friends has been studying those very same comments.
and that's before we get to the idea that we can't expect a large platform with huge numbers of commenters on X to also have huge numbers of experts on X, for all controversies X.
If twitter (or facebook) wants to do better, it's going to need to establish some trusted public figures... and good luck, because we collectively can't decide whether to trust Donald Trump or not, even if every techdirt poster comes to the same decision.
[ link to this | view in chronology ]
"It's completely natural for those who are on the receiving end of obviously bogus suspensions to take it personally"
I feel seen. GLARES
Another metric: how many suspensions go over 100 days with no response. There are to many competing goals in the mix & its making things much worse.
In response to a video of someone being 'kidnapped' for a party (victim did not know until later it was a prank) friend of mine said if someone grabbed her like that she would punch him.
12 hour timeout, having to remove the tweet, because it was wishing pain or harm on someone.
(HI KAT!)
"If someone assaults me I would hit them." - Banable tweet treated on the same level as someone threatening harm to all the Jews.
Zero tolerance policies still are bad ideas.
More timeouts for people threatening hypothetical imaginary people (who aren't part of one of the 10K protected groups on twitter) doesn't make the platform better or "safer".
The CoVid Misinfo AI is bugged, Senator Anti-Vaxx is still promoting all of his crazy ideas (can still see them on the platform) while some sociopath mocks morons who think there is a chip in the vaccines has the tweet hidden and is over 100 days in TwitMo waiting for assistance. I've offered to delete the tweet, but prying a phone number from my poor hands matters more than anything else it seems.
I guess if you use a trendy avatar, call yourself an immortal sociopath, you are held to higher standards than members of Congress.
[ link to this | view in chronology ]
That's why most people have learned to refer to the Disease-That-Must-Not-Be-Named using euphemisms. Because saying That Word in any context at all is a magnet for moderation "mistakes."
No, I don't think this is a desireable state of things.
[ link to this | view in chronology ]
Re:
And now that its the poster child for all the bad on the platform they are overreacting to everything assuming people will understand that 'out of an abundance of caution' we gave you a time out & then doubled down on it even if everyone else on the planet says we are wrong.
[ link to this | view in chronology ]
The Disease-That-Must-Not-Be-Named
syphilis would explain the insanity.
[ link to this | view in chronology ]
While they can train an AI to spot & flag "bad things", do they ever teach it the mistakes they have to fix?
[ link to this | view in chronology ]
Re:
@TAC, the problem is, let me take your "mistake" and now hold it up to ridicule with a word...
How you gonna teach an AI to know the difference between
"BS, Vaxxes work!"
and
"Vaxxes work!"
All it took was a word, or a small difference in context, and the meanings are opposite. It ain't gonna happen!
[ link to this | view in chronology ]
Re: Re:
I still insist it was because I said they claimed Trump was the messiah.
[ link to this | view in chronology ]
Re:
Sure they do. But as soon as they do, the groups they want to block change their strategy again. And again. And again.
And now you're 15 AIs in, and the 3rd "mistake" is now being used by an entirely unrelated group for entirely unrelated reasons, and the 2nd and 7th directly contradict the 11th and 12th respectively, and the 9th is actually being simultaneously used by three different "sides" of the issue, and the the 14th shouldn't have been included at all because the human moderators messed it up....
and your AI remains just as useless as before
[ link to this | view in chronology ]
Re: Re:
Their AI now is pretty useless already.
[ link to this | view in chronology ]
Bad Example of something that is purported to be moderation
https://www.theregister.com/2021/06/01/google_usenet/
The Usenet group comp.lang.tcl vanished from Google Groups for several hours before being restored on Tuesday.
...
Google took over the Usenet archive when it acquired Deja News in 2001.
Almost a year ago, comp.lang.forth and comp.lang.lisp, were also removed from Google Groups. And they remain unavailable. Likewise, comp.lang.c and comp.lang.python are currently inaccessible via Google Groups.
The suppression of Usenet groups has cultural, academic, and technical consequences. Some active systems, for example, still rely on Forth.
[ link to this | view in chronology ]
I agree with your analysis that moderation on the scale of the world is impossible to do right. But your conclusion that it's not personal, that moderation is always going to suck and people should accept it as inevitable as death and taxes I can't support. They should do it in a scale that they can do right.
If they're doing too much moderation to do it right, then they're doing too much moderation. At the point where the issue stops being humans disagreeing on what should be moderated (which truly is inevitable) and becomes instead humans being too busy to moderate so they let dumb algorithms auto-ban people and auto-deny appeals because it's easier than doing their jobs, then they are moderating too much.
Despite what some seem to think, people saying stupid things on Twitter is not the end of the world. It's not worth it to randomly ban normal people just for discussing a controversial topic in an effort to try to silence some conspiracy nuts' inane BS.
[ link to this | view in chronology ]
They literally can’t — not without spending far more money and hiring far more people than should ever be necessary for a moderation team.
The whole point of saying “moderation doesn’t scale” is to point out how moderating a small community is always going to be easier than moderating a large community. If you’re moderating a group of a couple dozen people or so, you’re gonna have an easier time of telling people to knock their bullshit off — mostly because a community that small will have its own quirks and contexts under which it can be moderated. But a community of a couple…oh, let’s say a couple hundred thousand people will require far more moderators and far harder calls to make because those people can branch into subcommunities and develop their own quirks and contexts that won’t parse universally. The ability to moderate all those people doesn’t scale well when compared to the ability to moderate a much smaller amount of people.
Even people who moderate small communities never get it right 100% of the time. How do you expect a company/service/“community” as large as Twitter to get it perfect?
[ link to this | view in chronology ]
And yet...
... the Russian bot farms can continue to spread blatant misinformation with impunity. They're not countered until someone with high visibility makes a case against them.
Why does Twitter protect the Russians?
[ link to this | view in chronology ]
Re: And yet...
Why do you make statements that has no factual basis?
Btw, Twitter bans thousands of bot-accounts they detect regularly, but you seem to know exactly how to unerringly identify these Russian bot farms. Perhaps you should inform Twitter how you do it so they can implement it.
[ link to this | view in chronology ]
The problem isn’t so much the moderation but the automated rejection of appeal and the blank generics of “why”.
TOS is nothing more than local civil law, at its most basic construct. When you are to be punished because you break the law you should be informed of exactly what you did. Not generic ‘section 4 subsection b’ but the actual infraction:
“Posting false information, in your case ‘C19 is not contagious’”
Appeals should not be automated. If the poster can show a source then the content, and account, should be restored and marked as “disputed” when the poster adds a link edit.
All of this can be avoided by a tos clause that “we can ban you at any time for any reason”. Your using private property.
When such a clause exists there’s nothing to do.
When it doesn’t and a ban is based on violations such violations need to be spelled out.
[ link to this | view in chronology ]