Content Moderation Case Study: Facebook Struggles To Correctly Moderate The Word 'Hoe' (2021)
from the language-is-a-funny-thing dept
Summary: One of the many challenges with content moderation is the flexibility of language. When applying blocklists — a list of prohibited terms considered not appropriate for the platform — moderators need to consider innocuous uses of words that, when removed from their context, appear to be violations of the platform’s terms of use.
Multiple platforms have run into the phenomenon known as the "Scunthorpe problem." In this famous case, a town whose name no one would ever mistake for offensive was deemed offensive by moderation blocklists simply because within the name of the town is the word “cunt” which many blocklists forbids.
Deploying automated blocklists can be even more challenging when dealing with specialized or niche content, which may use certain terms that are offensive outside of this specific context, but are essential to discussing and understanding the relevant subject matter. A paleontologists' conference was derailed when the moderation blocklist made it impossible for participants to use words like "bone," "pubic," "stream," and "beaver."
Facebook has worked continuously to refine its moderation processes, but it still occasionally makes the wrong call when it comes to their blocklists. In January 2021, residents of (and visitors to) a Devon, England landmark were surprised to find their posts and comments vanishing from the site. After a little investigation, it became clear Facebook was deleting posts containing references to the landmark known as Plymouth Hoe.
In addition to being the name of a common garden tool (more on that in a moment), "hoe" also refers to a "sloping ridge shaped like an inverted foot or heel," such as Plymouth Hoe, which is known locally as the Hoe. Users were temporarily forced to self-censor the harmless term to avoid moderation, either by adding unnecessary punctuation or dropping the "h." It appeared Facebook's automated processes believed these comments and posts were using a derogatory term for a romantic partner who is only in a relationship to better their own financial position.
Facebook soon apologized for the moderation error and stated it was "taking steps to rectify the error" and figure out what caused the mistaken moderation in the first place. Problem solved?
Not really.
The same problem popped up again, this time affecting a New York gardening group. WNY Gardeners, a group with more than 8,000 members, is the latest to be affected by Facebook's "hoe" pruning. A member responded to the prompt "most loved & indispensable weeding tool" with "Push pull hoe!" Not long after that, the member was informed by Facebook that the comment violated the site's policy on bullying and harassment.
Company Considerations:
- How could blocklists and keyword searches be better utilized to detect and remove violations of site policies?
- How much collateral damage from automated moderation should be considered acceptable? Is this an acceptable trade-off for lower moderation costs, which often relies on more automated moderation and fewer human moderators?
- Can AI-based moderation more reliably detect actual violations (rather than innocuous uses of blocklisted terms) as the technology advances? What are the trade-offs with AI-based moderation tools as compared to simple blocklists?
- What mitigation measures might be put in place to deal with a blocklist that catches words with different meanings depending on context?
- Who should be in charge of reviewing a blocklist and how frequently should it be updated?
Issue Considerations:
- Does prohibiting words like “hoe” make a significant dent in online harassment and abuse? Does the tech have the capability to "catch up" (or surpass) the ability of humans to route around moderation efforts?
- Should more resources go to staffing human moderators in order to prevent errors and/or allow for a more robust challenge process that allows content to remain "live" until the challenge process has concluded?
- What ways might automation and human reviewers be used in combination to avoid the more egregious automated blocklist mistakes?
Resolution: Once again, Facebook has apologized for not recognizing the word "hoe" in contexts where it's appropriate to use. But after two highly-publicized incidents in less than a year -- both involving the same word -- Facebook has added human moderators to backstop automated calls on flagged terms like these in order to prevent unjustified removals of posts, accounts, or groups.
Originally posted to the Trust & Safety Foundation website.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: content moderation, hoe, scunthorpe problem
Companies: facebook
Reader Comments
Subscribe: RSS
View by: Time | Thread
Has anyone seen my Hoe?
[ link to this | view in chronology ]
English only?
And you've not even considered the difficulties FaceBook and many other sites have with languages other than English... and even more difficult: languages with loan words from other languages where the loaned word has derogatory implications in one language, but not the other.
That's before we even come close to considering differences in culture in a multi-lingual cultures, or languages shared amongst different cultures with varying mores around offense in a written form.
[ link to this | view in chronology ]
Re: English only?
Chinese Filler words...
There is very common Chinese Filler word that happens to sound like a term for former slaves from africa in the US. It got at least one professor fired who was teaching a class on Chinese language.
Just wait till some voice AI hears a Chinese speaker and emergency bans an event over a filler word.
[ link to this | view in chronology ]
Re: Re: English only?
A Chinese filler word?
Ohhhhhh. That one.
[ link to this | view in chronology ]
Re: English only?
"languages with loan words from other languages where the loaned word has derogatory implications in one language, but not the other"
You don't even have to go as far as loan words. For example, the Swedish word for "end" is "slut". Good luck with a multilingual filter that has no false positives.
"That's before we even come close to considering differences in culture in a multi-lingual cultures, or languages shared amongst different cultures with varying mores around offense in a written form."
Then, of course, language evolves over time. A sentence written a decade ago could be perfectly innocent but contain some offensive connotation today. Then, dealing with acronyms. Whether your WAP comment is lascivious in nature or a boring discussion of wireless internet options could simply be a matter of timing.
[ link to this | view in chronology ]
Merry moderation
Hey Facebookmeta, Hoe, Hoe, Hoe!
[ link to this | view in chronology ]
Everyone is working to perfect artificial intelligence, but they've already perfected artificial stupidity.
[ link to this | view in chronology ]
For a while they were blocking posts in a group I know of that contained the word “white.” It boggles the mind that they have been left in charge of our cultural commons, by our communal laziness and inaction.
[ link to this | view in chronology ]
Re:
"For a while they were blocking posts in a group I know of that contained the word “white.”"
I've never heard that before so I'm guessing there must be some other context. Certainly I've never come across such things in groups I'm involved with, be that talking about a popular chef in my area with the surname White or talking about the movie White Christmas in movie related groups when the appropriate season arrives.
"It boggles the mind that they have been left in charge of our cultural commons, by our communal laziness and inaction."
It boggles the mind that someone who genuinely believes this is so lazy not to use their competitors instead. If they have any control over the commons at all, it's only because people choose to use them, so it's weird that someone who objects to them so strongly should choose to do so.
[ link to this | view in chronology ]
Re:
Maybe the word after "white" was "power" 🤔
[ link to this | view in chronology ]
what is a hoe
A hoe is ethnically slang english word for 'whore', its not "a derogatory term for a romantic partner who is only in a relationship to better their own financial position. That would be a prostitute.
[ link to this | view in chronology ]
Re: what is a hoe
"Whore" is a synonym for "prostitute".
[ link to this | view in chronology ]
Re: what is a hoe
Ho' is obviously derived from whore, but my (flawed?) understanding is that the meaning is more 'gold-digger', though more derogatory.
[ link to this | view in chronology ]
Farming implents and sex workers
At this point I've only seen ho used as colloquial slang for whore. It has its own problems, between Santa's laugh, the humorous slang for the Hostess product. Whoopi Goldberg's company is One Ho Productions
Whenever I've read hoe it was referring to the gardening tool...unless it was in a rant or screed rife with misspellings and apostrophes used in plurals rather than possessives. In that case, I still read it as a farm tool because it amuses me.
The solution in my opinion is to descandalize sex work. Especially considering ours is an economy fit to drive workers to desperation.
In the meantime, our moderating systems are not only going to have to monitor for a continuous run of slurs, but will also have to watch for misspellings, intentional or otherwise. I've only heard of simps around August.
[ link to this | view in chronology ]
Word filters never work. I once played a mobile game where the chat had such aggressive substring filters it made a normal conversation look like drunken sailors with all the bleep-outs. It would not only block the usual over-inclusive lists of bad words some intern googled, but also parts of bad words (shortest only 2 letters long), misspellings, and anagrams, all of it including across punctuation and in the middle of longer words. It was difficult to carry conversations, constantly having to evade the filters or explain the bleeped bits, especially with new players who weren't familiar with the bleeps.
ButdDespite turning the scunthorpe problem up to eleven, people still managed to cuss eachother just fine when they were actually trying to avoid the system.
[ link to this | view in chronology ]
Re:
I recall Neverwinter Nights blocked the word "ass" in multiplayer, which was a slight problem when one of the playable classes was "assassin"
[ link to this | view in chronology ]
Re: Re:
Did anybody try to write it as Arsearsein?
[ link to this | view in chronology ]
Yeah, moderation... it's a tough road to hoe.
[ link to this | view in chronology ]
Re:
Tough row to hoe. One does not hoe a road.
[ link to this | view in chronology ]
Re: Re:
unless it's using a backhoe
[ link to this | view in chronology ]
I was always under the impression the offensive word was spelled "ho", not "hoe". Of course with the state of people's grammar today, I wouldn't be surprised to see it spelled "howe".
[ link to this | view in chronology ]