Content Moderation Case Study: Twitter Suspends Users Who Tweet The Word 'Memphis' (2021)

from the memphis-memphis-memphis-memphis dept

Summary: Twitter users who made the mistake of tweeting out an innocuous word -- 'Memphis -- found themselves suspended from the service for 12 hours for apparently violating the terms of use.

According to messages sent to suspended users, the use of the Tennessee city name violated prohibitions on posting personal information.

The inadvertent damage quickly spread across Twitter as users trolled each other, trying to get unsuspecting accounts to tweet the suddenly-forbidden word. The apparent flaw in the auto-moderation system went unaddressed for several hours as more and more users found themselves temporarily prevented from using the service. Although some users noticed certain accounts (mainly verified ones) weren't being hit with bans, it affected enough users that the ripple effect was not only noticeable, but covered by many mainstream media outlets.

The bans were lifted several hours later with no explanation from Twitter other than that an unspecified "bug" had resulted in tweets containing the word "Memphis" being removed and features limited for those accounts.

That explanation was not entirely clear. Given the "Memphis" bug's link to alleged violations of Twitter's policies against posting other people's personal information, it was speculated the ban on a single city name may have been the result of an erroneously-completed form on the moderation side. Systems security professional SwiftOnSecurity took a plausible stab at the possible root cause of this improbable series of moderation events.

What's possible is a Twitter staffer tried to block a street address, but the postal syntax acted as an escape sequence, or the original was multi-line and they only pasted the city.

If so, every use of the word "Memphis" was considered to be a post containing a full address Twitter had targeted for removal under its personal information policy.

Decisions to be made by Twitter:

• Should moderation of the posting of personal information be handled with automation, given the potential for errors to scale and compound?
• Should a better stop-gap process be put in place to head off future events like these?
• Should users be given better explanations when moderation-at-scale results in features being limited for users affected by moderation bugs? 

• Should a stop-gap measure be put in place to prevent errors like this from becoming multi-hour failures?
• Have Twitter's moderation efforts resulted in a noticeable limitation of the spread of personal info? 
• Is all publication of personal information considered a violation of policy? Or are exceptions in place for information considered to be of public interest?

Originally posted to the Trust & Safety Foundation website.

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: blocklist, content moderation, memphis, memphis memphis, memphis memphis memphis
Companies: twitter