Content Moderation Case Study: Facebook Struggles To Correctly Moderate The Word 'Hoe' (2021)

from the language-is-a-funny-thing dept

Summary: One of the many challenges with content moderation is the flexibility of language. When applying blocklists — a list of prohibited terms considered not appropriate for the platform — moderators need to consider innocuous uses of words that, when removed from their context, appear to be violations of the platform’s terms of use.

Multiple platforms have run into the phenomenon known as the "Scunthorpe problem." In this famous case, a town whose name no one would ever mistake for offensive was deemed offensive by moderation blocklists simply because within the name of the town is the word “cunt” which many blocklists forbids.

Deploying automated blocklists can be even more challenging when dealing with specialized or niche content, which may use certain terms that are offensive outside of this specific context, but are essential to discussing and understanding the relevant subject matter. A paleontologists' conference was derailed when the moderation blocklist made it impossible for participants to use words like "bone," "pubic," "stream," and "beaver."

Facebook has worked continuously to refine its moderation processes, but it still occasionally makes the wrong call when it comes to their blocklists. In January 2021, residents of (and visitors to) a Devon, England landmark were surprised to find their posts and comments vanishing from the site. After a little investigation, it became clear Facebook was deleting posts containing references to the landmark known as Plymouth Hoe.

In addition to being the name of a common garden tool (more on that in a moment), "hoe" also refers to a "sloping ridge shaped like an inverted foot or heel," such as Plymouth Hoe, which is known locally as the Hoe. Users were temporarily forced to self-censor the harmless term to avoid moderation, either by adding unnecessary punctuation or dropping the "h." It appeared Facebook's automated processes believed these comments and posts were using a derogatory term for a romantic partner who is only in a relationship to better their own financial position.

Facebook soon apologized for the moderation error and stated it was "taking steps to rectify the error" and figure out what caused the mistaken moderation in the first place. Problem solved?

Not really.

The same problem popped up again, this time affecting a New York gardening group. WNY Gardeners, a group with more than 8,000 members, is the latest to be affected by Facebook's "hoe" pruning. A member responded to the prompt "most loved & indispensable weeding tool" with "Push pull hoe!" Not long after that, the member was informed by Facebook that the comment violated the site's policy on bullying and harassment.

Company Considerations:

• How could blocklists and keyword searches be better utilized to detect and remove violations of site policies? 
• How much collateral damage from automated moderation should be considered acceptable? Is this an acceptable trade-off for lower moderation costs, which often relies on more automated moderation and fewer human moderators?
• Can AI-based moderation more reliably detect actual violations (rather than innocuous uses of blocklisted terms) as the technology advances? What are the trade-offs with AI-based moderation tools as compared to simple blocklists? 
• What mitigation measures might be put in place to deal with a blocklist that catches words with different meanings depending on context?
• Who should be in charge of reviewing a blocklist and how frequently should it be updated? 

Issue Considerations:

• Does prohibiting words like “hoe” make a significant dent in online harassment and abuse? Does the tech have the capability to "catch up" (or surpass) the ability of humans to route around moderation efforts?
• Should more resources go to staffing human moderators in order to prevent errors and/or allow for a more robust challenge process that allows content to remain "live" until the challenge process has concluded?
• What ways might automation and human reviewers be used in combination to avoid the more egregious automated blocklist mistakes?

Resolution: Once again, Facebook has apologized for not recognizing the word "hoe" in contexts where it's appropriate to use. But after two highly-publicized incidents in less than a year -- both involving the same word -- Facebook has added human moderators to backstop automated calls on flagged terms like these in order to prevent unjustified removals of posts, accounts, or groups.

Originally posted to the Trust & Safety Foundation website.

Filed Under: content moderation, hoe, scunthorpe problem
Companies: facebook

Content Moderation Case Study: Linkedin Blocks Access To Journalist Profiles In China (2021)

from the appeasement dept

Summary: A major challenge for global internet companies is figuring how to deal with different rules and regulations within different countries. This has proven especially difficult for internet companies looking to operate in China — a country in which many of the most popular global websites are blocked.

In 2015, there was an article highlighting how companies like Evernote and LinkedIn had avoided getting blocked in China, mainly by complying with the Chinese government’s demands that they moderate certain content. In that article, LinkedIn’s then-CEO Jeff Weiner noted:

"We're expecting there will be requests to filter content," he said. "We are strongly in support of freedom of expression and we are opposed to censorship," he said, but "that's going to be necessary for us to achieve the kind of scale that we'd like to be able to deliver to our membership."

Swedish journalist Jojje Olsson tweeted the article when it came out. Six years later LinkedIn informed Olsson that his own LinkedIn profile would no longer be available in China after referencing the Tiananmen square massacre in his profile.

In Olsson’s tweet, he explains that his LinkedIn profile mentions that for his degree, he wrote an essay about the Tiananmen Square massacre. It quickly became clear that LinkedIn was in the process of blocking access to multiple journalists’ and academics’ accounts in China, including CNN Beijing bureau chief Steve Jiang and the editor-in-chief of the Taiwan Sentinel, J. Michael Cole. The Wall Street Journal found at least 10 other LinkedIn accounts that were blocked in China around the same time, and highlighted that LinkedIn officials were reprimanded in March of 2021 for keeping certain accounts available in China.

China’s internet regulator summoned LinkedIn officials in March to tell them to better regulate its content, according to people familiar with the matter. The social-networking site was given 30 days to clean up the content and promised to better regulate its site going forward, the people said.

Shortly after, LinkedIn said in a statement on its website that it would be pausing new member sign-ups as the platform worked “to ensure we remain in compliance with local law.” -- Liza Lin, Wall Street Journal

The NY Times report on that meeting noted that the 30-day pause on sign-ups was part of what Chinese officials ordered.

The users whose profiles were blocked received a notice from Linkedin about the block, saying “We will work with you to minimize the impact and can review your profile’s accessibility within China if you update the relevant sections of the profiles,” but also notes “the decision whether to update your profile is yours.” The notice also includes this paragraph:

While we strongly support freedom of expression, we recognized when we launched that we would need to adhere to the requirements of the Chinese government in order to operate in China. As a reminder, your profile will remain viewable throughout the rest of the countries in which LinkedIn is available.

It appears that LinkedIn was also directly removing some specific content as well. Former journalist Peter Humphrey told Bloomberg News that LinkedIn informed him that it had completely removed certain comments he made criticizing the Chinese government.

Company Considerations:

• How important is it to remain accessible in China?
• What compromises are worth making to remain accessible in China or other countries?
• If the company agrees to take down, or block access to, certain content to appease government demands, how should those decisions be communicated to impacted users?
• Under what conditions, if any, will the company push back on overbroad demands to block content in China?

Issue Considerations:

• Is a censored, but still mostly available, US-based service better to be available than to have the entire service blocked in China?
• Local regulations differ across every country. What kind of framework should a company use to determine where they draw the line, and what compromises they will agree to?

Resolution: Since the initial flurry of notices that got attention from May through July of 2021, it appears that even more journalists have found their profiles blocked in China. In September, Sophia Yan, the China correspondent for the UK’s Telegraph, noted that her LinkedIn profile was now blocked in China. In replying to Yan’s tweet, Liza Lin, the Wall Street Journal’s China correspondent, and author of the article quoted earlier discussing LinkedIn officials being reprimanded by Chinese officials, noted that she too had her profile blocked in China.

LinkedIn, for its part, has continued to make similar statements throughout, saying that it supports the principles of free speech but that in order to continue operating in China, it is required by the government to block access to these accounts.

Update: Just weeks after this case study was originally published, and LinkedIn was called out for even more such activity, the company announced that it was mostly exiting the country, as the demands for censorship were becoming too much.

Originally posted to the Trust & Safety Foundation website.

Filed Under: censorship, china, content moderation, journalists
Companies: linkedin, microsoft

Content Moderation Case Studies: Snapchat Disables GIPHY Integration After Racist 'Sticker' Is Discovered (2018)

from the sticker-moderation dept

Summary: Snapchat debuted to immediate success a decade ago, drawing in millions of users with its playful take on instant messaging that combined photos and short videos with a large selection of filters and "stickers." Stickers are graphics that can be applied to messages, allowing users to punch up their presentations (so to speak).

Snapchat’s innovations in the messaging space proved incredibly popular, moving Snapchat from upstart to major player in a few short years. It also created more headaches for moderators as sent messages soared past millions per day to billions.

Continuing its expansion of user options, Snapchat announced its integration with Giphy, a large online repository of GIFs, in February 2018. This gave users access to Giphy's library of images to use as stickers in messages.

But the addition of thousands of images to billions of messages quickly resulted in an unforeseen problem. In early March of 2018, Snapchat users reported a search of the GIPHY image database for the word "crime" surfaced a racist sticker, as reported by Josh Constine for TechCrunch:

“We first reported Instagram was building a GIPHY integration back in January before it launched a week later, with Snapchat adding a similar feature in February. But it wasn’t long before things went wrong. First spotted by a user in the U.K. around March 8th, the GIF included a racial slur.” — Josh Constine, TechCrunch

Both platforms immediately pulled the plug on the integration while they sorted things out with GIPHY.

Company Considerations:

• What measures can be put in place to prevent moderation problems from moving from one platform to another during cross-platform integration?
• What steps should be taken prior to launch to integrate moderation efforts between platforms? 
• What can "upline" content providers do to ensure content moving from their platforms to others meets the content standards of the "downline" platforms? 

Issue Considerations:

• What procedures aid in facilitating cross-platform moderation? 
• Which party should have final say on moderation efforts, the content provider or the content user?

Resolution: Instagram was the first to reinstate its connection with GIPHY, promising to use more moderators to examine incoming content from the image site:

“We’ve been in close contact with GIPHY throughout this process and we’re confident that they have put measures in place to ensure that Instagram users have a good experience” an Instagram spokesperson told TechCrunch.

GIPHY offered its own apology for the racist image, blaming the slipup on a bug in its filters. Here's what GIPHY's spokesperson told Gizmodo:

After investigation of the incident, this sticker was available due to a bug in our content moderation filters specifically affecting GIF stickers.

We have fixed the bug and have re-moderated all of the GIF stickers in our library.

The GIPHY staff is also further reviewing every GIF sticker by hand and should be finished shortly.

Snapchat was the last to reinstate its connection to GIPHY, stating it was working directly with the site to revamp both moderation systems to ensure offensive content would be prevented from being uploaded to GIPHY and/or making the leap to connected social media services.

Originally published to the Trust & Safety Foundation website.

Filed Under: 3rd parties, bigotry, integration, racisms, stickers
Companies: giphy, snapchat

Content Moderation Case Study: Tumblr's Approach To Adult Content (2013)

from the how-to-destroy-a-website dept

Summary: There are unique challenges in handling adult content on a website, whether it’s an outright ban, selectively allowed, cordoned off under content warnings, or (in some cases) actively encouraged.Tumblr’s early approaches to dealing with adult content on its site is an interesting illustration in the interaction between user tagging and how a site’s own tools interact with such tags.

Tumblr was launched in 2007 as a simple “blogging” platform that was quick and easy to setup, but would allow users to customize it however they wanted, and use their own domain names. One key feature of Tumblr that was different from other blogs was an early version of social networking features — such as the ability to “follow” other users and to then see a feed of those users you followed. While some of this was possible via early RSS readers, it was both technologically clunky and didn’t really have the social aspect of knowing who was following you or being able to see both followers and followees of accounts you liked. Tumblr was also an early pioneer in reblogging — allowing another user to repost your content with additional commentary.

Because of this more social nature, Tumblr grew quickly among certain communities. This included communities focused on adult content. In 2013, it was reported that 11.4% of Tumblr’s top domains were for adult content. In May of 2013, Yahoo bought Tumblr for $1.1 billion, with an explicit promise not to “screw it up.” Many people raised concerns about how Yahoo would handle the amount of adult content on the site, but the company’s founder, David Karp, insisted that they had no intention of limiting such content.

“We've taken a pretty hard line on freedom of speech, supporting our users' creation, whatever that looks like, and it's just not something we want to police.... I don't want to have to go in there to draw the line between this photo and this behind-the-scenes photo of Lady Gaga and, like, her nip.” -- David Karp

Yahoo CEO Marissa Mayer noted that the content on the site might prove more challenging for advertisers, but promised that they would employ “good tools for targeting” to help advertisers avoid having their brands appear next to adult content. However, she still supported allowing Tumblr to continue hosting such content.

“I think the richness and breadth of content available on Tumblr—even though it may not be as brand-safe as what’s on our site—is what’s really exciting and allows us to reach even more users.” — Marissa Mayer

A key part of how Tumblr managed this at the time was allowing its users to tag their content in a way that would indicate to others if there was adult content, while letting the users themselves set preferences regarding their own interest in avoiding such content. Tumbr’s terms of service at the time explained how this worked:

Tumblr is home to millions of readers and bloggers from a variety of locations, cultures, and backgrounds with different points of view concerning adult-oriented content. If you regularly post sexual or adult-oriented content, respect the choices of people in our community who would rather not see such content by flagging your blog (which you can do from the Settings page of each blog) as Not Suitable for Work ("NSFW"). This action does not prevent you and your readers from using any of Tumblr's social features, but rather allows Tumblr users who don't want to see NSFW content to avoid seeing it. — Tumblr’s 2012 Terms of Service

Notably, those same terms did ban “sexually explicit videos” with a somewhat explicit reason for that ban: “We're not in the business of profiting from adult-oriented videos and hosting this stuff is fucking expensive.”

Around the time of the Yahoo purchase, however, users began noticing a change. As shared in Tarleton Gillespie’s book “Custodians of the Internet”:

In May 2013, some Tumblr users noticed that blogs rated “adult” were no longer findable through the major search engines. A month later, Tumblr began using the ratings to selectively exclude posts from its own search tool. Posts from “NSFW” or “adult” blogs no longer appeared in Tumblr’s search results, even if the post itself was not explicit, and regardless of whether the search was explicit. Actually, it was even more complicated than that: if the searcher already followed the explicit blog, that blog’s posts would appear— if it was “NSFW.” If it was “adult,” the more explicit rating, those posts would not appear in the search results, even if the searcher already followed that blog. — Tarleton Gillespie

The end result of this was widespread confusion among Tumblr’s users and fans:

“Clear? No? It was an intricate and confusing arrangement, one that users had a hard time following and the company had a hard time explaining. The principle behind this intricate policy is not an unreasonable one: let users continue to post explicit pornography, while using the self- rating to shield users who do not want to encounter it. But enacting this principle meant codifying it in a series of if/then conditions that could be automated in Tumblr’s search algorithm. And what the policy meant in practice was that while an explicit blog’s existing followers could more or less still get to it, it would now be much more difficult for anyone new ever to find it, given that its posts would not appear in any search results.

In addition, there were other assumptions hiding in the new policy: that the rules should be different for mobile users than for users on their computers; that “logged-out” users (which includes users who have not yet signed up for Tumblr) should not encounter explicit blogs at all; and that explicit Tumblr blogs shouldn’t be appearing in search results on Google or Bing—or Yahoo. These represent somewhat different priorities, but get folded in with Tumblr’s apparent concern for balancing the right to share pornography and the right not to encounter it if you choose not to.” -- Tarleton Gillespie

Company Considerations:

• How important to Tumblr is the culture and community that has built up around those who share adult content on the site? How does that play into product decisions regarding discovery of new content on the site?
• How practical is user tagging for dealing with adult content on the site? How will the site handle it if there is a significant disagreement, such as someone posting content they insist is not adult content, but others feel is?
• What sorts of technical/algorithmic rules should be put in place to deal with such adult content on the site? How effective is it to remove such results from search? What are the consequences of removing NSFW results from search?

Issue Considerations:

• Adult content raises different kinds of challenges for social media websites. What kinds of policies should sites consider regarding such content, and how will it impact their userbase and communities?
• Tumblr argued that photos including nudity were different than “sexually explicit videos.” Are these distinctions meaningful in a way that can be explained to a team of content moderators?

Resolution: For many years after this, Tumblr continued to allow adult content. In 2017, the company made one major change, rolling out its safe mode option which would not just remove “NSFW” tagged content from search, but also from user feeds. So even if a user followed a certain blog, if they were in “safe mode” they would no longer see content tagged as NSFW.

In 2017, Verizon purchased Yahoo, including Tumblr. A year later, it became apparent that Verizon did not take the same view as Yahoo had regarding allowing Tumblr to continue hosting adult content. In December of 2018, it was announced that Tumblr would be banning adult content on its servers.

Many people worried about what this would do to the site and the communities that grew up around it. Others pointed out that the ban would negatively impact queer and sex-positive communities. The end result was that traffic to the site diminished noticeably in the following months. In August of 2019, Verizon sold Tumblr to blogging company Automattic for just a few million dollars, well below the $1.1 billion Yahoo had paid for it in 2013.

Originally posted to the Trust & Safety Foundation website.

Filed Under: adult content, content moderation
Companies: tumblr, verizon, yahoo

Content Moderation Case Study: Twitter's Self-Deleting Tweets Feature Creates New Moderation Problems

from the content-moderation-at-the-fleeting-level dept

Summary: In its 15 years as a micro-blogging service, Twitter has given users more characters per tweet, reaction GIFs, multiple UI options, and the occasional random resorting of their timelines.

The most recent offering was to give users the option to create posts designed to be swept away by the digital sands of time. Early in 2020, Twitter announced it would be rolling out "Fleets" — self-deleting tweets with a lifespan of only 24 hours. This put Twitter on equal footing with Instagram's "Stories" feature, which allows users to post content with a built-in expiration date.

In the initial, limited rollout of Fleets, Twitter reported that the feature showed advantages over the platform's standard offering. Twitter Comms tweeted that initial testing looked promising, stating that it was seeing "less abuse with Fleets" with only a "small percentage" of Fleets being reported each day.

Whether this early indicator was a symptom of the limited rollout or users viewing self-deleting abuse as a problem that solves itself, the wider rollout wasn't nearly as easy as earlier indicators nor was it relatively abuse free. Fleet’s full debut arrived in the wake of an incredibly contentious U.S. presidential election — one marred by election interference accusations and a constant barrage of misinformation. The full rollout also came after nearly a year of a worldwide pandemic, which resulted in a constant flow of misinformation across multiple social media platforms globally.

While amplification of misinformation contained in Fleets was somewhat tempered by their innate ephemerality, as well as very limited interaction options, it seemed unclear how — or how well — Twitter was handling moderating misinformation spread by the new communication option. Extremism researcher Marc-Andre Argentino was able to send out a series of "fleets" containing misinformation and banned URLS, noting that Twitter only flagged one that asserted a link between the virus and cell phone towers.

Samantha Cole reported other Fleet moderation issues. Writing for Motherboard, Cole noted that apparent glitches were allowing users to see Fleets from people they had blocked, as well as Fleets from people who had blocked them. Failing to maintain settings that users set up to block or mute others created more avenues for abuse. Cole also pointed out that users weren't being notified when their tweets were added to Fleets, providing abusive users with another option to harass while the targets of abuse remain unaware.

Company Considerations:

• How can Twitter prevent new features from duplicating existing moderation problems?
• How can companies test a feature’s initial rollout to better detect possible abuses, and therefore resulting in less moderation needs in the wider rollout? 
• How does ephemeral content affect moderation efforts and moderation response time? 
• If issues remain unsolved or poorly-addressed, who has the power to shut down or temporarily disable a new feature? 
• How much time should moderation teams be given to adjust to new responsibilities and new inputs when a new feature is rolled out? What metrics would be useful to determine whether moderation responses are successfully addressing new abuses and problems?

Issue Considerations:

• What processes should companies have in place to mitigate damage if a feature doesn't perform in the expected way and/or creates unforeseen problems?
• Does "fleeting" content have the potential to cause moderators to view abusive posts as problems that will solve themselves? How can this mindset be discouraged or counteracted?

Resolution: Twitter's immediate response to the issues during the full rollout was to temporarily slow the deployment of the feature to users. While the issues that impacted moderation never really dissipated, the feature itself did. Twitter noted that Fleets did not have the uptake it expected. Although Fleets was supposed to encourage more engagement from Twitter users who lurked more than posted, observers noted that the feature appeared to be used mostly by users who were already heavily-engaged with the platform.

With the feature never being much more than a novelty for Twitter die-hards, Twitter killed off the feature on August 3, 2021, taking with it the moderation problems the self-killing Fleets had created.

Originally posted to the Trust & Safety Foundation website

Filed Under: content moderation, fleets, transitory content
Companies: twitter

Content Moderation Case Studies: Coca Cola Realizes Custom Bottle Labels Involve Moderation Issues (2021)

from the content-moderation-is-everywhere dept

Summary: Content moderation questions can come from all sorts of unexpected places — including custom soda bottle labels. Over the years, Coca Cola has experimented with a variety of different promotional efforts regarding more customized cans and bottles, and not without controversy. Back in 2013, as part of its “Share a Coke” campaign, the company offered bottles with common first names on the labels, which angered some who felt left out. In Israel, for example, people noticed that Arabic names were left off the list, although Coca Cola’s Swedish operation said that this decision was made after the local Muslim community asked not to have their names included.

This controversy was only the preamble to a bigger one in the summer of 2021, when Coca Cola began its latest version of the “Share a Coke” effort — this time allowing anyone to create a completely custom label up to 36 characters long. Opening up custom labels immediately raised content moderation questions.

Some people quickly noticed some surprising terms and phrases that were blocked (such as “Black Lives Matter”) while others that were surprisingly not blocked (like “Nazis”).

As CNN reporter Alexis Benveniste noted, it was easy to get offensive terms through the blocks (often with a few tweaks), and there were some eye-opening contrasts:

For example, "Black Lives Matter," is blocked. But "White Lives Matter" isn't. Coke included a special rainbow label for pride month, but you can't write "Gay Pride" on the bottle. However, you can write "I hate gays." "Hitler" and "Nazi" are banned, but users can customize bottles with the phrases, "I am Hitler" or "I am a Nazi." — Alexis Benveniste

The fact that “I am Hitler” was allowed while “Hitler” by itself was not suggests that Coca Cola was using a filter that included blocking entire phrases, rather than just a list of words (enabling simple adjustments to get through — which might explain why “Nazi” is blocked but “Nazis” apparently was not).

Coca Cola insisted that the automated blocks in its web tool were not the only system of review, just the first filter before being passed on to production, and that “actual bottles are not made with words that are inconsistent with the program's intent.”

Company Considerations:

• What kinds of tools, systems, staff, and processes should be put in place to deal with potential “abuse” of a custom labels program?
• How should the “intent” of the program be communicated to consumers who want their own bottles, but may ask for problematic content on the labels?
• How could the website more clearly inform consumers that the final text will still be reviewed by staff before production, so as not to let the public assume that if a word or phrase was not rejected in the web form, it will be printed?

Issue Considerations:

• Customization systems are often put in place because they are considered fun and engaging, and a way for consumers to connect with a brand. How should companies weigh such benefits against the likelihood of abuse?
• How should companies wishing to use these types of customization options consider the potential consumer backlash to what those users believe is both over-moderation and under-moderation?
• As it becomes easier to mass produce customized products, how should companies set up campaigns to minimize possible abuses, while balancing the backlash if they disallow certain words or phrases that are important to certain groups?

Resolution: Coca Cola admitted to CNN that the process is constantly being adjusted. “We're continuously refining and improving our Share A Coke personalization tool to ensure it is used only for its intended purpose." The company also noted that it added language to the preview screen to say that “proposed language may require further review.” The company did not explain why terms like “Black Lives Matter” were not approved.

Originally posted to the Trust & Safety Foundation website.

Filed Under: content moderation, custom content, customization, filter lists, soda bottles
Companies: coca cola

Content Moderation Case Study: Bing Search Results Erases Images Of 'Tank Man' On Anniversary Of Tiananmen Square Crackdown (2021)

from the tank-man's-gone-missing dept

Summary: On the 32nd anniversary of the Tiananmen Square protests, internet users noticed Microsoft's Bing search engine was producing some interesting results. Or, rather, it wasn't producing expected search results for some possibly interesting reasons.

Users searching for the most iconic image of the protests — that of the unidentified person known only as "Tank Man" — were coming up empty. It appeared that Microsoft's search engine was blocking results for an image that often serves as shorthand for rebellion against the Chinese government.

As was reported by several web users, followed by several news outlets, the apparent blocking of search results could be observed in both the United States and the United Kingdom, leaving users with the impression the Chinese government had pressured Microsoft to moderate search results for "tank man" in hopes of reducing any remembrance of the Tiananmen Square Massacre, which resulted in the deaths of 2,500-3,500 protesters.

The apparent censorship was blamed on Microsoft's close relationship with the Chinese government, which allowed its search engine to be accessed by Chinese residents in exchange for complying with government censorship requests.

This led to Microsoft being criticized by prominent politicians for apparently allowing the Chinese government to dictate what users around the world could access in relation to the Tiananmen Square protests.

Company Considerations:

• When complying to one government's interests, how can Microsoft ensure these considerations don't affect users located elsewhere in the world?
• How can compliance departments assist in handling edge cases and/or overly-broad moderation demands? 
• What are the tradeoffs for content providers when weighing offended users against offended governments?
• What ethical concerns should be taken into consideration when entering markets controlled by oppressive governments? Is there a line companies should not be willing to cross when seeking to expand their user base so as not to offend or alienate the user base they already have? 

Issue Considerations:

• What options can companies pursue when seeking to do business in countries with historically-censorial/repressive regimes to prevent collateral moderation damage to users located elsewhere?
• What sort of cost/benefit analysis, both human and fiscal, should take place before offering a product in countries with known human rights issues?

Resolution: Shortly after the apparent censorship of the iconic "Tank Man" image was reported, Microsoft claimed the very timely removal of relevant search results was the byproduct of "accidental human error." 

However, the company refused to offer any additional explanation. And, while searching the term "Tank Man" produced search results in Bing, it did not generate the expected results.

Image via The Verge

Several hours after the first "fix," things returned to normal, with "Tank Man" searches bringing up the actual Tank Man, rather than just tanks or tanks with men near or on the tanks.

Image via Twitter user Steven F

More clarification and comment was sought, but Microsoft apparently had nothing more to say about this "human error" and its conspicuous timing. Nor did it offer any details on whether or not this "human error" originated with its Beijing team. It also didn't explain why the first fix resulted in images very few people would associate with the term "Tank Man."

Originally posted to the Trust & Safety Foundation website.

Filed Under: bing, china, content moderation, search, search results, tank man, tiananmen square
Companies: microsoft

Content Moderation Case Study: Twitter Removes 'Verified' Badge In Response To Policy Violations (2017)

from the verified-or-a-stamp-of-approval? dept

Summary: Many social networks have enabled users to use a pseudonym as their identity on that network. Since users could use whatever name they wanted, they could pretend to be someone else, creating certain challenges for those platforms. For example, for sites that allowed such pseudonyms, how would they identify who the actual person was and who was merely an impostor? Some companies, such as Facebook, went the route of requiring users to use their real names. Twitter went another way, allowing pseudonyms.

But what can a company do when there are multiple accounts of the same, often famous, person?

In 2009, Twitter began experimenting with a program to “verify” celebrities.

The initial intent of this program was to identify which Twitter account actually belongs to the person or organization of that Twitter handle (or name). Twitter’s announcement of this feature explains it in straightforward terms:

With this feature, you can easily see which accounts we know are 'real' and authentic. That means we've been in contact with the person or entity the account is representing and verified that it is approved. (This does not mean we have verified who, exactly, is writing the tweets.)

This also does not mean that accounts without the 'Verified Account' badge are fake. The vast majority of accounts on the system are not impersonators, and we don't have the ability to check 100% of them. For now, we've only verified a handful of accounts to help with cases of mistaken identity or impersonation.

From the start, Twitter denoted “verified” accounts with a, now industry-standard, “blue checkmark.” In the initial announcement, Twitter noted that this was experimental, and the company did not have time to verify everyone who wanted to be verified. It was not until 2016 that Twitter first opened up an application process for anyone to get verified.

A year later, in late 2017, the company closed the application process, noting that people were interpreting “verification” as a stamp of endorsement, which it had not intended. Recognizing this unintended perception, Twitter began removing verification checkmarks from accounts that violated certain policies, starting with high-profile white supremacists.

While this policy received some criticism for “blurring” the line between speakers and speech, it was a recognition of the concerns about how the checkmark was seen as an “endorsement” of someone whose views and actions (even those off of Twitter) were not those Twitter wished to endorse. In that way, the removal of the verification became a content moderation tool for a type of subtle negative endorsement.

Even though those users were “verified” as authentic, Twitter recognized that being verified was a privilege and that removing it was a tool in the content moderation toolbox. Rather than suspending or terminating accounts, the company said that it would also consider removing the verification on accounts that violated its new hateful conduct and abusive behavior policies.

Company Considerations:

• What is the purpose of a verification system on social media? Should it just be to prove that a person is who they say they are, or should it also signal some kind of endorsement? How should the company develop a verification system to match that purpose? 
• If the public views verification as a form of endorsement, how important is it for a company to reflect that in its verification program? Are there any realistic ways to have the program not be considered an endorsement?
• Under what conditions does it make sense to use removal of verification as a content moderation tool? Is removing verification an effective content moderation tool? If not, are there ways to make it more effective?

Issue Considerations:

• What are the consequences of using the verification (and de-verification) process as a content moderation tool to “punish” rule violators?
• What are both the risks and benefits of embracing verification as a form of endorsement?
• Are there other subtle forms of content moderation similar to the removal of privileges like the blue checkmark, and how effective can they be?

Resolution: It took many years until Twitter reopened its verification system, and then it did so only in a very limited manner. The system has already ran into problems, as journalists discovered multiple fake accounts that were verified.

However, a larger concern over the new verification rules is that it allows for significant subjective decision-making by the company over how the rules are applied. Activist Albert Fox Cahn explained how the new program is making it “too easy” for journalists to get verified but “too difficult” for activists, showing the challenging nature of any such program.

“When Angela Lang, founder and executive director of the Milwaukee-based civic engagement group BLOC, decided to get a checkmark, she thought, ‘I've done enough. Let’s check out how to be verified.’ Despite Lang and BLOC’s nationally recognized work on Black civic engagement, she found herself shut out. When Detroit-based activist and Data 4 Black Lives national organizing director Tawana Petty applied, her request was promptly rejected. Posting on the platform that refused to verify her, Petty said, ‘Unbelievable that creating a popular hashtag would even be a requirement. This process totally misses the point of why so many of us want to be verified.’ Petty told me, ‘I still live with the anxiety that my page might be duplicated and my contacts will be spammed.’ Previously, she was forced to shut down pages on other social media platforms to protect loved ones from this sort of abuse.

“According to anti-racist economist Kim Crayton, verification is important because ‘that blue check automatically means that what you have to say is of value, and without it, particularly if you’re on the front lines, particularly if you’re a Black woman, you’re questioned.’ As Lang says, ‘Having that verification is another way of elevating those voices as trusted messengers.’ According to Virginia Eubanks, an associate professor of political science at the University at Albany, SUNY, and author of Automating Inequality, ‘The blue check isn't about social affirmation, it’s a safety issue. Someone cloning my account could leave my family or friends vulnerable and could leave potential sources open to manipulation.’” — Albert Fox Cahn

Originally published to the Trust & Safety Foundation website.

Filed Under: case study, content moderation, endorsement, verification badges
Companies: twitter

Content Moderation Case Study: Spam "Hacks" in Among Us (2020)

from the very-sus dept

Summary: From August to October of 2020, as the COVID-19 pandemic had no end in sight and plenty of people were still stuck at home, on lockdown, unable to gather with others, the video game Among Us became incredibly popular as a kind of party game when there were no parties. The game had already been out for a while, but for unclear reasons it became the go-to game during the pandemic. It was so popular that the company behind it, InnerSloth, cancelled its plans for a sequel, promising instead to focus on fixing up the existing game and dealing with some of the bugs that were popping up from such widespread usage.

Among the bugs that InnerSloth had to deal with was the ability to hack the game with various apps and tools that allowed users to possess more powers in the game than they should be able to have.

This came to a head in late October of 2020, when the game was apparently overrun by spam promoting a YouTuber named “Eris Loris.” Some of the spam had political messaging, but all of it told people to subscribe to that user’s YouTube account. Sometimes it came with vaguely worded threats of hacking if you didn’t subscribe. Other times it just told people to subscribe.

While this attack was variously described as both a “hack” and a “spammer,” it appears that it was a combination of both at work. The end result was spamming players in the game and making it impossible to keep playing, but it was also carried out via a hack that filled the game with bots designed to spread the message. The person who goes by the name Eris Loris told the website Kotaku that he did it because he thought it was funny:

“I was curious to see what would happen, and personally I found it funny,” Loris told Kotaku in a DM. “The anger and hatred is the part that makes it funny. If you care about a game and are willing to go and spam dislike some random dude on the internet because you can’t play it for 3 minutes, it’s stupid.” — “Eris Loris” to Kotaku reporter Nathan Grayson

InnerSloth admitted that it was aware of the problem and asked players to “bare with us” [sic] and only play private games or with players they knew and trusted until updates were made to the server. A developer for the game separately warned users that he was rolling out changes using a “faster method than I’ve done before” and, as such, that things might break.

Company Considerations:

• How much effort should be put towards preventative measures to try to block spamming, even before an app or service becomes wildly popular?
• At what level does spamming reach a point that it is critical to change the code of a game, perhaps even using “faster” and less reliable methods to combat the spamming than would normally be used?
• How do you balance resource allocations between having engineers improving the product and adding new features as compared to fighting back against malicious actors?

Issue Considerations:

• When something becomes popular, there are always those with nefarious intentions who want to take advantage of the platform's popularity. Should companies proactively prepare for the unintended consequences of success? What can companies put in place to anticipate the actions of bad actors?
• Spammers and hackers sometimes go hand in hand with popular games and platforms. What are other risks (beyond just losing players/customers) if companies allow, or are slow at the removal, of those bad actors from the platform?
• Many developers leave platforms somewhat open to encourage third party developers to build on additional tools and services that make a game or service more useful. How does a developer determine the trade-offs between an open system to promote innovation and someone abusing that openness?

Resolution: The rapid updates Among Us developers made to the Among Us servers appeared to do the trick, and the Eris Loris spam quickly diminished soon after. There were some questions about whether or not there would be legal consequences for whoever was behind the attacks, but to date, nothing has happened.

There still remain a number of Among Us hacks out there, and some people have attempted to follow in the footsteps of Eris Loris — including someone going by the name Sire Soril (Eris Loris backwards) — but it appears that none of these have had much success at all, suggesting that InnerSloth’s initial fix was pretty successful in limiting the kinds of attacks that overwhelmed the system in October of 2020.

Originally posted to the Trust & Safety Foundation website.

Filed Under: among us, content moderation, eris loris, games, hacking, spam
Companies: innersloth

Content Moderation Case Study: YouTube Deals With Disturbing Content Disguised As Videos For Kids (2017)

from the finding-the-monsters-among-us dept

Summary: YouTube offers an endless stream of videos that cater to the preferences of users, no matter their age and has become a go-to content provider for kids and their parents. The market for kid-oriented videos remains wide-open, with new competitors surfacing daily and utilizing repetition, familiarity, and strings of keywords to get their videos in front of kids willing to spend hours clicking on whatever thumbnails pique their interest, and YouTube is leading this market.

Taking advantage of the low expectations of extremely youthful viewers, YouTube videos for kids are filled with low-effort, low-cost content - videos that use familiar songs, bright colors, and pop culture fixtures to attract and hold the attention of children.

Most of this content is innocuous. But a much darker strain of content was exposed by amateur internet sleuths, which was swiftly dubbed "Elsagate," borrowing the name of the main character of Disney's massively popular animated hit, Frozen. At the r/ElsaGate subreddit, redditors tracked down videos aimed at children that contained adult themes, sexual activity, or other non-kid-friendly content.

Among the decidedly not-safe-for-kids subject matter listed by r/ElsaGate are injections, gore, suicide, pregnancy, BDSM, assault, rape, murder, cannibalism, and use of alcohol. Most of these acts were performed by animated characters (or actors dressed as the characters), including the titular Elsa as well as Spiderman, Peppa Pig, Paw Patrol, and Mickey Mouse. According to parents, users, and members of the r/Elsagate subreddit, some of this content could be accessed via the YouTube Kids app — a kid-oriented version of YouTube subject to stricter controls and home to curated content meant to steer child users clear of adult subject matter.

Further attention was drawn to the issue by James Bridle's post on the subject, entitled "Something is Wrong on the Internet." The post — preceded by numerous content warnings — detailed the considerable amount of disturbing content that was easily finding its way to youthful viewers, mainly thanks to its kid-friendly tags and innocuous thumbnails.

The end result, according to Bridle, was nothing short of horrific:

“To expose children to this content is abuse. We’re not talking about the debatable but undoubtedly real effects of film or videogame violence on teenagers, or the effects of pornography or extreme images on young minds, which were alluded to in my opening description of my own teenage internet use. Those are important debates, but they’re not what is being discussed here. What we’re talking about is very young children, effectively from birth, being deliberately targeted with content which will traumatise and disturb them, via networks which are extremely vulnerable to exactly this form of abuse. It’s not about trolls, but about a kind of violence inherent in the combination of digital systems and capitalist incentives. It’s down to that level of the metal.” — James Bridle

"Elsagate" received more mainstream coverage as well. A New York Times article on the subject wondered what had happened and suggested the videos had eluded YouTube's algorithms that were meant to ensure content that made its way to its Kids channel was actually appropriate for children. YouTube's response when asked for comment was that this content was the "extreme needle in the haystack," perhaps, an immeasurably small percentage of the total amount of content available on YouTube Kids. Needless to say, this answer did not make critics happy, and many suggested the online content giant rely less on automated moderation when dealing with content targeting kids.

Company Considerations:

• How should content review and moderation be different for content targeting younger YouTube users?
• How could a verification process be deployed to vet users creating content for children?
• What processes can be used to make it easier to find and remove/restrict content that appears to be kid-friendly but is actually filled with adult content?
• When content like what was described in the case study does get through the moderation process, what can be done to restore the trust of users, especially those with younger children?

Issue Considerations:

• Should a product targeting children be siloed off from the main product to ensure the integrity of the content, as well as make it easier to manage moderation issues?
• Does creating a product specifically for children increase the chance of direct regulation or intervention by government entities? If so, how can a company prepare itself for this inevitability?
• If creating a “restricted” product for children, should it require all content be fully and thoroughly vetted? If so, would that become prohibitively costly, making it significantly less likely that companies will create products for children? Is there a way to balance those things?

Resolution: Immediately following these reports, YouTube purged content from YouTube Kids that did not meet its standards. It delisted videos and issued new guidelines for contributors. It added a large number of new human moderators, bringing its total of moderators to 10,000. YouTube also removed the extremely popular "Toy Freaks" channel, which users had suggested contained child abuse, after investigating its content.

YouTube wasn't the only entity to act after the worldwide exposure of "Elsagate" videos. Many of these videos originated in China, prompting the Chinese government to block certain search keywords to limit local access to the disturbing content, as well as shuttering at least one company involved in the creation of these videos.

Originally posted to the Trust & Safety Foundation website.

Filed Under: content moderation, elsagate, kids
Companies: youtube