Content Moderation Case Study: Nextdoor Faces Criticism From Volunteer Moderators Over Its Support Of Black Lives Matter (June 2020)

from the everything-is-politics dept

Summary: Nextdoor is the local “neighborhood-focused” social network, which allows for hyper-local communication within a neighborhood. The system works by having volunteer moderators from each neighborhood, known as “leads.” For many years, Nextdoor has faced accusations of perpetuating racial stereotyping from people using the platform to report sightings of black men and women in their neighborhood as somehow “suspicious.”

With the various Black Lives Matter protests spreading around the country following the killing of George Floyd in Minnesota, many companies have put out messages of support for the Black Lives Matter movement, including Nextdoor, which put up a short blog post entitled Black Lives Matter, with a few links to various groups supporting the movement.

This happened around the same time that the site started facing criticism because users posting support of the Black Lives Matter movement were finding their own posts being removed as leads were claiming that posts about the protests violated guidelines not to discuss “national and state” political issues (even when the posts were about local protests).

Meanwhile, many of the “leads” were using their own forum to complain about the company’s public support for Black Lives Matter at the same time that they believed discussing such issues on the platform violated rules. The ensuing discussion (which in many ways mimicked the wider national discussion) highlighted how frequently local leads are bringing their own political viewpoints into their moderation decisions.

When the company also added posts to local Nextdoor communities that highlighted black-owned businesses, as part of its support for Black Lives Matter, it again angered some leads who felt that such posts violated the rules they had been told to enforce.

Decisions to be made by Nextdoor:

• When there are national conversations around movements like Black Lives Matter, when is it appropriate to take a public stand? How will that stand be perceived by users and by local moderators?
• If the company is taking a stand on an issue like Black Lives Matter, should it then make it clear that related content should be kept on the platform -- even if some moderators believe it violates other guidelines?
• How much leeway and power should local, volunteer moderators have regarding what content is on the platform?
• How much communication and transparency should there be with those local moderators?
• How involved should the company get with regards to implicit or unconscious bias that may come from non-employee, volunteer moderators?
• Is it feasible to have a rule that suggests that local communities should not be a platform for discussing state or national political issues?  How does that rule play out when those “national or state” issues also involve local activism?

Questions and policy implications to consider:

• When issues of national importance, such as civil rights, come to the forefront of the public discussion, there is often the likelihood of them becoming politically divisive. When is it important to take a stand despite this, and how should any messaging be handled -- especially in cases where some staff or volunteers may feel otherwise?
• Issues of race are particularly controversial to some, and yet vitally important. How should companies handle these questions and debates?
• Using volunteer moderators to help moderate targeted niche communities has obvious benefits, but how might historical bias and prejudice manifest itself in doing so?

Resolution: Nextdoor has continued to support the Black Lives Matter movement, and Gordon Strause, the company’s director of community, went onto the forum where some leads were complaining to explain the company’s position and why they were supporting Black Lives Matter, and to push back against those who argued that the movement itself was discriminatory, while also highlighting how there were a variety of perspectives, and there was value in learning about other viewpoints:

In an attempt to quell the furor, Gordon Strause, the company’s director of community, wrote on the leads forum on Monday from his “own perspective” and not “on behalf of Nextdoor.” Noting that “it’s of course absolutely true all live [sic] matters, whether they are black, white, brown, blue, or any other color,” he explained his views on Black Lives Matter.

“The goal of the BLM movement, at least as I understand it, is simply to make the point that black lives matter as much as any other lives but too often in America that isn’t actually what happens in practice and this dynamic needs to change,” he wrote.

“While no one that I know or respect believes that looting helps anything, there are folks that I respect (including people in my own family) who believe that riots may be a necessary step to help the country finally understand the scale of injustice that has been happening," he wrote, "while other folks I respect believe that the riots will be counterproductive and will only undermine the goals they are meant to achieve.” Strause then went on to recommend a book from psychologist Jonathan Haidt and urged leads “to listen and not to judge.”

“While Nextdoor is generally not the place for discussions of national issues, I think it’s going to [sic] hard to restrain those discussions in the coming days without being perceived as taking sides. So rather than trying to do so, I would recommend that Leads instead focus on a different goal: keeping the discussions as civil and issue focused (rather than personality focused) as possible,” he wrote.

Filed Under: black lives matter, case studies, content moderation, local, politics, racism, social media
Companies: nextdoor

Content Moderation Case Study: Amazon's Attempt To Remove 'Sock Puppet' Reviews Results In The Deletion Of Legitimate Reviews (November 2012)

from the real-and-fake dept

Summary: As is the case on any site where consumer products are sold, there's always the chance review scores will be artificially inflated by bogus reviews using fake accounts, often described as "sock puppets."

Legitimate reviews are organic, prompted by a buyer's experience with a product. "Sock puppets," on the other hand, are bogus accounts created for the purpose of inflating the number of positive (or -- in the case of a competitor -- negative) reviews for a seller's product. Often, they're created by the seller themself. Sometimes these faux reviews are purchased from third parties. "Sock puppet" activity isn't limited to product reviews. The same behavior has been detected in comment threads and on social media platforms.

In 2012 -- apparently in response to "sock puppet" activity, some of it linked to a prominent author -- Amazon engaged in a mass deletion of suspected bogus activity. Unfortunately, this moderation effort also removed hundreds of legitimate book reviews written by authors and book readers.

In response to authors' complaints that their legitimate reviews had been removed (along with apparently legitimate reviews of their own books), Amazon pointed to its review guidelines, claiming they forbade authors from reviewing other authors' books.

We do not allow reviews on behalf of a person or company with a financial interest in the product or a directly competing product. This includes authors, artists, publishers, manufacturers, or third-party merchants selling the product. As a result, we've removed your reviews for this title. Any further violations of our posted Guidelines may result in the removal of this item from our website.

Multiple authors sought to have their legitimate reviews reinstated (including reviews of their books written by readers), but Amazon refused, insisting that authors reviewing other authors' books constituted a violation of its review guidelines, even if authors had no financial interest in the books they were reviewing.

Amazon's handling of reviews in response to sock puppet activity continues to be criticized periodically, most recently over the mass removal of one-star reviews for Hillary Clinton's 2017 book about her presidential election run.

Decisions to be made by Amazon:

• What characteristics do “sock puppet” reviews have, that make them distinct from legitimate reviews?
• Do more steps need to be added to the process of verifying reviewers?
• When targeting sock puppet activity, are options considered that might reduce the chance of negatively affecting legitimate reviews?
• Would more flexibility in moderation decisions help or harm efforts targeting abusers of the review system?
• Is the loss of goodwill towards the company by sellers an acceptable tradeoff for moderation efforts that remove possibly legitimate reviews of their products?
• Can moderation efforts be handled with more human interaction to reduce the number of legitimate reviews inadvertently targeted?

Questions and policy implications to consider:

• As the number of vendors and products continues to expand, how reasonable is it to expect reviewers to avoid violating the rule forbidding reviews of products by someone offering a competing product?
• How much moderation should be left to automatic mechanisms when dealing with suspected sock puppet activity?
• Does the inevitable collateral damage of these efforts raise or lower the legitimacy of the remaining reviews in the eyes of potential customers?
• Would more transparency on review moderation efforts lead to more or less abuse of the review system?
• Do mishandled moderation efforts harm buyers or sellers more? Which harm is more acceptable?

Resolution: Amazon reacted to news reports about sock puppet activity involving major authors by engaging in mass removals of anything that appeared questionable to moderators. Legitimate reviews/reviewers were caught up in the sweep, resulting in several authors publicly criticizing the company for not being more careful with its moderation efforts.

Filed Under: content moderation, reviews, sock puppets
Companies: amazon

Content Moderation Case Study: Twitter Acts To Remove Accounts For Violating The Terms Of Service By Buying/Selling Engagement (March 2018)

from the fake-followers dept

Summary: After an investigation by BuzzFeed uncovered several accounts trafficking in paid access to "decks" -- Tweetdeck accounts from which buyers could mass-retweet their own tweets to make them go "viral" -- Twitter acted to shut down the abusive accounts.

Most of the accounts were run by teens who leveraged the tools provided by Twitter-owned Tweetdeck to provide mass exposure to tweets for paying customers. Until Twitter acted, users who saw their tweets go viral under other users' names tried to police the problem by naming paid accounts and putting them on blocklists.

Twitter's Rules expressly forbid users from "artificially inflating account interactions”. But most accounts were apparently removed under Twitter's anti-spam policy -- one it beefed up after BuzzFeed published its investigation. The biggest change was the removal of the ability to simultaneously retweet tweets from several different accounts, rendering these "decks" built by "Tweetdeckers" mostly useless. Tweetdeckers responded by taking a manual approach to faux virality, sending direct messages requesting mutual retweets of posted content.

Unlike other corrective actions taken by Twitter in response to mass abuse, this cleanup process appears to have resulted in almost no collateral damage. Some users complained their follower counts had dropped, but this was likely the result of near-simultaneous moderation efforts targeting bot accounts.

Decisions to be made by Twitter:

• Do additional moderation efforts -- AI or otherwise -- need to be deployed to detect abuse of Twitter Rules?
• How often do these efforts mistakenly target legitimately "viral" content?
• Will altering Tweetdeck features harm users who aren't engaged in the buying and selling of "engagement?"
• Will power users or those seeking to abuse the rules move to other third-party offerings to avoid moderation efforts?
• Is there any way to neutralize "retweet for retweet" requests in direct messages without raising concerns about user privacy?

Questions and policy implications to consider:

• Does targeting spam more aggressively risk alienating advertisers who rely on repetitive/scheduled posts and active user engagement?
• Does spam (in whatever form -- including the manufactured virality seen above) still provide some value for Twitter as a company, considering it relies on active users and engagement to secure funding and/or sell ad space to companies?
• Do viral posts still add value for Twitter users, even if the source of the virality is illegitimate?
• Will increased moderation of spam reduce user engagement during events where advertising efforts and user engagement are routinely expected to increase (elections, sporting events, etc.)?  

Resolution: Twitter moved quickly to disable and delete accounts linked to the marketing of user engagement. It chose to use its anti-spam rules as justification for account removals, even though users were allegedly engaged in other violations of the terms of service. The buying and selling of Twitter followers -- along with retweets and likes -- continues to be a problem, but Twitter clearly has a toolset in place that is effective against the behavior seen here. Due to the reliance on spam rules, the alterations to Tweetdeck -- a favorite of Twitter power users -- appear to have done minimal damage to legitimate users who enjoy the advantages of this expanded product.

Filed Under: buying engagement, case study, content moderation, fake followers, tweetdeck
Companies: twitter

Creating Family Friendly Chat More Difficult Than Imagined (1996)

from the the-kids-will-find-a-way dept

Summary: Creating family friendly environments on the internet presents some interesting challenges that highlight the trade-offs in content moderation. One of the founders of Electric Communities, a pioneer in early online communities, gave a detailed overview of the difficulties in trying to build such a virtual world for Disney that included chat functionality. He described being brought in by Disney alongside someone from a kids’ software company, Knowledge Adventure, who had built an online community in the mid-90s called “KA-Worlds.” Disney wanted to build a virtual community space, HercWorld, to go along with the movie Hercules. After reviewing Disney’s requirements for an online community, they realized chat would be next to impossible:

Even in 1996, we knew that text-filters are no good at solving this kind of problem, so I asked for a clarification: "I’m confused. What standard should we use to decide if a message would be a problem for Disney?"

The response was one I will never forget: "Disney’s standard is quite clear:

No kid will be harassed, even if they don’t know they are being harassed."...

"OK. That means Chat Is Out of HercWorld, there is absolutely no way to meet your standard without exorbitantly high moderation costs," we replied.

One of their guys piped up: "Couldn’t we do some kind of sentence constructor, with a limited vocabulary of safe words?"

Before we could give it any serious thought, their own project manager interrupted, "That won’t work. We tried it for KA-Worlds."

"We spent several weeks building a UI that used pop-downs to construct sentences, and only had completely harmless words – the standard parts of grammar and safe nouns like cars, animals, and objects in the world."

"We thought it was the perfect solution, until we set our first 14-year old boy down in front of it. Within minutes he’d created the following sentence:

I want to stick my long-necked Giraffe up your fluffy white bunny.

In that initial 1996 project, chat was abandoned, but as they continued to develop HercWorld, they quickly realized that they still had to worry about chat, even without a chat feature:

It was standard fare: Collect stuff, ride stuff, shoot at stuff, build stuff… Oops, what was that last thing again?

"…kids can push around Roman columns and blocks to solve puzzles, make custom shapes, and buildings.", one of the designers said.

I couldn’t resist, "Umm. Doesn’t that violate the Disney standard? In this chat-free world, people will push the stones around until they spell Hi! or F-U-C-K or their phone number or whatever. You’ve just invented Block-Chat^TM. If you can put down objects, you’ve got chat. We learned this in Habitat and WorldsAway, where people would turn 100 Afro-Heads into a waterbed." We all laughed, but it was that kind of awkward laugh that you know means that we’re all probably just wasting our time.

Decisions for family-friendly community designers:

• Is there a way to build a chat that will not be abused by clever kids to reference forbidden content (e.g., swearing, innuendo, harassment, abuse)?
• Can you build a chat that does not require universal moderation and pre-approval of everything that users will say?
• Are there ways in which kids will still be to communicate with others even without an actual chat feature?
• How much of a “community” do you have with no chat or extremely limited chat?

Questions and policy implications to consider:

• Is it possible to create an online family friendly environment that will work?
  • If so how do you prevent abuse?
  • If not, how do you handle the fact that kids will get online whether they are allowed to or not?
• How do you incentivize companies to create spaces that actually remain as child-friendly as possible?
• If “the kids will always find a way” to get around limitations, does it make sense to hold the companies themselves responsible?
• Should family friendly environments require full-time monitoring, or pre-vetting of any usage?

Resolution: Disney eventually abandoned the idea of HercWorld due to all of the issues raised. However, the interview highlights the fact that they tried again a couple of years later, with an online chat where users could only pull from a pre-selected list of sentences, but it did not have much success:

"The Disney Standard" (now a legend amongst our employees) still held. No harassment, detectable or not, and no heavy moderation overhead.

Brian had an idea though: Fully pre-constructed sentences – dozens of them, easy to access. Specialize them for the activities available in the world. Vaz Douglas, our project manager working with Zoog, liked to call this feature "Chatless Chat." So, we built and launched it for them. Disney was still very tentative about the genre, so they only ran it for about six months; I doubt it was ever very popular.

The same interview notes that Disney tried once again in 2002 with a new world called “ToonTown”, with pulldown menus that allowed you to construct very narrowly tailored speech within the chat to try to avoid anything that violated the rules.

As the story goes, Disney still had problems with this. To make sure people were only communicating with people they knew in real life, one of the restrictions in this new world was that you had to have a secret code from any user you wished to chat with. The thinking was that parents would print these out for kids who could then share them with their friends in real life, and they could link up and “chat” in the online world.

And yet, once again, people figured out how to get around the restrictions:

Sure enough, chatters figured out a few simple protocols to pass their secret code, several variants are of this general form:

User A:"Please be my friend."
User A:"Come to my house?"
User B:"Okay."
A:[Move the picture frames on your wall, or move your furniture on the floor to make the number 4.]
A:"Okay"
B:[Writes down 4 on a piece of paper and says] "Okay."
A:[Move objects to make the next letter/number in the code] "Okay"
B:[Writes…] "Okay"
A:[Remove objects to represent a "space" in the code] "Okay"
[Repeat steps as needed, until…]
A:"Okay"
B:[Enters secret code into Toontown software.]
B:"There, that worked. Hi! I’m Jim 15/M/CA, what’s your A/S/L?"

Incredibly, there was an entire Wiki page on the Disney Online Worlds domain that included a variety of other descriptions on how to exchange your secret number within the game, even as users were not supposed to be doing so:

For example, let's say you have a secret code (1hh 5rj) which you would like to give to a toon named Bob.

First, you should make clear that you want to become their SF.
You: Please be my friend!
You: (random SF chat)
You: I can't understand you
You: Let's work on that
Bob: Yes
Now, start the secret.
You: (Jump 1 time and say OK. Jump 1 time because that is the first thing in your code. Say OK to confirm that was part of your secret.)
Bob: OK (Wait for this, as this means he has written down or otherwise recorded the 1)
You: Hello! OK (Say hello because the first letter of hello is h, which is the second part of your secret.)
Bob: OK (again, wait for confirmation)
Repeat above step, as you have the same letter for the third part of your secret.
Bob: OK (by now you should know to wait for this)
You: (Jump 5 times and say OK. Jump 5 times as this is the 4th part of your secret)
Bob: OK
You: Run! OK (The 5th part of your secret is r, and "Run!" starts with r)
Bob: OK
You: Jump! OK (Say this because j is the last part of your secret.)
Bob: OK
At this point, you have successfully transmitted the code to Bob.
Most likely, Bob will understand, and within seconds, you will be Secret Friends!

So even though Disney eventually did enable a very limited chat, with strict rules to keep people safe, it still left open many challenges for early trust & safety work.

Images from HabitChronicles

Filed Under: chat, content moderation, family friend
Companies: disney

Content Moderation Case Study: Social Media Services Respond When Recordings Of Shooting Are Uploaded By The Person Committing The Crimes (August 2015)

from the real-time-decision-making dept

Summary: The ability to instantly upload recordings and stream live video has made content moderation much more difficult. Uploads to YouTube have surpassed 500 hours of content every minute (as of May 2019), making any form of moderation inadequate.

The same goes for Twitter and Facebook. Facebook's user base exceeds two billion worldwide. Over 500 million tweets are posted to Twitter every day (as of May 2020). Algorithms and human moderators are incapable of catching everything that violates terms of service.

When the unthinkable happens -- as it did on August 26, 2015 -- these two social media services swiftly responded. But even their swift efforts weren't enough. The videos posted by Vester Lee Flanagan, a disgruntled former employee of CBS affiliate WDBJ in Virginia, showed him tracking down a WDBJ journalist and cameraman and shooting them both.

Both platforms removed the videos and deactivated Flanagan's accounts. Twitter's response took only minutes. But the spread of the videos had already begun, leaving moderators to try to track down duplicates before they could be seen and duplicated yet again. Many of these ended up on YouTube, where moderation efforts to contain the spread still left several reuploads intact. This was enough to instigate an FTC complaint against Google, filed by the father of the journalist killed by Flanagan. Google responded by stating it was still removing every copy of the videos it could locate, using a combination of AI and human moderation.

Users of Facebook and Twitter raised a novel complaint in the wake of the shooting, demanding "autoplay" be opt in -- rather than the default setting -- to prevent them from inadvertently viewing disturbing content.

Moderating content as it is created continues to pose challenges for Facebook, Twitter, and YouTube -- all of which allow live-streaming.

Decisions to be made by social media platforms:

• What efforts are being put in place to better handle moderation of streaming content?
• What efforts -- AI or otherwise -- are being deployed to potentially prevent the streaming of criminal acts? Which ones should we adopt?
• Once notified of objectionable content, how quickly should we respond?
• Are there different types of content that require different procedures for responding rapidly?
• What is the internal process for making moderation decisions on breaking news over streaming?
• While the benefits of auto-playing content are clear for social media platforms, is making this the default option a responsible decision -- not just for potentially-objectionable content but for users who may be using limited mobile data?

Questions and policy implications to consider:

• Given increasing Congressional pressure to moderate content (and similar pressure from other governments around the world), are platforms willing to "over-block" content to demonstrate their compliance with these competing demands? If so, will users seek out other services if their content is mistakenly blocked or deleted?
• If objectionable content is the source for additional news reporting or is of public interest (like depictions of violence against protesters, etc.), do these concerns override moderation decisions based on terms of service agreements?
• Does the immediate removal of criminal evidence from public view hamper criminal investigations? 
• Are all criminal acts of violence considered violations of content guidelines? What if the crime is being committed by government agents or law enforcement officers? What if the video is of a criminal act being performed by someone other than the person filming it? 

Resolution: All three platforms have made efforts to engage in faster, more accurate moderation of content. Live-streaming presents new challenges for all three platforms, which are being met with varying degrees of success. These three platforms are dealing with millions of uploads every day, ensuring objectionable content will still slip through and be seen by hundreds, if not thousands, of users before it can be targeted and taken down.

Content like this is a clear violation of terms of service agreements, making removal -- once notified and located -- straightforward. But being able to "see" it before dozens of users do remains a challenge.

Filed Under: case study, content moderation, live video, moderation, streaming, video streaming

Content Moderation Case Study: Facebook Nudity Filter Blocks Historical Content And News Reports About The Error (June 2020)

from the content-moderation-is-hard dept

Summary: Though social media networks take a wide variety of evolving approaches to their content policies, most have long maintained relatively broad bans on nudity and sexual content, and have heavily employed automated takedown systems to enforce these bans. Many controversies have arisen from this, leading some networks to adopt exceptions in recent years: Facebook now allows images of breastfeeding, child-birth, post-mastectomy scars, and post-gender-reassignment surgery photos, while Facebook-owned Instagram is still developing its exception for nudity in artistic works. However, even with exceptions in place, the heavy reliance on imperfect automated filters can obstruct political and social conversations, and block the sharing of relevant news reports.

One such instance occurred on June 11, 2020 following controversial comments by Australian Prime Minister Scott Morrison, who stated in a radio interview that “there was no slavery in Australia”. This sparked widespread condemnation and rebuttals from both the public and the press, pointing to the long history of enslavement of Australian Aboriginals and Pacific Islanders in the country. One Australian Facebook user posted a late 19th century photo from the state library of Western Australia, depicting Aboriginal men chained together by their necks, along with a statement:

Kidnapped, ripped from the arms of their loved ones and forced into back-breaking labour: The brutal reality of life as a Kanaka worker - but Scott Morrison claims ‘there was no slavery in Australia’

Facebook removed the post and image for violation of their policy against nudity, although no genitals are visible, and restricted the user’s account. The Guardian Australia contacted Facebook to determine if this decision was made in error and, the following day, Facebook restored the post and apologized to the user, explaining that it was an erroneous takedown caused by a false positive in the automated nudity filter. However, at the same time, Facebook continued to block posts that included The Guardian’s news story about the incident, which featured the same photo, and placed 30-day suspensions on some users who attempted to share it. Facebook’s community standards report shows that in the first three months of 2020, 39.5-million pieces of content were removed for nudity or sexual activity, over 99% of those takedowns were automated, 2.5-million appeals were filed, and 613,000 of the takedowns were reversed.

Decisions to be made by Facebook:

• Can nudity filters be improved to result in fewer false-positives, and/or is more human review required?
• For appeals of automated takedowns, what is an adequate review and response time?
• Should automated nudity filters be applied to the sharing of content from major journalistic sources such as The Guardian?
• Should questions about content takedowns from major news organizations be prioritized over those from regular users?
• Should 30-day suspensions and similar account restrictions be manually reviewed only if the user files an appeal?

Questions and policy implications to consider:

• Should automated filter systems be able to trigger account suspensions and restrictions without human review?
• Should content that has been restored in one instance be exempted from takedown, or flagged for automatic review, when it is shared again in future in different contexts?
• How quickly can erroneous takedowns be reviewed and reversed, and is this sufficient when dealing with current, rapidly-developing political conversations?
• Should nudity policies include exemptions for historical material, even when such material does include visible genitals, such as occurred in a related 2016 controversy over a Vietnam War photo?
  • Should these policies take into account the source of the content?
  • Should these policies take into account the associated messaging?

Resolution: Facebook’s restoration of the original post was undermined by its simultaneous blocking of The Guardian’s news reporting on the issue. After receiving dozens of reports from its readers that they were blocked from sharing the article and in some cases suspended for trying, The Guardian reached out to Facebook again and, by Monday, June 15, 2020, users were able to share the article without restriction. The difference in response times between the original incident and the blocking of posts is possibly attributable to the fact that the latter came to the fore on a weekend, but this meant that critical reporting on an unfolding political issue was blocked for several days while the subject was being widely discussed online.

Photo Credit (for first photo):
State Library of Western Australia
[Screenshot is taken directly from a Twitter embed]

Filed Under: case study, consistency, content moderation, historical content, nudity, reporting
Companies: facebook

Content Moderation Case Studies: Misleading Information From Official Sources (2020)

from the protest-edition dept

Summary: With news breaking so rapidly, it’s possible that even major newspapers or official sources may get information wrong. Social media sites, like Twitter, need to determine how to deal with “news” tweets that later turn out to be misleading -- even when coming from major news organizations, citing official government organizations.

With widespread protests around the United States calling attention to police brutality and police activity disproportionately targeting the black community, the NY Post tweeted a link to an article discussing an internal communication by the NY Police Department (NYPD) warning of “concrete disguised as ice cream cups” that were supposedly found at some of the protests, with the clear implication being that this was a way to disguise items that could be used for violence or property destruction.

The article was criticized widely by people who pointed out that the items in fact appear to be part of a standard process for testing concrete mixtures, with the details of each mixture written on the side of the containers. Since these were found at a construction site, it seems likely that the NYPD’s “alert” was, at best, misleading.

In response to continuing criticism, the NY Post made a very minor edit to the story, noting only that the markings on the cups make them “resemble concrete sample tests commonly used on construction sites.” However, the story and its title remained unchanged and the NY Post retweeted it a day later -- leading some to question why the NY Post was publishing misinformation, even if it was accurately reporting the content of an internal police memo.

Questions for Twitter:

• Should it flag potentially misleading tweets when published in major media publications, such as the NY Post?
• Should it matter if the information originated at an official government source, such as the NYPD?
• How much investigation should be done to determine the accuracy (or not) of the internal police report? How should the NY Post’s framing of the story reflect this investigation?
• Does it matter that the NY Post retweeted the story a day after the details were credibly called into question?

Questions and policy implications to consider:

• Do different publications require different standards of review?
• Does it matter if underlying information is coming from a governmental organization?
• If a media report accurately reports on the content of an underlying report that is erroneous or misleading, does that make the report itself misleading?
• How much does wider context (protests, accusations of violence, etc.) need to be considered when making determinations regarding moderation?

Resolution: To date, Twitter has left the tweets up, and the NY Post article remains online with only the very minor edit that was added a few hours after the article received widespread criticism. The NY Post tweets have not received any fact check or other moderation to date. There are, however, many replies and quote tweets calling out what people feel to be misleading aspects of the story (as well as plenty from people taking the content of the story at face value, and worrying about how the items might be used for violence).

Filed Under: case studies, content moderation, nypd, police, protests

Moderation Of Racist Content Leads To Removal Of Non-Racist Pages & Posts (2020)

from the moderation-mistakes dept

Summary: Social media platforms are constantly seeking to remove racist, bigoted, or hateful content. Unfortunately, these efforts can cause unintended collateral damage to users who share surface similarities to hate groups, even though many of these users take a firmly anti-racist stance.

A recent attempt by Facebook to remove hundreds of pages associated with bigoted groups resulted in the unintended deactivation of accounts belonging to historically anti-racist groups and public figures.

The unintentional removal of non-racist pages occurred shortly after Facebook engaged in a large-scale deletion of accounts linked to white supremacists, as reported by OneZero:

Hundreds of anti-racist skinheads are reporting that Facebook has purged their accounts for allegedly violating its community standards. This week, members of ska, reggae, and SHARP (Skinheads Against Racial Prejudice) communities that oppose white supremacy are accusing the platform of wrongfully targeting them. Many believe that Facebook has mistakenly conflated their subculture with neo-Nazi groups because of the term “skinhead.”

The suspensions occurred days after Facebook removed 200 accounts connected to white supremacist groups and as Mark Zuckerberg continues to be scrutinized for his selective moderation of hate speech.

Dozens of Facebook users from around the world reported having their accounts locked or their pages disabled due to their association with the "skinhead" subculture. This subculture dates back to the 1960s and predates the racist/fascist tendencies now commonly associated with that term.

Facebook’s policies have long forbidden the posting of racist or hateful content. Its ban on "hate speech" encompasses the white supremacist groups it targeted during its purge of these accounts. The removals of accounts not linked to racism -- but linked to the term "skinhead' -- were accidental, presumably triggered by a term now commonly associated with hate groups.

Questions to consider:

• How should a site handle the removal of racist groups and content?
• Should a site use terms commonly associated with hate groups to search for content/accounts to remove?
• If certain terms are used to target accounts, should moderators be made aware of alternate uses that may not relate to hateful activity?
• Should moderators be asked to consider the context surrounding targeted terms when seeking to remove pages or content?
• Should Facebook provide users whose accounts are disabled with more information as to why this has happened? (Multiple users reported receiving nothing more than a blanket statement about pages/accounts "not following Community Standards.")
• If context or more information is provided, should Facebook allow users to remove the content (or challenge the moderation decision) prior to disabling their accounts or pages?

Resolution: Facebook's response was nearly immediate. Facebook apologized to users shortly after OneZero reported the apparently-erroneous deletion of non-racist pages. Guy Rosen (VP- Integrity at Facebook) also apologized for the deletion on Twitter to the author of the OneZero post, saying the company had removed these pages in error during its mass deletion of white supremacists pages/accounts and said the company is looking into the error.

Filed Under: bias, case studies, content moderation, mistakes, racist speech, skinheads
Companies: facebook

Content Moderation Case Study: Talking About Racism On Social Media (2019)

from the what's-racist,-and-what's-a-discussion dept

Summary: With social media platforms taking a more aggressive stance regarding racist, abusive, and hateful language on their platforms, there are times when those efforts end up blocking conversations about race and racism itself. The likelihood of getting an account suspended or taken down has been referred to as “Facebooking while Black.”

As covered in USA Today, the situations can become complicated quickly:

A post from poet Shawn William caught [Carolyn Wysinger’s] eye. "On the day that Trayvon would've turned 24, Liam Neeson is going on national talk shows trying to convince the world that he is not a racist." While promoting a revenge movie, the Hollywood actor confessed that decades earlier, after a female friend told him she'd been raped by a black man she could not identify, he'd roamed the streets hunting for black men to harm.

For Wysinger, an activist whose podcast The C-Dubb Show frequently explores anti-black racism, the troubling episode recalled the nation's dark history of lynching, when charges of sexual violence against a white woman were used to justify mob murders of black men.

"White men are so fragile," she fired off, sharing William's post with her friends, "and the mere presence of a black person challenges every single thing in them."

This post was quickly deleted by Facebook, claiming that it violated the site’s “hate speech” policies. She was also warned that attempting to repost the content would lead to her being banned for 72 hours.

Facebook’s rules are that an attack on a “protected characteristic” -- such as race, gender, sexuality or religion -- violates its “hate speech” policies. In this case, the removal was because Wysinger’s post was speech that targeted a group based on a “protected characteristic” (in this case “white men”) and thus it was flagged for deletion.

Questions to consider:

• How should a site handle sensitive conversations regarding discrimination?
• If a policy defines “protected characteristics,” are all groups defined by one of those characteristics to be treated equally?
  • If so, is that in itself a form of disparate treatment for historically oppressed groups?
  • If not, does that risk accusations of bias?
• Is there any way to take wider context into account during human or technological reviews?
• Should the race/gender/sexuality/religion of the speaker be taken into account? What about the target of the speech?
• Is there a way to determine if a comment is “speaking up” to power or “speaking down” from a position of power?

Resolution: In the case described above, Wysinger chose not to risk losing her Facebook access for any amount of time, and simply chose not to repost the statement about Liam Neeson. Facebook, for its part, has continued to adapt and adjust is policy. It streamlined its “appeals” process to try to deal with many of these kinds of cases, and has announced (after two years of planning and discussion) the first members of its Facebook Oversight Board, an independent body that will be tasked with reviewing particularly tricky content takedown cases on the platform.

Filed Under: case study, content moderation, content moderation case study, racism

Content Moderation Case Study: Can An Open Encyclopedia Handle Disinformation? (2005)

from the disinfo-on-an-open-platform dept

Summary: Wikipedia was founded in 2001 and the open encyclopedia that anyone could edit grew much faster than most people (including its founders) expected. In 2005, one of the first big controversies concerning disinformation on Wikipedia arose, when journalist and political figure John Seigenthaler wrote an article in USA Today calling out claims on Wikipedia that, among other things, he was involved in the assassinations of both John F. Kennedy and Robert F. Kennedy (for whom he worked for a time).

The entry in question read:

John Seigenthaler Sr. was the assistant to Attorney General Robert Kennedy in the early 1960s. For a short time, he was thought to have been directly involved in the Kennedy assassinations of both John, and his brother, Bobby. Nothing was ever proven.

John Seigenthaler moved to the Soviet Union in 1972, and returned to the United States in 1984.

He started one of the country's largest public relations firms shortly thereafter.

Two months before publishing the USA Today article, Seigenthaler reached out to Wikipedia cofounder Jimmy Wales and requested that his entry be deleted. It was, but other Wikipedia users soon brought it back, leading Seigenthaler to publish the story.

Seigenthaler referred to this entire incident as “character assassination” and it suddenly put the fact checking of Wikipedia into tremendous focus, leading some to question whether or not the site could even survive.

Seigenthaler’s article raised many questions about Wikipedia, how it was edited, and how reliable it was as a research tool. For years, Seigenthaler insisted he wouldn’t edit his own entry because it would lend legitimacy to the service that he wanted no part of.

The controversy received widespread media coverage, and raised many questions about how Wikipedia can and should deal with misinformation on its platform while staying true to its principles and concept as an encyclopedia built on anyone’s contributions.

Decisions to be made by Wikipedia:

• Should it step in and edit Seigenthaler’s entry?
• Should it block attempts to restore the false information?
• Should it block further edits and lock down Seigenthaler’s page?
• Should it change its overall policies regarding open editing?

Questions and policy implications to consider:

• Are there ways to limit “vandalism” on an open system?
• Were certain pages at greater risk of false information that others? If so, how should that difference be handled?
• Will any changes decrease the trust people have in Wikipedia?
• Will any changes increase the trust people have in Wikipedia?

Resolution: In some ways the controversy itself solved the “problem” in that the widespread attention resulted in many Wikipedia editors/users making sure that Seigenthaler’s entry was accurate. However, it also prompted Wikipedia to implement some wider changes, including restricting the ability to create new pages to registered users, and creating a new editing policy regarding “biographies of living persons.” A few years later, it adjusted the policy some more to push for a more thorough review of edits to pages about living individuals.

Just weeks after the incident and controversy, Nature published a study claiming that Wikipedia was equally as reliable as the esteemed Encyclopedia Britannica.

Seigenthaler continued to question the entire Wikipedia concept for years. Separately, while in the USA Today piece, Seigenthaler detailed his failed efforts to track down who had made the edits (including explaining how he considered possible legal action), the person who created the page, Brian Chase came forward a few weeks after the story went viral, saying it was just a joke that he regretted.

Filed Under: case study, content moderation, disinformation, john segenthaler
Companies: wikipedia