Content Moderation Case Study: Ask.fm Responds After A Teen's Suicide Is Linked To Bullying On The Site (August 2013)

from the difficult-content-moderation-questions dept

Summary: After a UK teen took her own life in response to bullying on social networking site, ask.fm, her father asked both the site and the UK government to take corrective measures to prevent further tragedies. This wasn't an isolated incident. Reports linked multiple suicides to bullying on the teen-centered site.

Ask.fm's problems with bullying and other abuse appeared to be far greater than those observed on other social media sites. Part of this appeared to be due to the site's user base, which was much younger than more-established social media platforms. This -- combined with the option to create anonymous accounts -- seemed to have made ask.fm a destination for abusive users. What moderation existed before these problems became headline news was apparently ineffective, resulting in a steady stream of horrific stories until the site began to make serious efforts to curb a problem now too big to ignore.

Ask.fm's immediate response to both the teen's father and UK Prime Minister David Cameron's criticism (Cameron called for a boycott of the site) was to point to existing moderation efforts put in place to deter bullying and other terms of service violations.

After major companies pulled their advertising, ask.fm pledged to assist police in investigating the circumstances behind the teen's suicide, as well as consult with a law firm to see if moderation efforts could be improved. It also hired more moderators and a safety officer, and made its "Report" button more prominent.

More than a year after ask.fm became the target of criticism around the world, the site implemented its first Safety Advisory Board. The group of experts on teens and their internet use was tasked with reducing the amount of bullying on the platform and making it safer for its young users.

More significantly, ask.fm's founders -- who were viewed as unresponsive to criticism -- were removed by the site's new owners, InterActiveCorp (IAC). IAC pledged to work more closely with US law enforcement and safety experts to improve moderation efforts.

Decisions to be made by ask.fm:

• Should anonymous accounts be eliminated (or stop-gapped by gathering IP address/personal info) to limit abusive behavior?
• Does catering to a younger user base create unique problems not found at sites that skew older?
• Would more transparency about moderation efforts/features nudge more users towards reporting abuse?
• Should the site directly intervene when moderators notice unhealthy/unwanted user interactions?

Questions and policy implications to consider:

• Given the international reaction to the teen's suicide, does a minimal immediate response make the perceived problem worse?
• Does having a teen user base increase the risk of direct regulation or unfavorable legislation, given the increased privacy protections for minors in many countries?
• Are moderation efforts resulting from user reports vetted periodically to ensure the company isn't making bullying/trolling problems worse by allowing abusive users to get others suspended or banned?

Resolution: When immediate steps did little to deter criticism, ask.fm formed a Safety Committee and, ultimately, dismissed founders that appeared to be unresponsive to users' concerns. The site made changes to its moderation strategies, hired more moderators, and made users more aware of the features they could use to report users and avoid unwanted interactions.

Filed Under: bullying, case study, content moderation, suicide
Companies: ask.fm

Content Moderation Case Study: Twitter's Algorithm Misidentifies Harmless Tweet As 'Sensitive Content' (April 2018)

from the content-moderation-isn't-easy dept

Summary: While some Twitter users welcome the chance to view and interact with "sensitive" content, most do not. Twitter utilizes algorithms to detect content average users would like to avoid seeing, especially if they've opted in to Twitter's content filtering via their user preferences.

Unfortunately, software can't always tell what's offensive and what just looks offensive to the programmable eye that constantly scans uploads for anything that should be hidden from public view unless the viewer has expressed a preference to see it.

A long-running and well-respected Twitter account that focused on the weirder aspects of Nintendo's history found itself caught in Twitter's filters. The tweeted image featured an actor putting on his Princess Peach costume. It focused on the massive Princess Peach head, which apparently contained enough flesh color and "sensitive" shapes to get it -- and the Twitter account -- flagged as "sensitive."

The user behind the account tested Twitter to see if it was its algorithm or something else setting off the "sensitive" filter. Dummy accounts tweeting the image were flagged almost immediately, indicating it was the image -- rather than other content contained in the user's original account -- that had triggered the automatic moderation.

Unfortunately, the account was likely followed by several users who never expected it to suddenly shift to "sensitive" content. Thanks to the algorithm, the entire account was flagged as "sensitive," possibly resulting in the account losing followers.

Twitter ultimately removed the block, but the user was never directly contacted by Twitter about the alleged violation.

Decisions to be made by Twitter:

• Are false positives common enough that a notification process should be implemented?
• Should the process be stop-gapped by human moderators? If so, at what point does double-checking the algorithm become unprofitable?
• Would a challenge process that involved affected users limit collateral damage caused by AI mistakes?
• Does sensitive content negatively affect enough users that over-blocking/over-moderation is acceptable?

Questions and policy implications to consider:

• Should Twitter change its content rules to further deter the posting of sensitive content?
• Given Twitter's reputation as a porn-friendly social media platform, would stricter moderation of sensitive content result in a noticeable loss of users?
• Should Twitter continue to remain one of the only social media outlets that welcomes "adult" content?
• If users are able to opt out of filtering at any point, is Twitter doing anything to ensure younger users aren't exposed to sensitive material?

Resolution: Twitter removed the flag on the user's account. According to the user behind the account, it took the work of an employee "behind the scenes" to remove the "sensitive content" warning. Since there was no communication between Twitter and the user, it's unknown if Twitter has implemented any measures to limit future mischaracterizations of uploaded content.

Filed Under: content moderation, misidentification, offensive content
Companies: twitter

Content Moderation Case Study: Twitter Freezes Accounts Trying To Fact Check Misinformation (2020)

from the misinformation-vs-fact-checking dept

Summary: President Trump appeared on Fox News’ “Fox & Friends” and made some comments that were considered by many experts to be misinformation regarding the COVID-19 pandemic. One quote that particularly stood out was: "If you look at children, children are almost -- and I would almost say definitely -- but almost immune from this disease. They don't have a problem. They just don't have a problem." This is false. While it has been shown that children are less likely to get seriously ill or die from the disease, that is very different from being “immune.”

In response to this both Twitter and Facebook decided to remove clips of the video including those posted by the Trump Campaign. Given both platforms’ aggressive policies regarding COVID-19 disinformation (and the criticism that both have received for being too slow to act) this was not all that surprising. For additional context, just a week and half earlier there was tremendous controversy over a decision to remove a video of some doctors giving speeches in front of the Supreme Court that also presented misleading information regarding COVID-19. While the major platforms all blocked the video, they received criticism from both sides for it. Some argued the video should not have been taken down, while others argued it took the platforms too long to take it down.

Thus it was not surprising that Facebook and Twitter reacted quickly to this video, even though it was statements made by the President of the United States. However, more controversy arose because in taking down those video clips, Twitter also ended up removing reporters, such as Aaron Rupar, who were fact checking the claims, and activists, like Bobby Lewis, who were highlighting the absurdity of the clip.

Decisions to be made by Twitter:

• How aggressively should content moderation rules be applied to statements from the President of the United States?
• How important is it to remove potentially harmful information regarding health and immunity to a disease like COVID-19?
• Is it better to have such videos taken down too quickly, or too slowly?
• How do you determine who is fact-checking or debunking a video and who is spreading the misinformation?
• How do you handle situations where different people are sharing the same video for divergent purposes (some to spread misinformation, some to debunk it)?

Questions and policy implications to consider:

• Should the President’s speech receive special consideration?
• The same content can be used by different users for different reasons. Should content moderation take into account how the content is being used?
• Counterspeech can often be useful in responding to disinformation. What role is there in content moderation to promote or allow counterspeech?

Resolution: The Twitter accounts that were temporarily suspended removed the “offending” tweets in order to continue tweeting. There is no indication of any change in policy so far from Twitter, which has focused on removing all such videos, regardless of context of the tweet surrounding them.

Filed Under: content moderation, fact checking, journalism, misinformation, reporting
Companies: twitter

Content Moderation Case Study: Usenet Has To Figure Out How To Deal With Spam (April 1994)

from the the-original-content-moderation-case-study dept

Summary: In the early 1990s, facing increased pressure from the commercial sector who sensed there might be some value in the nascent “Internet,” the National Science Foundation began easing informal restrictions on commercial activity over the Internet. This gave rise to the earliest internet companies -- but also to spam. Before the World Wide Web had really taken off, the place where a great deal of internet communication took place was Usenet, created in 1980, which was what one might think of as a proto-Reddit, with a variety of “newsgroups” dedicated to different subjects that users could post to.

Usenet was a decentralized service based on the Network News Transfer Protocol. Users needed a Usenet reader, from which they would connect to any number of Usenet servers and pull down the latest content in the newsgroups they followed. In early 1994, a husband and wife lawyer team, Laurence Canter and Martha Siegel, decided that they would advertise their legal services regarding immigration to the US (specifically help with the infamous “Green Card Lottery” to get a green card to the US) on Usenet.

They hired a programmer to write a perl script that posted their advertisement on 5,500 separate news groups. While cross-posting was possible (a single post designated for multiple newsgroups), this particular message was posted individually to each newsgroup, which made it even more annoying for users -- since most Usenet reader applications would have recognized the same message as “read” in different newsgroups if it had merely been cross-posted. Posting it this way guaranteed that many people saw the message over and over and over again.

It is generally considered one of the earliest examples of commercial “spam” on the internet -- and certainly the most “successful” at the time. It also angered a ton of people. According to Time Magazine, Canter and Siegel faced immediate backlash:

In the eyes of many Internet regulars, it was a provocation so bald-faced and deliberate that it could not be ignored. And all over the world, Internet users responded spontaneously by answering the Spammers with angry electronic- mail messages called "flames." Within minutes, the flames -- filled with unprintable epithets -- began pouring into Canter and Siegel's Internet mailbox, first by the dozen, then by the hundreds, then by the thousands. A user in Australia sent in 1,000 phony requests for information every day. A 16-year-old threatened to visit the couple's "crappy law firm" and "burn it to the ground." The volume of traffic grew so heavy that the computer delivering the E-mail crashed repeatedly under the load. After three days, Internet Direct of Phoenix, the company that provided the lawyers with access to the Net, pulled the plug on their account.

It wasn’t just Usenet users. Immigration lawyers were also upset in part because Canter and Siegel were asking for money to do what most people could easily do for free:

Unfortunately, it also provided an opportunity for charlatans to charge exorbitant fees to file lottery entries for hopeful immigrants.

In truth, all it took to enter the drawing was a postcard with your name and address mailed to the designated location.

Canter and Siegel, a husband-and-wife law firm, decided to join the lottery frenzy by pitching their own overpriced services to immigrant communities.

The two were unrepentant, later claiming they made over $100,000 from the advertisement. They quickly set up a new company called “Cybersell” to do this for others -- and signed a contract to write a book for HarperCollins originally called "How To Make A Fortune On The Information Superhighway."

Decisions to be made by Usenet server providers:

• Would they need to start being more aggressive in monitoring and moderating their newsgroups?
• Would it even be possible to prevent spam?
• Should they even carry news groups that allowed for open contributions?

Decisions to be made by ISPs:

• Should they allow Canter and Siegel to use their internet access to spam newsgroups?
• How should they handle the backlash from users angry about the spam campaigns?

Questions and policy implications to consider:

• What is the boundary between allowed commercial speech or advertising and spam? How do you distinguish it?
• Is it possible to have distributed systems (as opposed to centralized ones) that don’t end up filled with spam?
• What are the legal implications of spam?

Resolution: Canter and Siegel remained a scourge on the internet for some time. Various service providers were quick to kick them off as soon as it was discovered that they were using them. Indeed, many seemed willing to talk publicly about their decisions, such as Netcom, which shut down their account soon after the original spam happened and after Canter and Siegel had announced plans to continue spamming:

NETCOM On-Line Communications has taken the step of cancelling the service of Laurence Canter of Canter and Siegel, the lawyer commonly referred to as the "Green Card Lawyer". Mr. Canter had been a customer of NETCOM in the past. He had been cautioned for what we consider abuse of NETCOM's system resources and his systematic and willful actions that do not comply with the codes of behavior of USENET.

Mr. Canter has been widely quoted in the print and on-line media about his intention to continue his practice of advertising the services of his law firm using USENET newsgroups. He has also widely posted his intention to sell his services to advertise for others using the newsgroups. We do not choose to be the provider that will carry his messages.

That link also has notices from other service providers, such as Pipeline and Performance Systems, saying they were removing internet access.

Others focused on trying to help Usenet server operators get rid of the spam. Programmer Arnt Gulbrandsen quickly put together a tool to help fight this kind of spam by “cancelling” the messages when spotted. This actually helped establish the early norm that it was okay to block and remove spam.

As for Canter and Siegel, they divorced a couple years later, though both kept promoting themselves as internet marketing experts. Canter was disbarred in Tennessee for his internet advertising practices, though he had already moved on from practicing law. Cybersell, the company they had setup to do internet advertising, was apparently dissolved in 1998.

Filed Under: content moderation, distributed systems, spam, usenet

Content Moderation Case Study: Twitter Removes Account For Pointing Users To Leaked Documents Obtained By A Hacking Collective (June 2020)

from the reporting-on-hacking dept

Summary: Late in June 2020, a leak-focused group known as "Distributed Denial of Secrets" (a.k.a., "DDoSecrets") published a large collection of law enforcement documents apparently obtained by the hacking collective Anonymous.

The DDoSecrets' data dump was timely, released as protests over the killing of a Black man by a white police officer continued around the nation neared their second consecutive month. Links to the files hosted at DDoSecrets' website spread quickly across Twitter, identified by the hashtag #BlueLeaks.

The 269-gigabyte trove of law enforcement data, emails, and other documents was taken from Netsential, which confirmed a security breach had led to the exfiltration of these files. The exfiltration was further acknowledged by the National Fusion Center Association, which told affected government agencies the stash included personally identifiable information. While this trove of data proved useful to activists and others seeking uncensored information about police activities, some expressed concern the personal info could be used to identify undercover officers or jeopardize ongoing investigations.

The first response from Twitter was to mark links to the DDoSecret files as potentially harmful to users. Users clicking on links to the data were told it might be unsafe to continue. The warning suggested the site might steal passwords, install malicious software, or harvest personal data. The final item on the list in the warning was a more accurate representation of the link destination: it said the link led to content that violated Twitter's terms of service.

Twitter's terms of service forbid users from "distributing" hacked content. This ban includes links to other sites hosting hacked content, as well as screenshots of forbidden content residing elsewhere on the web.

Shortly after the initial publication of the document trove, Twitter went further. It permanently banned DDoSecrets' Twitter account over its tweets about the hacked data. It also began removing tweets from other accounts that linked to the site.

Decisions to be made by Twitter:

• Should the policy against the posting of hacked material be as strictly enforced when the hacked content is potentially of public interest?
• Should Twitter have different rules for “journalists” or “journalism organizations” with regards to the distribution of information?
• How should Twitter distinguish “hacked” information from “leaked” information?
• Should all hacked content be treated as a violation of site terms, even if it does not contain personal info and/or trade secrets?
• How should Twitter handle mirrors of such content?
• How should Twitter deal with the scenario in which someone links to the materials because of their newsworthiness, without even knowing the material was hacked?

Questions and policy implications to consider:

• Does a strict policy against "distributing" hacked content negatively affect Twitter's value as a source of breaking news?
• Does the mirroring of hacked content significantly increase the difficulty and cost of moderation efforts?

Resolution: While DDoSecrets' site remains up and running, its Twitter account does not. The permanent suspension of the account and additional moderation efforts have limited the spread of URLs linking to the apparently illicitly-obtained documents.

Filed Under: content moderation, ddosecrets, hacking, journalism, leaks, reporting
Companies: twitter

Content Moderation Case Study: Pinterest's Moderation Efforts Still Leave Potentially Illegal Content Where Users Can Find It (July 2020)

from the pin-this dept

Summary: Researchers at OneZero have been following and monitoring Pinterest's content moderation efforts for several months. The "inspiration board" website hosts millions of images and other content uploaded by users.

Pinterest's moderation efforts are somewhat unique. Very little content is actually removed, even when it might violate the site's guidelines. Instead, as OneZero researchers discovered, Pinterest has chosen to prevent the content from surfacing by blocking certain keywords for generating search results.

The problem, as OneZero noted, is that hiding content and blocking keywords doesn't actually prevent users from finding questionable content. Some of this content includes images that sexually exploit children.

While normal users may never see this using Pinterest's built-in search tools, users more familiar with how search functions work can still access content Pinterest feels violates its guidelines, but hasn't actually removed from its platform. By navigating to a user's page, logged-out users can perform searches that seem to bypass Pinterest's keyword-blocking. Using Google to search the site -- instead of the site's own search engine -- can also surface content hidden by Pinterest.

Pinterest's content moderation policy appears to be mostly hands-off. Users can upload nearly anything they want to with the company only deleting (and reporting) clearly illegal content. For everything else that's questionable (or potentially harms other users), Pinterest opts for suppression, rather than deletion.

“Generally speaking, we limit the distribution of or remove hateful content and content and accounts that promote hateful activities, false or misleading content that may harm Pinterest users or the public’s well-being, safety or trust, and content and accounts that encourage, praise, promote, or provide aid to dangerous actors or groups and their activities,” Pinterest’s spokesperson said of the company’s guidelines.

Unfortunately, users who manage to bypass keyword filters or otherwise stumble across buried content will likely find themselves directed to other buried content. Pinterest's algorithms surface content related to whatever users are currently viewing, potentially leading users even deeper into the site's "hidden" content.

Decisions to be made by Pinterest:

• Is hiding content effective in steering users away from subject matter/content Pinterest would rather they didn't access?
• Would deletion -- rather than hiding -- result in affected users leaving the platform?
• Is questionable content a severe enough problem the company should rethink its moderation protocols?
• Should "related content" algorithms be altered to prevent the surfacing of hidden content?

Questions and policy implications to consider:

• Does hiding -- rather than removing -- content potentially encourage users to use this invisibility to engage in surreptitious distribution of questionable or illegal content?
• Does the possibility of hidden content resurfacing steer ad buyers away from the platform?
• Will this approach to moderation -- hidden vs. deletion -- remain feasible as pressure for sites to aggressively police misinformation and "fake news" continues to mount?

Resolution: Pinterest's content moderation strategy remains mostly unchanged. As the site's spokesperson stated, the site appears to feel the hiding of content addresses most raised concerns, even if it does allow more determined site users to locate content the site would rather they never saw.

Filed Under: content moderation, search
Companies: pinterest

Content Moderation Case Study: Detecting Sarcasm Is Not Easy (2018)

from the kill-me-now dept

Summary: Content moderation becomes even more difficult when you realize that there may be additional meaning to words or phrases beyond their most literal translation. One very clear example of that is the use of sarcasm, in which a word or phrase is used either in the opposite of its literal translation or as a greatly exaggerated way to express humor.

In March of 2018, facing increasing criticism regarding certain content that was appearing on Twitter, the company did a mass purge of accounts, including many popular accounts that were accused of simply copying and retweeting jokes and memes that others had created. Part of the accusation for those that were shut down, was that there was a network of accounts (referred to as “Tweetdeckers” for the user of the Twitter application Tweetdeck) who would agree to mass retweet some of those jokes and memes. Twitter suggested that these retweet brigades were inauthentic and thus banned from the platform.

In the midst of all of these suspensions, however, there was another set of accounts and content suspended, allegedly for talking about “self -harm.” Twitter has policies regarding glorifying self-harm which it had just updated a few weeks before this new round of bans.

However, in trying to apply that, Twitter took down a bunch of tweets that had people sarcastically using the phrase “kill me.” This included suddenly suspending many accounts despite many of those tweets being from many years earlier. It appeared that Twitter may have just done a search on “kill me” or other similar words and phrases including “kill myself,” “cut myself,” “hang myself,” “suicide,” or “I wanna die.”

While some of these may indicate intentions for self-harm, in many other cases they were clearly sarcastic or just people saying odd things, and yet Twitter temporarily suspended many of those accounts and asked the users to delete the tweets. In at least some cases, the messages from Twitter did include some encouraging words, such as “Please know that there are people out there who care about you, and you are not alone.” But that did not appear to be on all of the messages. That language, at least, suggested a specific response to concerns about self-harm.

Decisions to be made by Twitter:

• How do you handle situations where users indicate they may engage in self-harm?
• Should such content be removed or are there other approaches?
• How do you distinguish between sarcastic phrases and real threats of self-harm?
• What is the best way to track and monitor claims of self-harm? Does a keyword or key phrase list search help?
• Does automated tracking of self-harm messages work? Or is it better to rely on user reports?
• Does it change if the supposed messages regarding self-harm are years old?

Questions and policy implications to consider:

• Is suspending people for self-harm likely to prevent the harm? Or is it just hiding useful information from friends, family, officials, who might help?
• Detecting sarcasm creates many challenges; should internet platforms be the arbiters of what counts as reasonable sarcasm? Or must it take all content literally?
• Automated solutions to detect things like self-harm may cover a wider corpus of material, but is also more likely to misunderstand context. How should these issues be balanced?

Resolution: This continues to be a challenge for various platforms, including Twitter. The company has continued to tweak its policies regarding self-harm over the year, including partnering with suicide prevention groups in various locations to seek to help those who indicate that they are considering self-harm.

Filed Under: case study, content moderation, kill me, sarcasm
Companies: twitter

Content Moderation Case Study: Amazon Alters Publishing Rules To Deter Kindle Unlimited Scammers (April 2016)

from the it's-always-the-scammers dept

Summary: In July 2014, Amazon announced its "Netflix, but for ebooks" service, Kindle Unlimited. Kindle Unlimited allowed readers access to hundreds of thousands of ebooks for a flat rate of $9.99/month.

Amazon paid authors from a subscriber fee pool. Authors were paid per page read by readers -- a system that was meant to reward more popular writers with a larger share of the Kindle Unlimited payment pool.

This system was abused by scammers once it became clear Amazon wasn't spying on Kindle Users to ensure books were actually being read -- i.e., keeping track of time spent on pages of text by readers or total amount of time spent reading. Since Amazon had no way to verify if readers were actually reading the content, scammers deployed a variety of tricks to increase their unearned earnings.

Part of the scam relied on Amazon's willingness to pay authors for partially-read books. If only 100 pages of a 500-page book were read, the author still got credit for the 100 pages read by an Unlimited user. Scammers inflated "pages read" counts by moving the table of contents to the end of the book or offering dozens of different languages in the same ebook, relying on readers skipping hundreds of pages into the ebook to access the most popular translation. Other scammers offered readers chances to win free products and gift cards via hyperlinks that brought readers to the end of the scammers' ebooks -- books that sometimes contained thousands of pages.

The other part of the scam equation was Amazon's hands-off approach to self-publishing. Amazon has opened its platform and appears to do very little to police the content of ebooks, other than requiring authors to follow certain formatting rules. Amazon is neither a publisher nor an editor, which has created a market for algorithmically-generated content as well as a home for writers seeking a distribution outlet for their bigoted and hateful writing.

Once Amazon realized the payout system was being gamed, it altered the way Kindle Unlimited operated. It began removing scammers, notifying authors and customers that it was doing this in response to Unlimited readers' complaints.

Some in the community have contacted us about the activities of a small minority of publishers who may attempt to inflate sales or pages read through the use of various techniques, such as adding unnecessary or confusing hyperlinks, misplacing the TOC [table of contents] or adding distracting content.

Unfortunately, Amazon's moderation efforts did affect a very small number of legitimate authors. Writer Walter Jon Williams was blocked from selling his ebook because his table of contents was located near the end of his book. Williams pointed out he had done this to maximize the amount of content prospective readers/purchasers could access using Amazon's "Look Inside" feature. After some back-and-forth, Williams' book and buy button were restored by Amazon.

Amazon continues to work to minimize abuse of the Kindle Unlimited system. The most noticeable and major change has been to cap earnings at 3,000 pages per ebook per reader. This limits the amount of money scammers can pull from the Unlimited payout pool. It also limits the number of times Kindle Unlimited readers will find themselves scrolling through ebooks solely designed to inflate page counts.

Decisions to be made by Amazon:

• Can automated moderation alone determine whether an uploaded ebook is a legitimate offering?
• Does altering the payout rules for Kindle Unlimited negatively affect legitimate authors?
• Does the ongoing abuse of various Amazon ebook programs justify more data collection on customers and their reading habits?
• Should authors be notified ahead of changes to Amazon services or would more transparency result in more abuse by scammers?

Questions and policy implications to consider:

• Does the flat rate subscriber fee cover the costs of policing an ebook publishing ecosystem of this size?
• Who deserves more protection? Sellers/writers or customers? How do you strike the correct balance that provides more value to both sides of the transaction?
• Is more vetting needed on the front end (ID verification, etc.) to prevent further abuse?

Resolution: Amazon reacted to abuse of its Unlimited system by clarifying rules for content placement and removing ebooks that violated the company's publishing guidelines. It also changed the way the pool of Kindle Unlimited funds were paid out, limiting the amount of pool money scammers could remove from the system by artificially inflating "read pages" counts.

Filed Under: content moderation, kindle, kindle unlimited, scammers
Companies: amazon

Content Moderation Case Study: Facebook Responds To A Live-streamed Mass Shooting (March 2019)

from the live-content-moderation dept

Summary: On March 15, 2019, the unimaginable happened. A Facebook user -- utilizing the platform's live-streaming option -- filmed himself shooting mosque attendees in Christchurch, New Zealand.

By the end of the shooting, the shooter had killed 51 people and injured 49. Only the first shooting was live-streamed, but Facebook was unable to end the stream before it had been viewed by a few hundred users and shared by a few thousand more.

The stream was removed by Facebook almost an hour after it appeared, thanks to user reports. The moderation team began working immediately to find and delete re-uploads by other users. Violent content is generally a clear violation of Facebook's terms of service, but context does matter. Not every video of violent content merits removal, but Facebook felt this one did.

The delay in response was partly due to limitations in Facebook's automated moderation efforts. As Facebook admitted roughly a month after the shooting, the shooter's use of a head-mounted camera made it much more difficult for its AI to make a judgment call on the content of the footage.

Facebook's efforts to keep this footage off the platform continue to this day. The footage has migrated to other platforms and file-sharing sites -- an inevitability in the digital age. Even with moderators knowing exactly what they're looking for, platform users are still finding ways to post the shooter's video to Facebook. Some of this is due to the sheer number of uploads moderators are dealing with. The Verge reported the video was re-uploaded 1.5 million times in the 48 hours following the shooting, with 1.2 million of those automatically blocked by moderation AI.

Decisions to be made by Facebook:

• Should the moderation of live-streamed content involve more humans if algorithms aren't up to the task?
• When live-streamed content is reported by users, are automated steps in place to reduce visibility or sharing until a determination can be made on deletion?
• Will making AI moderation of livestreams more aggressive result in over-blocking and unhappy users?
• Do the risks of allowing content that can't be moderated prior to posting outweigh the benefits Facebook gains from giving users this option?
• Is it realistic to "draft" Facebook users into the moderation effort by giving certain users additional moderation powers to deploy against marginal content?

Questions and policy implications to consider:

• Given the number of local laws Facebook attempts to abide by, is allowing questionable content to stay "live" still an option?
• Does newsworthiness outweigh local legal demands (laws, takedown requests) when making judgment calls on deletion?
• Does the identity of the perpetrator of violent acts change the moderation calculus (for instance, a police officer shooting a citizen, rather than a member of the public shooting other people)?
• Can Facebook realistically speed up moderation efforts without sacrificing the ability to make nuanced calls on content?

Resolution: Facebook reacted quickly to user reports and terminated the livestream and the user's account. It then began the never-ending work of taking down uploads of the recording by other users. It also changed its rules governing livestreams in hopes of deterring future incidents. The new guidelines provide for temporary and permanent bans of users who livestream content that violates Facebook's terms of service, as well as prevent these accounts from buying ads. The company also continues to invest in improving its automated moderation efforts in hopes of preventing streams like this from appearing on users' timelines.

Filed Under: case study, christchurch, content moderation, live streaming, new zealand, shooting
Companies: facebook

Content Moderation Case Study: US Army Bans Users For Asking About War Crimes On Twitch & Discord (July 2020)

from the the-1st-amendment-matters dept

Summary: Content moderation questions are not just about the rules that internet platforms create for themselves to enforce: they sometimes involve users themselves enforcing some form of the site’s rules, or their own rules, within spaces created on those platforms. One interesting case study involves the US Army’s esports team and how it has dealt with hecklers.

The US Army has a variety of different channels for marketing itself to potential recruits, and lately it’s been using its own “professional esports team” as something of a recruiting tool. Like many esports teams, the US Army team set up a Discord server. After some people felt that the Army was trying to be too “cute” on Twitter -- by tweeting the internet slang “UwU” -- a bunch of users set out to see how quickly they could be banned from the Army’s Discord server. In fact, many users started bragging about how quickly they were being banned -- often by posting links or asking questions related to war crimes, and accusations of the US Army’s involvement in certain war crimes.

This carried over to the US Army’s esports streaming channel on Twitch, where it appears that the Army set up certain banned words and phrases, including “war crimes,” leading at least one user -- esports personality Rod “Slasher” Breslau -- to try to get around that filter by typing “w4r cr1me” instead. This made it through and a few seconds later Breslau was banned from the chat by the Army’s esports player Green Beret Joshua “Strotnium” David, with David saying out loud during the stream “have a nice time getting banned, my dude.” Right before saying this David was mocking “internet keyboard monsters” for this kind of activity.

When asked about this, the Army told Vice News that it considered the questions to be a form of harassment, and in violation of Twitch’s stated rules, even though it was the Army that was able to set the specific moderation rules on the account and choose who to ban:

"The U.S. Army eSports Team follows the guidelines and policies set by Twitch, and they did ban a user from their account," a representative of the U.S. Army esports team said in a statement. "Team members are very clear when talking with potential applicants that a game does not reflect a real Army experience. They discuss their career experiences in real terms with factual events. Team members ensure people understand what the Army offers through a realistic lens and not through the lens of a game meant for entertainment. This user's question was an attempt to shift the conversation to imply that Soldiers commit war crimes based on an optional weapon in a game, and we felt that violated Twitch's harassment policy. The U.S. Army offers youth more than 150 different careers, and ultimately the goal of the Army eSports Team is to accurately portray that range of opportunities to interested youth."

Decisions to be made by the US Army:

• Is it appropriate or reasonable to set up filters to block certain keywords that might be seen as harassing?
  • Is it legal to do so under the 1st Amendment?
• How aggressive should the Army be in blocking words or phrases and banning accounts?
• How does the Army determine who is harassing the esports team vs. who is asking legitimate questions?
  • Is it the Army’s decision as to who is violating Twitch’s rules?
  • If the Army is banning accounts, should it take direct responsibility for that, or should it point to Twitch’s rules instead?
• Will banning accounts or blocking phrases lead to even greater unwanted attention?
• Are there alternative ways to deal with those who wish to highlight what they perceive to be war crimes by the US Army?

Decisions to be made by Twitch:

• Should government accounts be allowed to enforce Twitch’s rules any way they see fit?
• Given litigation concerning US politician blocks on social media accounts violating the 1st Amendment, should we even allow US government accounts to have tools for blocking and banning?

Questions and policy implications to consider:

• Unlike private companies, the US government is bound by the limitations of the 1st Amendment in moderating content. How does its use of social media deal with users who appear to be harassing government employees?
• In situations like this one, bans and blocks will often get more attention, leading to even more people trying on purpose to get blocked. Are there policies in place to deal with this?  Is this kind of behavior something worth banning? Are there alternative approaches?
• How much of the moderation for US government accounts should be managed by the companies, rather than the users?
• If the companies are moderating on behalf of a US government account, do they also need to keep 1st Amendment limitations in mind?

Resolution: So far the U.S. Army has stood by the claim that it believed people asking about war crimes was “harassment” and thus against Twitch’s rules. However, the ACLU has already stepped forward to say publicly that the comments were not harassment and the bans are in fact unconstitutional. It is possible that the ACLU or other 1st Amendment groups may bring litigation to this effect.

Indeed, after this case study was originally published, the Knight 1st Amendment Center, which brought the legal challenge that had Trump's Twitter blocking being declared a violation of the 1st Amendment, warned the Army about this, leading the Army to temporarily stop using Twitch.

However, the Army has recently returned to Twitch after first unbanning the users it had banned. Separately, Twitch told the Army to stop sharing fake prize giveaways after that practice was called out by The Nation.

The Army's return to Twitch has not been without issues. Upon return, the chat for the Army's Twitch channel was flooded with references to war crimes, the issue over which it was initially banning users.

Filed Under: 1st amendment, content moderation, us army
Companies: twitch