Content Moderation Case Study: Twitch Allows Users To Enable Emote-Only Chats (2016)

from the can't-be-a-troll-when-you-can-only-emote dept

Summary: Dealing with content moderation during real-time chats always presents an interesting challenge. Whether it’s being able to police language in real time, or dealing with trolling and harassment, chat has always been one of the most difficult content moderation challenges going back to its earliest days.

In 2016, Twitch decided to enable a new feature for its users: an “emote-only” mode for the chat. Emotes, on Twitch, are basically a custom set of what are more traditionally called emoji on most other websites/platforms. With Twitch, though, they are almost entirely custom, and users at certain levels are able to add their own.

Emote-only is one of a bunch of different modes and features that Twitch streamers can use to try to tame their chat. Twitch itself suggests using this as a way to stop harassment in the chats.

Turning on and off the feature is a choice for the streamer themselves, rather than Twitch. It’s just one of a few tools that Twitch users can enable to deal with potentially harassing behavior in the chat alongside their streams.

Decisions for Twitch:

• What tools should you provide to users to deal with abusive or harassing chat participants?
• Do features like this give more power to Twitch users, or are they offloading moderation demands from the company itself?
• Should there be any exceptions to emote-only mode?
• Are there times where a custom emote would be considered a policy violation because it takes on a harassing meaning in a certain context? 
• What sort of emote review processes should be put in place?

Questions and policy implications to consider:

• Rather than offering entirely binary options (allow/disallow) are there more creative alternatives for dealing with harassing behavior?
• Are there ways in which even emote-only mode might be abused for harassment?

Resolution: Emote-only mode was launched quietly with little fanfare from Twitch in 2016. While it may not be widely used, many streamers do find it useful. It is not just used for stopping harassment, but sometimes to stop people in a chat from revealing spoilers or other information that may impact what they’re streaming (such as information about the video game they are playing).

Originally posted to the Trust & Safety Foundation website.

Filed Under: content moderation, emotes, harassment
Companies: twitch

Content Moderation Case Study: GitHub Attempts To Moderate Banned Words Contained In Hosted Repositories (2015)

from the word-filters dept

Summary: GitHub solidified its position as the world's foremost host of open source software not long after its formation in 2008. Twelve years after its founding, GitHub is host to 190 million repositories and 40 million users.

Even though its third-party content is software code, GitHub still polices this content for violations of its terms of service. Some violations are more overt, like possible copyright infringement. But much of it is a bit tougher to track down.

A GitHub user found themself targeted by a GitHub demand to remove certain comments from their code. The user's code contained the word "retard" -- a term that, while offensive in certain contexts, isn't offensive when used as a verb to describe an intentional delay in progress or development. But rather than inform the user of this violation, GitHub chose to remove the entire repository, resulting in users who had forked this code to lose access to their repositories as well.

It wasn't until the user demanded an explanation that GitHub finally provided one. In an email sent to the user, GitHub said the code contained content the site viewed as "unlawful, offensive, threatening, libelous, defamatory, pornographic, obscene, or otherwise objectionable." More specifically, GitHub told the user to remove the words "retard" and "retarded," restoring the repository for 24 hours to allow this change to be made.

Decisions for GitHub:

• Is the blanket banning of certain words a wise decision, considering the idiosyncratic language of coding (and coders)?
• Should GitHub account for downstream repositories that may be negatively affected by removal of the original code when making content moderation decisions, and how?
• Could banned words inside code comments be moderated by only removing the comments, which would avoid impacting the functionality of the code?

Questions and policy implications to consider:

• Is context considered when moderating possible terms of service violations?
• Is it possible to police speech effectively when the content hosted isn't what's normally considered speech?
• Does proactive moderation of certain terms deter users from deploying code designed to offend?

Resolution: The user's repository was ultimately restored after the offending terms were removed. So were the repositories that relied on the original code GitHub decided was too offensive to allow to remain unaltered.

Unfortunately for GitHub, this drew attention to its less-than-consistent approach to terms of service violations. Searches for words considered "offensive" by GitHub turned up dozens of other potential violations -- none of which appeared to have been targeted for removal despite the inclusion of far more offensive terms/code/notes.

And the original offending code was modified with a tweak that substituted the word "retard" with the word "git" -- terms that are pretty much interchangeable in other parts of the world. The not-so-subtle dig at GitHub and its inability to detect nuance may have pushed the platform towards reinstating content it had perhaps pulled too hastily.

Originally posted on the Trust & Safety Foundation website.

Filed Under: code, content moderation, repositories
Companies: github

Content Moderation Case Study: Twitter Removes Account Of Human Rights Activist (2018)

from the perspective dept

Summary: Manzoor Ahmed Pashteen is a human rights activist in Pakistan, calling attention to unfair treatment of the Pashtun ethnic group, of which he is a member. In 2018, days after he led a rally in support of the Pashtun people in front of the Mochi Gate in Lahore, his Twitter account was suspended.

A number of human rights activists and other supporters of Pashteen quickly began to use Twitter to ask the company, and CEO Jack Dorsey, why Pashteen’s account was suspended.

Decisions to be made by Twitter:

• How do you distinguish human rights activists organizing protests from users trying to foment violence?
• How do you weigh reports from governments against activists who criticize the governments making the reports?
• How responsive should you be to users calling out suspensions they feel were unfair or mistaken?

Questions and policy implications to consider:

• Governments that are criticized by activists often accuse those activists of being “terrorists” or trying to foment violence. Is it possible to have a policy that distinguishes different types of protests and protest movements?
• Can shutting down the accounts of activists harm important causes?

Resolution: Twitter quickly restored Pashteen’s account in response to the outcry, though no public explanation of either the takedown or return was provided. Pashteen has continued to speak out in Pakistan, as well as to organize protests. Following one such protest in January of 2020, Pashteen was arrested and charged with criminal conspiracy and sedition.

At the time of this case study, it appears he is still facing those charges, though his Twitter account remains active with efforts to organize more protests and get the Pashtun people to protest government actions that have harmed them.

Originally posted to the Trust & Safety Foundation website.

Filed Under: civil rights, content moderation, human rights, manzoor ahmed pashteen, pakistan, pashtun
Companies: twitter

Content Moderation Case Study: Social Media Upstart Parler Struggles To Moderate Pornography (2020)

from the not-so-easy-to-be-clean dept

Summary: Upstart social network Parler (which is currently offline, but attempting to come back) has received plenty of attention for trying to take on Twitter -- mainly focusing on attracting many of the users who have been removed from Twitter or who are frustrated by how Twitter’s content moderation policies are applied. The site may only boast a fraction of the users that the social media giants have, but its influence can't be denied.

Parler promised to be the free speech playground Twitter never was. It claimed it would never "censor" speech that hadn't been found illegal by the nation's courts. When complaints about alleged bias against conservatives became mainstream news (and the subject of legislation), Parler began to gain traction.

But the company soon realized that moderating content (or not doing so) wasn't as easy as it hoped it would be. The problems began with Parler's own description of its moderation philosophy, which cited authorities that had no control over its content (the FCC), and the Supreme Court, whose 1st Amendment rulings apply to what the government may regulate regarding speech, but not private websites.

Once it became clear Parler was becoming the destination for users banned from other platforms, Parler began to tighten up its moderation efforts, resulting in some backlash from users. CEO John Matze issued a statement, hoping to clarify Parler's moderation decisions.

Here are the very few basic rules we need you to follow on Parler. If these are not to your liking, we apologize, but we will enforce:

- When you disagree with someone, posting pictures of your fecal matter in the comment section WILL NOT BE TOLERATED
- Your Username cannot be obscene like "CumDumpster"
- No pornography. Doesn't matter who, what, where,

Parler's hardline stance on certain content appeared to be more extreme than the platforms (Twitter especially) that Parler’s early adopters decried as too restrictive. In addition to banning content allowed by other platforms, Parler claimed to pull the plug on the sharing of porn, even though it had no Supreme Court/FCC precedent justifying this act.

Parler appears to be unable -- at least at this point -- to moderate pornographic content. Despite its clarification of its content limitations, Parler does not appear to have the expertise or the manpower to dedicate to removing porn from its service.

A report by the Houston Chronicle (which builds on reporting by the Washington Post) notes that Parler has rolled back some of its anti-porn policies. But it still wishes to be seen as a cleaner version of Twitter -- one that caters to "conservative" users who feel other platforms engage in too much moderation.

According to this report, Parler outsources its anti-porn efforts to volunteers who wade through user reports to find content forbidden by the site's policies. Despite its desires to limit the spread of pornography, Parler has become a destination for porn seekers.

The Post's review found that searches for sexually explicit terms surfaced extensive troves of graphic content, including videos of sex acts that began playing automatically without any label or warning. Terms such as #porn, #naked and #sex each had hundreds or thousands of posts on Parler, many of them graphic. Some pornographic images and videos had been delivered to the feeds of users tens of thousands of times on the platform, according to totals listed on the Parler posts.

Parler continues to struggle with the tension of upholding its interpretation of the First Amendment and ensuring its site isn't overrun by content it would rather not host.

Decisions to be made by Parler:

• Does forbidding porn make Parler more attractive to undecided users?
• Do moderation efforts targeting content allowed on other platforms undermine Parler's assertions that it's a "free speech" alternative to Big Tech "censorship"?
• Can Parler maintain a solid user base when its moderation decisions conflict with its stated goals?

Questions and policy implications to consider:

• Does limiting content removal to unprotected speech attract unsavory core users?
• Is it possible to limit moderation to illegal content without driving users away?
• Does promising very little moderation of pornography create risks that the platform will also be filled with content that violates the law, including child sexual abuse material?

Resolution: Parler’s Chief Operating Officer responded to these stories after they were published by insisting that its hands-off approach to pornography made sense, but also claiming that he did not want pornographic “spam.”

After this story was published online, Parler Chief Operating Officer Jeffrey Wernick, who had not responded to repeated pre-publication requests seeking comment on the proliferation of pornography on the site, said he had little knowledge regarding the extent or nature of the nudity or sexual images that appeared on his site but would investigate the issue.

“I don’t look for that content, so why should I know it exists?" Wernick said, but he added that some types of behavior would present a problem for Parler. “We don’t want to be spammed with pornographic content.”

Given how Parler’s stance on content moderation of pornographic material has already changed significantly in the short time the site has been around, it is likely to continue to evolve.

Originally posted to the Trust & Safety Foundation website.

Filed Under: content moderation, free speech, pornography, social media
Companies: parler, twitter

Content Moderation Case Study: Facebook Targets Misinformation Spread By The Philippines Government (2020)

from the misinformation-challenges dept

Summary: Philippines president Rodrigo Duterte's rise to power was greatly aided by Facebook and its overwhelming popularity within the country. An estimated 97% of Filipinos have Facebook accounts and the company itself co-sponsored a Q&A session with local journalists that was broadcast on 200 radio and television stations and livestreamed on the platform. Questions were crowdsourced from Facebook users, helping propel the mayor of Davao to the highest office in the country.

Duterte's run for office was also directly assisted by Facebook, which flew a team of reps in to help the candidate's campaign staff maximize the platform's potential. As his campaign gathered critical mass, he and his team began weaponizing the tools handed to him by Facebook, spreading misinformation about other candidates and directly targeting opponents and their supporters with harassment and threats of violence.

Not much has changed since Duterte took office in 2016. Facebook continues to be his preferred social media outlet. But Facebook's latest attempt to tackle the spread of misinformation on its platform may prompt Duterte to find another outlet to weaponize. In September 2020, Facebook's moderation team announced they had removed a "network" linked to the Philippines government for violating its rules against "coordinated inauthentic behavior."

We also removed 64 Facebook accounts, 32 Pages and 33 Instagram accounts for violating our policy against foreign or government interference which is coordinated inauthentic behavior on behalf of a foreign or government entity. This network originated in the Philippines and focused on domestic audiences. (Updated on October 12, 2020 at 6:35PM PT to reflect the latest enforcement numbers.)

Facebook's removal of this content prompted immediate comments from President Duterte. The president's response to Facebook's moderation efforts was a reminder from Duterte that he has the power to shut the platform down in his country if he believes he's being treated unfairly.

“I allow you to operate here,” Mr. Duterte said. “You cannot bar or prevent me from espousing the objectives of government. Is there life after Facebook? I don’t know. But we need to talk.”

Questions and policy implications to consider:

• Does targeting official government accounts increase the risk of the platform being banned or blocked in targeted countries?
• Does the possible loss of market share affect moderation decisions targeting governments?
• Should Facebook be directly involved in setting up social media campaigns for political figures/government entities?
• Does Facebook have any contingency plans in place to mitigate collateral damage to citizens in countries where the platform has been subjected to retaliatory actions by governments whose content/accounts have been removed?

Resolution: Facebook's Philippines office offered no comment in response to Duterte's threats and the targeted accounts remain deactivated. Facebook is still operating in the country and, although Duterte threatened harsher regulation may be in the works, nothing has surfaced to date.

Originally posted to the Trust & Safety Foundation website.

Filed Under: content moderation, coordinated inauthentic behavior, philippines, rodrigo duterte, trolls
Companies: facebook

Content Moderation Case Study: Using Hashes And Scanning To Stop Cloud Storage From Being Used For Infringement (2014)

from the cloud-storage-scanning dept

Summary: Since the rise of the internet, the recording industry has been particularly concerned about how the internet can and will be used to share infringing content. Over time, the focus of that concern has shifted as the technology (as well as copyright laws) have shifted. In the early 2000s, most of the concern was around file sharing applications, services and sites, such as Napster, Limewire, and The Pirate Bay. However, after 2010, much of the emphasis switched to so-called “cyberlockers.”

Unlike file sharing apps, that involved person-to-person sharing directly from their own computers via intermediary technologies, a cyberlocker was more of a hard drive on the internet. The issue was that some would store large quantities of music files, and then make them available for unlicensed downloading.

While some cyberlockers were built directly around this use-case, at the same time, cloud storage companies were trying to build legitimate businesses, allowing consumers and businesses to store their own files in the cloud, rather than on their own hard drive. However, technologically, there is little to distinguish a cloud storage service from a cyberlocker, and as the entertainment industry became more vocal about the issue, some services started to change their policies.

Dropbox is one of the most well-known cloud storage companies. Wishing to avoid facing comparisons to cyberlockers built off of the sharing of infringing works, the company put in place a system to make it more difficult to use the service for sharing works in an infringing manner, while still allowing the service to be useful for storing personal files.

Specifically, if Dropbox received a DMCA takedown notice for a specific file, the company would create a hash (a computer generated identifier that would be the same for all identical files), and then if you shared any file from your Dropbox to someone else (such as by creating a shareable link), Dropbox would create a hash and check it against the database of hashes of files that had previously received DMCA takedown notices.

This got some attention in 2014 when a user on Twitter highlighted that he had been blocked from sharing a file because of this, raising concerns that Dropbox was looking at everyone’s files.

Dropbox quickly clarified that it is not scanning every file, nor was it looking at everyone’s files. Rather it was using an automated process to check files that were being shared and see if they matched files that had previously been subject to a DMCA takedown notice:

“There have been some questions around how we handle copyright notices. We sometimes receive DMCA notices to remove links on copyright grounds. When we receive these, we process them according to the law and disable the identified link. We have an automated system that then prevents other users from sharing the identical material using another Dropbox link. This is done by comparing file hashes. We don’t look at the files in your private folders and are committed to keeping your stuff safe.”

Decisions to be made by Dropbox:

• How proactive does the company need to be to remain on the compliant side of copyright law?
• Will blocking sharing of files that might be shared for non-infringing purposes, make the service less useful to users?
• What steps are necessary to avoid being accused of supporting infringement by traditional copyright industries?

Questions and policy implications to consider:

• There may be legitimate, non-infringing reasons to share a file that in other contexts may be infringing.
• Is it appropriate for a company to block that possibility?
• What measures could be put in place to allow for those possibilities?
• The recording and movie industries have a history of being aggressive litigants against technologies used for infringement. What level of response is appropriate for new startups and technology companies?
• Will there be limitations on innovation to services like cloud storage imposed by the need to avoid angering certain industries?

Resolution: Dropbox has continued to use a similar setup, and for the most part has avoided being compared to traditional cyberlockers. Since 2014, the issue of DMCA takedowns leading to future blocking of files has not received all that much attention either. There have been a few articles and forum discussions about how it works, with some users looking for workarounds, but for the most part this technological setup appears to have prevented Dropbox from being considered a cyberlocker-style site for infringing file sharing.

Originally published on the Trust & Safety Foundation website.

Filed Under: cloud storage, copyright, cyberlockers, dmca, hashes, private storage, takedowns
Companies: dropbox

Content Moderation Case Study: Dealing With Demands From Foreign Governments (January 2016)

from the gets-tricky-quickly dept

Summary: US companies obviously need to obey US laws, but dealing with demands from foreign governments can present challenging dilemmas. The Sarawak Report, a London-based investigative journalism operation that reports on issues and corruption in Malaysia, was banned by the Malaysian government in the summer of 2015. The publication chose to republish its own articles on the US-based Medium.com website (beyond its own website) in an effort to get around the Malaysian ban.

In January of 2016, the Sarawak Report had an article about Najib Razak, then prime minister of Malaysia, entitled: “Najib Negotiates His Exit BUT He Wants Safe Passage AND All The Money!” related to allegations of corruption that were first published in the Wall Street Journal, regarding money flows from the state owned 1MDB investment firm.

The Malaysian government sent Medium a letter demanding that the article be taken down. The letter claimed that the article contained false information and that it violated Section 233 of the Communications and Multimedia Act, a 1998 law that prohibits the sharing of offensive and menacing content. In response, Medium requested further evidence of what was false in the article.

Rather than responding to Medium’s request for the full “content assessment” from the Malaysian Communications and Multimedia Commission (MCMC), the MCMC instructed all Malaysian ISPs to block all of Medium throughout Malaysia.

Decisions to be made by Medium:

• How do you handle demands from foreign governments to take down content?
• Does it matter which government? If so, how do you determine which governments to trust?
• How do you determine the accuracy of claims from a foreign government regarding things like “false reporting”?
• What are the trade-offs of being blocked entirely by a country?

Questions and policy implications to consider:

• Taking down content that turns out to be credible accusations of corruption can serve to support that corruption and censor important reporting. Yet, leaving up information that turns out to be false can lead to political unrest. How should a website weigh those two sides?
• Should it be the responsibility of websites to investigate who is correct in these scenarios?
• What is the wider impact of an entire website for user generated content being blocked in a country like Malaysia?

Resolution: The entire Medium.com domain remained blocked in Malaysia for over two years. In May of 2018, Najib Razak was replaced as Prime Minister by Mahathir Mohamad (who had been Prime Minister from 1981 to 2003). However, in 2018, he was representing the Pakatan Harapan coalition, which was the first opposition party to the Barisan Nasional coalition to win a Malaysian election since Malaysian independence (Mahathir Mohamad had previously ruled as part of the Barisan Nasional). Part of Pakatan Harapan’s platform was to allow for more press freedom.

Later that month, people noticed that Medium.com was no longer blocked in Malaysia. Soon after, the MCMC put out a statement saying that Medium no longer needed to be blocked because an audit of 1MDB had been declassified days earlier, and once that report was out, there no longer was a need to block the website: “In the case of Sarawak Report and Medium, there is no need to restrict when the 1MDB report has been made public.”

Originally published on the Trust & Safety Foundation website.

Filed Under: blocking, content moderation, foreign governments, malaysia, takedown orders
Companies: medium, sarawak report

Content Moderation Case Study: Yelp Attempts To Tackle Racism On Its Platform (2020)

from the a-challenge dept

Summary: Running a site that relies on third-party content means having to deal with the underside of human existence. While most people engage in good faith, a small minority of people engage with the sole purpose of disparaging others.

Yelp is no exception. Designed to provide potential customers with useful information about goods and services, the site's popularity lent itself to brigading (negative reviews delivered en masse in response to current outrages) and the lowest common denominators of the general public: bigots.

The potential to ruin a business's reputation over their views on immigration policy, their employment of minorities, or other perceived slights made it possible for the most-respected review site to be weaponized by racists.

Yelp recognized this inevitability. Moderators patrol the site to limit the spread of bigoted content that skew review scores based on the racist predilections of reviewers.

Communities have always turned to Yelp in reaction to current events at the local level. As the nation reckons with issues of systemic racism, we’ve seen in the last few months that there is a clear need to warn consumers about businesses associated with egregious, racially-charged actions to help people make more informed spending decisions. Yelp’s User Operations team already places alerts on business pages when we notice an unusual uptick in reviews that are based on what someone may have seen in the news or on social media, rather than on a first-hand experience with the business. Now, when a business gains public attention for reports of racist conduct, such as using racist language or symbols, Yelp will place a new Business Accused of Racist Behavior Alert on their Yelp page to inform users, along with a link to a news article where they can learn more about the incident.

This move may have seemed laudable but it lent itself to subjective interpretations of decisions made by businesses, as well as individual actions by employees. Employing a racist person is not the same as running a racist business, but Yelp's blanket policy seemed to indicate both were equally racist.

Further comments by Yelp clarified some of its employees would make the final determination on alleged racism by businesses or business owners. Any company flagged for racist behavior would be sheltered from further comment until a determination was made.

Decisions to be made by Yelp:

• Does allowing users to unilaterally declare businesses to be "racist" thwart monetization efforts by Yelp?
• Is it wise to succumb to the "wisdom of the crowd," especially when Yelp feels an interstitial warning is an acceptable replacement for due diligence?

Questions and policy implications to consider:

• Does Yelp's reliance on income from businesses seeking to expand their reach conflict with allegations of racism by business owners/employees?
• Do policies like this actually encourage bad faith behavior by hiding reviews behind an ominous warning that suggests the complaints are legitimate?

Resolution: This use of warnings and the hiding of unverified reviews (at least temporarily) is still company policy. While its moderation efforts may eventually lead to a satisfactory resolution, its decision to flag businesses based on unverified claims has the potential to result in a lot of collateral damage.

Originally posted on the Trust & Safety Foundation website.

Filed Under: content moderation, racism, reviews
Companies: yelp

Content Moderation Case Study: SoundCloud Combats Piracy By Giving Universal Music The Power To Remove Uploads (2014)

from the trusted-removers? dept

Summary: Any site that relies on uploaded content has to be wary of hosting pirated content. In most cases, allegedly infringing content is removed at the request of rights holders following the normal DMCA takedown process. A DMCA notice is issued and the site responds by removing the content and -- in some cases -- allowing the uploader to challenge the takedown.

SoundCloud has positioned itself as a host of user-created audio content, relying on content creators to upload original works. But, like any content hosting site, it often found itself hosting infringing content not created by the uploader.

Realizing the potential for SoundCloud to be overrun with infringing content, the platform became far more proactive as it gained users and funding.

Rather than allow the normal DMCA process to work, SoundCloud allowed one major label to set the terms of engagement. This partnership resulted in Universal being able to unilaterally remove content it believed was infringing without any input from SoundCloud or use of the normal DMCA process.

One user reported his account was closed due to alleged infringement contained in his uploaded radio shows. When he attempted to dispute the removals and the threatened shuttering of his account, he was informed by the platform it was completely out of SoundCloud's hands.

Your uploads were removed directly by Universal. This means that SoundCloud had no control over it, and they don't tell us which part of your upload was infringing.

The control of removing content is completely with Universal. This means I can't tell you why they removed your uploads and not others, and you would really need to ask them that question.

Unfortunately, there was no clear appeal process for disputing the takedown, leaving the user without his account or his uploads. A little less than 18 months later, SoundCloud finalized a licensing deal with Universal Music, shortly before the site's subscription service debuted.

Decisions to be made by SoundCloud:

• Does allowing labels to perform their own takedowns bypass protections SoundCloud should be granting to their users?
• Are resulting licensing deals more profitable than subscription fees collected from users?
• Is the risk of litigation too high to allow for a more equitable takedown system?

Questions and policy implications to consider:

• Will alienating/shedding users via skewed takedown processes result in decreased funding in the future?
• Do policies like this make SoundCloud an extension of major labels, rather than a platform that caters to independent creators?

Resolution: SoundCloud continues to allow labels like Universal to perform content removals without utilizing the DMCA process or engaging with the platform directly. Users are still on their own when it comes to content declared infringing by labels. This appears to flow directly from SoundCloud's long-running efforts to secure licensing agreements with major labels. And that appears to flow directly from multiple threats of copyright litigation from some of the same labels SoundCloud is now partnered with.

Originally posted to the Trust & Safety Foundation website.

Filed Under: copyright, moderation, trusted users
Companies: soundcloud, universal music

Content Moderation Case Study: Dealing With Controversial & Sexual Fan Fiction (May 2007)

from the fan-friction dept

Summary: Sexual content can be challenging for content moderation on a number of different levels -- especially when it involves fictional content about taboo, controversial, or even illegal activities. Literary fiction around these topics has been controversial throughout history, including books like Vladimir Nabokov’s Lolita, which focuses on a story told (somewhat unreliably) by a middle-aged male English professor who becomes obsessed with a 12 year-old girl.

But while there have been widespread public debates about whether or not such written works have artistic merit or are obscene, the debate becomes different when such content is hosted on social media platforms, and raises questions about whether or not it complies with terms of service.

LiveJournal, the very popular blogging platform in the mid-2000s, faced that question in 2007. A religious group called “Warriors for Innocence,” that was ostensibly set up to track down child abuse online, launched a public campaign accusing LiveJournal (at the time owned by another blogging company, SixApart) of harboring people promoting child sexual abuse. In response, LiveJournal suspended approximately 500 accounts. Many of the suspended accounts, however, hosted fictional writings, including fan fiction about the Harry Potter universe, as well as a (Spanish-language) LiveJournal that hosted a discussion about Nabokov’s Lolita.

Many of the LiveJournal users were upset about this, and argued that even if they were writing about taboo sexual content, fiction about criminal behavior is quite different than supporting or engaging in the same criminal behavior.

However given that all the material in question is fiction and artwork it seems preposterous to censor these communities. If works of fiction that address illegal or immoral activities are going to be subject to this treatment surely crime thrillers and murder mysteries should be censored just as heavily as erotic material. Part of the reason I use livejournal is because of the freedom it allowed for writers such as myself who deal with difficult and unconventional subject matter. If this purge continues I will be forced to leave livejournal find another outlet for my writing and I am sure I am not the only lj user who feels this way.

Decisions to be made by SixApart:

• How do you distinguish fictional stories that describe criminal activities from supporting or encouraging such activities?
• How responsive should a website be to public campaigns that seek to condemn certain content online?
• Should the company be judging the literary merit of written works?

Questions and policy implications to consider:

• Where, if anywhere, is the line to be drawn between fictional works depicting abuse, and policies against abuse?
• Different people may view these works through very different lenses. How do you handle concerns raised by some, as compared to the creative outlet it provides others?

Resolution: SixApart’s CEO later apologized, saying that they screwed up the removal process.

For reasons we are still trying to figure out what was supposed to be a well planned attempt to clean up a few journals that were violating LiveJournal's policies that protect minors turned into a total mess. I can only say I’m sorry, explain what we did wrong and what we are doing to correct these problems and explain what we were trying to do but messed up so completely.

Many of the suspended journals were put back online, after each was manually reviewed by the company. He admitted that they struggled with some content that “used a thin veneer of fictional or academic interest” to actually promote that activity, and the company sought to shut down those accounts.

Another issue we needed to deal with was journals that used a thin veneer of fictional or academic interest in events and storylines that include child rape, pedophilia, and similar themes in order to actually promote these activities. While there are stories, essays, and discussions that include discussion of these issues in an effort to understand and prevent them, others use a pretext to promote these activities. It’s often very hard to tell the difference.

It is also worth noting that approximately six months after this incident, SixApart sold LiveJournal.

Originally posted on the Trust & Safety Foundation website.

Filed Under: blogging, content moderation, fan fiction, sexual content
Companies: sixapart