Content Moderation Case Study: Dealing With Controversial & Sexual Fan Fiction (May 2007)
from the fan-friction dept
Summary: Sexual content can be challenging for content moderation on a number of different levels -- especially when it involves fictional content about taboo, controversial, or even illegal activities. Literary fiction around these topics has been controversial throughout history, including books like Vladimir Nabokov’s Lolita, which focuses on a story told (somewhat unreliably) by a middle-aged male English professor who becomes obsessed with a 12 year-old girl.
But while there have been widespread public debates about whether or not such written works have artistic merit or are obscene, the debate becomes different when such content is hosted on social media platforms, and raises questions about whether or not it complies with terms of service.
LiveJournal, the very popular blogging platform in the mid-2000s, faced that question in 2007. A religious group called “Warriors for Innocence,” that was ostensibly set up to track down child abuse online, launched a public campaign accusing LiveJournal (at the time owned by another blogging company, SixApart) of harboring people promoting child sexual abuse. In response, LiveJournal suspended approximately 500 accounts. Many of the suspended accounts, however, hosted fictional writings, including fan fiction about the Harry Potter universe, as well as a (Spanish-language) LiveJournal that hosted a discussion about Nabokov’s Lolita.
Many of the LiveJournal users were upset about this, and argued that even if they were writing about taboo sexual content, fiction about criminal behavior is quite different than supporting or engaging in the same criminal behavior.
However given that all the material in question is fiction and artwork it seems preposterous to censor these communities. If works of fiction that address illegal or immoral activities are going to be subject to this treatment surely crime thrillers and murder mysteries should be censored just as heavily as erotic material. Part of the reason I use livejournal is because of the freedom it allowed for writers such as myself who deal with difficult and unconventional subject matter. If this purge continues I will be forced to leave livejournal find another outlet for my writing and I am sure I am not the only lj user who feels this way.
Decisions to be made by SixApart:
- How do you distinguish fictional stories that describe criminal activities from supporting or encouraging such activities?
- How responsive should a website be to public campaigns that seek to condemn certain content online?
- Should the company be judging the literary merit of written works?
- Where, if anywhere, is the line to be drawn between fictional works depicting abuse, and policies against abuse?
- Different people may view these works through very different lenses. How do you handle concerns raised by some, as compared to the creative outlet it provides others?
For reasons we are still trying to figure out what was supposed to be a well planned attempt to clean up a few journals that were violating LiveJournal's policies that protect minors turned into a total mess. I can only say I’m sorry, explain what we did wrong and what we are doing to correct these problems and explain what we were trying to do but messed up so completely.
Many of the suspended journals were put back online, after each was manually reviewed by the company. He admitted that they struggled with some content that “used a thin veneer of fictional or academic interest” to actually promote that activity, and the company sought to shut down those accounts.
Another issue we needed to deal with was journals that used a thin veneer of fictional or academic interest in events and storylines that include child rape, pedophilia, and similar themes in order to actually promote these activities. While there are stories, essays, and discussions that include discussion of these issues in an effort to understand and prevent them, others use a pretext to promote these activities. It’s often very hard to tell the difference.
It is also worth noting that approximately six months after this incident, SixApart sold LiveJournal.
Originally posted on the Trust & Safety Foundation website.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: blogging, content moderation, fan fiction, sexual content
Companies: sixapart
Reader Comments
Subscribe: RSS
View by: Time | Thread
Ah yes, the lolicon debate
Lolicon is crazy popular in Japan, with a fandom that has infected much of the world, including the US. Generally it's recognized like most porn as fantasy and wish-fulfillment. We're not sure why Japan likes Lolicon so much, but its popularity there gave the rest of the world a chance to see what they thought.
Turns out there's fandom everywhere, and according to our moral guardians, that's terrible.
Also throughout much of the world, there's a bit of confusion between art that represents sexuality or even celebrates it and art that encourages sexual misconduct, and as such here in the US the criminality of lolicon (which often depicts child sex abuse) varies from county to county.
Bluer states will have it decriminalized throughout, red states don't like porn or hentai featuring adults, let alone kids... or fictional para-human species that are pedomorphic (aliens elves, demons and spirits, typically). Some states haven't sorted out a statewide position, and so it's decided county to county whether Sandman (Volume 2) #14: Collectors is or is not a felony to have or read.
Australia at the furthest extreme not only bans Lisa Simpson porn but even pictures of small-chested women, lest an Australian be excited to sexual mania and lose control of his sensibilities.
It's only going to get worse as we are able to render realistic human beings at any age and engaged in many activities. On the other hand, if realistic CGI child porn can be decriminalized, it could kill the industry that actually exploits kids (or reduce it to a very niche market).
Meanwhile podcasts are now talking about the two movie productions of Nabokov's Lolita, and their emphasis on romance and the illusion of informed, sound-minded consent, which implies there's more chronophilia interest in our society than we let on.
And don't get me started about child beauty pageants.
[ link to this | view in chronology ]
Re: Ah yes, the lolicon debate
A debate I expect to have for the rest of my life.
My position, it's fiction, bugger off. (Not attacking you Uriel-238 but the anitas, brittany ventis and other twitter moral warriors who decide to open their traps about it.)
Making it illegal in my region of the world won't stop me from enjoying it.
[ link to this | view in chronology ]
Re: Ah yes, the lolicon debate
Japan is also the place that invented underage schoolgirls dating each other for practice, because doing it with actual boys would be icky, then getting confused when said schoolgirls would rather dare each other once they get older.
Love wins.
[ link to this | view in chronology ]
Re: Ah yes, the lolicon debate
Love wins.
[ link to this | view in chronology ]
Fuck off, NAMBLA.
[ link to this | view in chronology ]
Re:
There's no need to. Give it time. Support for the LGBT community was a long time coming, and the truth that being LGBT is natural centered around the Greeks as a core argument. One of the most enlightened, celebrated cultures, which had a social system of pairing boys with older men as teachers and lovers. This was natural, accepted, and encouraged.
Now as a society we are more diverse than ever before. The LGBT acronym has expanded to include innumerable definitions of sexuality, just as it should be. Give it time. Pedarasty will come to be as natural as it was treated millennia ago and conservative values will lose their stranglehold on social norms.
For what it's worth, I'm not a pedarast. Children are a sexually transmitted disease borne from men who can't see that the ideal partner is someone who is naturally equipped to reach the prostate gland. My friends call such foolish males "breeders".
[ link to this | view in chronology ]
I’d insult you, but you’re doing a pretty good job of that by yourself.
[ link to this | view in chronology ]
Um, No.
Support for the LGBT+ community is based on the notion that people should have liberty to do what they want so long as it doesn't harm anyone. That pederasty was embraced by ancient civilizations (or how those civilizations are regarded today) has no bearing on the modern movement of tolerance and recognition that not all people have the same sexual interests.
To be fair, in the Hellenist states, children didn't even have rights; the first regulation of prostitution was in Rome, requiring a child to be at least six before he / she is used as a sex worker. That is to say toddlers were doing sex work before the regulation, and children seven and up would continue to do so. Progress!
Our miserable modern debacle of a civilization is still working out basic matters of consent, because it's still to easy to think it's okay to leverage a colleague's job or reputation for sex, or to take advantage of someone who is inebriated or unconscious.
Industrialized societies are still struggling to figure out what kids should be allowed to do, and with whom. Children seem to want to experiment with their peers, and doing so seems to be a healthy thing -- usually -- but kids also are not clear on consent, so for now we punt and leave that to parents.
We also know kids respond with trust and credulity to people with seniority and authority, which is one of the reasons the industrialized world has a youth-ministry problem.
We know tweens should learn the risks of sexting each other, but not be jailed for distributing child porn when a friend distributes a candid shot they gave a sweetheart. TechDirt is full of articles about that very sort of thing happening.
So your dreams of swimming with Little Fishes as did Tiberius Caesar are not likely to manifest in this society Anonymous Coward, at least not legally.
[ link to this | view in chronology ]
Re: Um, No.
It's fascinating to see people cherry-pick and espouse millennia old practices that are wrong on so many levels, even in jest. What they don't understand is that if they pretend to be idiots to get their jollies they will soon find that the company they keep only consists of real idiots.
[ link to this | view in chronology ]
Re: Re: Um, No.
Kind of you to cherry pick what millenia old practices you feel are wrong. Seems like we should just wait a few more and see if we can find any more.
[ link to this | view in chronology ]
I hope you’re not saying that because you think child rape is justified.
[ link to this | view in chronology ]
Re:
Well, he either is to stupid to get the context of the thread...or he actually thinks child rape is ok.
[ link to this | view in chronology ]
Re: Re: Um, No.
It's similar to other individuals who espouse views of a "glorious past", which weren't anywhere near as sunny as they think it was.
If the best defense is some people (probably in positions of authority on their slave) in a barbaric and primitive society practiced it, perhaps you don't have a very good defense for such practices? Rome had gladiators fight against lions in the coliseum for entertainment. Is this humane? Should we bring it back?
The risk of exploitation is too high. And under a certain age, it can cause severe psychological, and possibly, physical damage. These things cannot be understated. If you do something to a robot, it's not a sentient and vulnerable being, I could care less what you do it, even if Congress would rather denounce that.
[ link to this | view in chronology ]
Re:
Hahaahh! Always count on you for the Hot Takes Of The Desperate And Judgmental!
[ link to this | view in chronology ]
Yes, “desperate” is a good descriptor — for your insurrectionist friends in D.C. yesterday.
[ link to this | view in chronology ]
Re:
In the interest of preserving the severity of an armed insurrection against the will of the people, can we refrain from degrading it to the status of a derogatory comment?
Thank you.
[ link to this | view in chronology ]
Re: Ah yes, the lolicon debate
You're talking about the decision by the classification board in Australia ("small boob ban")? I don't know if that is still in effect, but it is a moot effort, as someone can just access mainstream porn sites, which almost certainly would contain the content, and I doubt someone would be arrested for a lot of these things (unless, they're looking at lolicon, particularly lolicon of eight year olds).
[ link to this | view in chronology ]
Re: Re: Ah yes, the lolicon debate
Lolicon ranges from infants to grown teens. And then there are those who are in the bodies of child or adolescent humans but are really elves or thousand-year-old spirits.
Here in the states, there are plenty of things we Americans could get arrested for (and incarcerated as long as we imprison murderers). We usually don't thanks to prosecutorial discretion but if someone wants a guy put way (say because he embarrassed a VIP, or the sheriff wants his land or because his skin is too dark) then these matters come up.
So yeah, laws that aren't enforced except sometimes are notorious for creating chilling effects and making undesirables disappear. At least here in the US, and lately Australia has been aspiring to be as ugly and cruel as the States.
[ link to this | view in chronology ]
Re: Ah yes, the lolicon debate
Realistic CGI child porn (which is indistinguishable from reality) would have to be decriminalized at the federal level. There is blocking Supreme Court precedent.
It would make it really hard to find and remove (and arrest) child porn (as any piece of apparent "child porn" could really be a simulation someone made on their laptop when they were bored), which would almost certainly sink such an effort. I feel the government is more of the persuasion to eradicate child porn no matter what, regardless of the cost.
[ link to this | view in chronology ]
Is it real or CGI
This became a matter of issue kick-started by Ashcroft's eagerness to curb pornography and indecent material (leading to Eric Idle's second song about radio censorship.†). Famously, Ashcroft required the breast of Iustitia to be covered for press briefings.
For a short time, thanks to many efforts to censor porn, the courts decided the US had no cause to criminalize CGI rendered porn, even if it would be illegal were it to involve models (child, snuff, rape, revenge, etc.) This created a market of processed porn, in which photographs of children would then be artified -- rendered to look drawn or created with graphics software. Rather than creating narrower categories of acceptable and unacceptable media, the courts just returned CGI porn to be regarded like photographed porn.
So yeah, it's unlikely CGI will be revisited anytime in the near future, though in the 2010s law enforcement started using rendered little girls to lure online child-porn enthusiasts into sting traps.
† Mr. Idle added part two in 2018!
[ link to this | view in chronology ]
Apropos of nothing, to Mike or whoever handles the HTML around here: The HTML for the "Time" link next to the "View in" text above the comments is broken. I assume it’s a template issue since it’s that way on every page. Just lettin’ you know.
[ link to this | view in chronology ]
So I rolled in here, just out of curiosity to see if techdirt had anything to say about recent events, and there's a content moderation post.
Don't ever change techdirt.
[ link to this | view in chronology ]
Recent events
Are you new here?
Be patient. In the meantime check your news listings for unfolding developments.
TD mostly does post-hoc analysis.
[ link to this | view in chronology ]
Re:
https://www.techdirt.com/articles/20180810/17491840412/defense-slow-news.shtml
[ link to this | view in chronology ]
Who dont Love the debate of Who is correct about MORALS?
Even the religion we declare we Love so much, has so many different ideas in it, that HOW in Hell does 1-2 groups out of 40+ decide WHO/WHAT is right or wrong?
Its strange that no one knows What fruit was eaten in Eden. But we Can declare it to be anything we wish. at least some can.
But they will Never talk about BEFORE, that fruit. When there were NO RULES, except 1.
And the big debate of what to name that tree, it came from.
Tree of knowledge?
Forbidden knowledge?
Of knowledge of good and evil?
Iv heard 3-4 different names and its always something abit off about that name.
before that point there was NO GUILT.
Now understanding that. Think about all the strange things MAN has done to MAN, over time. And thats a long history of "who is RIGHT?", and who do you enslave.
[ link to this | view in chronology ]
Morality and religion
Not long ago the internet was buzzing about trolley problems and I added my take and came to some conclusions informed partly about how our religious communities (who are all very big on creeds, or deontological ethics) were still able to take a very pragmatic approach to the 2016 and 2020 election, and suspend their judgement for the miscreant they voted for. (White Protestant Evangelists and Catholics both voted 80% for renowned adulterer and racketeer Donald Trump even though Clinton was significantly less creepy.)
My conclusions were (TLDR):
~ People don't adhere to codes of morality, whether they are religious or ethicists or not. They do what they feel.
~ But that isn't particularly horrible. We tend to have high respect and regard for our immediate community and only transgress out of desperation. Also, we're freaked out by weirdos and nations of millions. And...
~ This informs why we tend to stand by our cohorts in The Prisoner's Dilemma even when we don't trust our buddy or even sacrifice ourselves so our mates might live. (Incidentally, other social animals do as well.)
[ link to this | view in chronology ]
Re: Morality and religion
People have priorities. Some things are weighed heavier than others.
He pushes policies no other president would be able to push. And he has a certain charisma which jibes with his base. That charisma is his confidence. No matter what he does, he has the utmost confidence behind it, and that is contagious.
So long as he does what they want, the oddities and idiocy can be overlooked by them. And by playing the fool, he plays the role of someone who can slam through terrible policy, but who no one wants to argue with, because you are literally arguing with a fool.
[ link to this | view in chronology ]
I have to question if these people find it similarly "hard to tell the difference" in all the fiction involving crimes of a non-sexual nature.
[ link to this | view in chronology ]
Good question! I believe there are multiple parties at play here.
Pedophilia could be considered a sexual disorder, or even sexual orientation (this doesn't mean it is a good thing, or that it should be acted upon) in the sense that it's an innate trait, it's stable over time, and it contains elements of romantic attraction, rather than merely just being a sexual reaction to a particular stimulus.
There are variants of it too. But, this is a basic model to think about. The clinical criteria goes as far as requiring someone's attraction to children to overpower their attraction to adults, or for it to have a significant impact on their lives, which they otherwise cannot navigate around. Some have absolutely zero attraction to adults, and sometimes even actual revulsion.
I don't believe many people who consume this content meet this specific criteria. But, perhaps there are some who do. And there are yet some teenagers who may have fantasies about engaging in activities in peers, who most certainly would not be considered to be pedophilic. Whether they should be engaged in uploading, or consuming fantasies is a whole other debate, altogether involving factors like sex education.
I don't believe people are convinced into committing terrible acts by consuming fictional content. Rather, people have largely been socialized by society to see pedophilic activities as bad. I am sure you too hold this opinion, no?
Most people would be so socialized, that any depiction of "happy consensual relationships" in a piece of fiction, would be seen as the fantasy that it is. As for depictions which depict rape or clearly non-consensual themes, they wouldn't of their own existence convince someone that suffering is good, and that they should go about and do it.
Even in the case of pedophiles, they have an attraction to children, therefore they look at content involving children. It isn't the content that makes them like that, nor the demand. Depriving them of the content meaninglessly may even serve to frustrate them, and further deepen their adversarial relationship with society, which could lead to an increased number of clashes.
https://www.sciencedaily.com/releases/2010/11/101130111326.htm
Some research even suggests that some material (perhaps artificial) might even serve to reduce crime, although this sort of research is highly stigmatized, and very few are willing to engage in it.
So, what does convince someone to commit a crime then? If we apply Maslow's Hierarchy of Needs, human beings have a certain number of needs, which they need to meet to reach fulfillment. A pedophile obviously has unmet sexual needs. They may also find rationalizations for their behavior.
A fictional media (if it were sufficient for their needs) would waste time they could be spending with actual children, burn the energy they would need to do so, and give them a minuscule, but certain level of fulfillment in their lives. Producing such content might even give their lives some semblance of purpose and meaning, which might otherwise be spent pursuing anti-social goals. This is not an uncommon thing, and a report written in Europe, which I currently cannot locate has theorized as such.
https://www.theatlantic.com/health/archive/2016/01/can-child-dolls-keep-pedophiles-from-offend ing/423324/
There are also some reports that other aids like child sex dolls can also help to prevent pedophiles from committing crimes. Dr. Cantor's Non-Offending Pedophiles Paper also makes note of this.
Rationalizations for behavior which lead to crime could come in the form of nambla-like rhetoric, which try to downplay the damage which sexual activity does to children, and may even try to spin it as a "positive thing". However, this is not a piece of fiction which is doing this, but a very specific piece of advocacy from a specific group of people, who may employ the power of persuasion, and cite largely discredited research. Others who are of a more anti-social persuasion may be even more utilitarian, and lack the need for a "belief" to make themselves feel better about their actions.
In this way, I believe there is a very big difference between advocacy from an organization, and whether a specific piece of fiction promotes a specific activity. I truly hope that interactions with nambla-like individuals does not color your thoughts on this matter.
Frankly speaking, nambla-like ideology could even be seen as a sort of religion, for desperate people facing a societal dead-end.
https://www.vice.com/en/article/59nbnq/first-man-sentenced-to-chemical-castration-in-indon esia-said-he-would-rather-die
Of course, there are things which are purported to be solutions, but they involve a great degree of suffering, and due to the coercive nature of society pressing down on them, it may even lead to resentment which could lead to worse conditions like mysopedia.
Regardless of that, it is simply not right to not afford people the proper due process and freedoms here. Just because someone who bears a similarity to them (even in a superficial way for those who aren't really pedophiles), it doesn't mean they should have to suffer for the acts of someone else.
Sometimes, individuals who do commit such acts also have other mental disorders, which could have been treated at a far earlier time. But, which aren't due to flaws in the mental healthcare system.
Many pedophiles, like most people really, probably realize that their actions would have a terrible impact on children, and avoid engaging in sexual activity with them. Unfortunately, some of the loudest people also tend to be the least scrupulous, or the most deluded.
[ link to this | view in chronology ]
The money quote -- extending Masnick's impossibility Theorem
SixApart’s CEO:
While there are stories, essays, and discussions that include discussion of these issues in an effort to understand and prevent them, others use a pretext to promote these activities. It’s often very hard to tell the difference.
Not only can a few words of context can change the intent of a piece of content, but the audience has a mind of its own and can react exactly opposite to the intent, or in any other direction for that matter.
As an example, we are so tied up in knots about pornography that I don't think we have solid data on its actual effects on people's behavior -- it's really unclear in which way what causes might flow, except that the ability to sell or benefit somehow seems to cause people to make content.
I'm wondering if the ethical answer to child porn, for example, isn't to flood the market so there's little external incentive to make it, and those that need it act on the fantasy rather than real children. Is that what is happening with Lolicon in Japan?
[ link to this | view in chronology ]
This looks like a classic response to outside pressure. When someone says you're not doing enough, you over react to make it look like you're "doing something", and smash any account in your line of fire.
Ironically, the accounts most likely to be in the line of fire are less likely to be the most problematic ones to begin with. This may too be an example of "moderation at scale is hard". Tumblr nuked all the porn on their platform to try to get at a few pieces of child porn.
Even if we agree that action was necessary, for child porn being an absolute evil (which it is, although we can't shutdown society and free expression for it, it will always exist, and we have to accept that on some level), a simple nudity filter trained on real human beings would have done the exact same job, and many furry / hentai artists would not have been affected.
But, no one ever thinks of the collateral damage from their policies, or even if they entirely make sense.
[ link to this | view in chronology ]