Content Moderation Case Study: YouTube Deals With Disturbing Content Disguised As Videos For Kids (2017)
from the finding-the-monsters-among-us dept
Summary: YouTube offers an endless stream of videos that cater to the preferences of users, no matter their age and has become a go-to content provider for kids and their parents. The market for kid-oriented videos remains wide-open, with new competitors surfacing daily and utilizing repetition, familiarity, and strings of keywords to get their videos in front of kids willing to spend hours clicking on whatever thumbnails pique their interest, and YouTube is leading this market.
Taking advantage of the low expectations of extremely youthful viewers, YouTube videos for kids are filled with low-effort, low-cost content - videos that use familiar songs, bright colors, and pop culture fixtures to attract and hold the attention of children.
Most of this content is innocuous. But a much darker strain of content was exposed by amateur internet sleuths, which was swiftly dubbed "Elsagate," borrowing the name of the main character of Disney's massively popular animated hit, Frozen. At the r/ElsaGate subreddit, redditors tracked down videos aimed at children that contained adult themes, sexual activity, or other non-kid-friendly content.
Among the decidedly not-safe-for-kids subject matter listed by r/ElsaGate are injections, gore, suicide, pregnancy, BDSM, assault, rape, murder, cannibalism, and use of alcohol. Most of these acts were performed by animated characters (or actors dressed as the characters), including the titular Elsa as well as Spiderman, Peppa Pig, Paw Patrol, and Mickey Mouse. According to parents, users, and members of the r/Elsagate subreddit, some of this content could be accessed via the YouTube Kids app — a kid-oriented version of YouTube subject to stricter controls and home to curated content meant to steer child users clear of adult subject matter.
Further attention was drawn to the issue by James Bridle's post on the subject, entitled "Something is Wrong on the Internet." The post — preceded by numerous content warnings — detailed the considerable amount of disturbing content that was easily finding its way to youthful viewers, mainly thanks to its kid-friendly tags and innocuous thumbnails.
The end result, according to Bridle, was nothing short of horrific:
“To expose children to this content is abuse. We’re not talking about the debatable but undoubtedly real effects of film or videogame violence on teenagers, or the effects of pornography or extreme images on young minds, which were alluded to in my opening description of my own teenage internet use. Those are important debates, but they’re not what is being discussed here. What we’re talking about is very young children, effectively from birth, being deliberately targeted with content which will traumatise and disturb them, via networks which are extremely vulnerable to exactly this form of abuse. It’s not about trolls, but about a kind of violence inherent in the combination of digital systems and capitalist incentives. It’s down to that level of the metal.” — James Bridle
"Elsagate" received more mainstream coverage as well. A New York Times article on the subject wondered what had happened and suggested the videos had eluded YouTube's algorithms that were meant to ensure content that made its way to its Kids channel was actually appropriate for children. YouTube's response when asked for comment was that this content was the "extreme needle in the haystack," perhaps, an immeasurably small percentage of the total amount of content available on YouTube Kids. Needless to say, this answer did not make critics happy, and many suggested the online content giant rely less on automated moderation when dealing with content targeting kids.
Company Considerations:
- How should content review and moderation be different for content targeting younger YouTube users?
- How could a verification process be deployed to vet users creating content for children?
- What processes can be used to make it easier to find and remove/restrict content that appears to be kid-friendly but is actually filled with adult content?
- When content like what was described in the case study does get through the moderation process, what can be done to restore the trust of users, especially those with younger children?
Issue Considerations:
- Should a product targeting children be siloed off from the main product to ensure the integrity of the content, as well as make it easier to manage moderation issues?
- Does creating a product specifically for children increase the chance of direct regulation or intervention by government entities? If so, how can a company prepare itself for this inevitability?
- If creating a “restricted” product for children, should it require all content be fully and thoroughly vetted? If so, would that become prohibitively costly, making it significantly less likely that companies will create products for children? Is there a way to balance those things?
Resolution: Immediately following these reports, YouTube purged content from YouTube Kids that did not meet its standards. It delisted videos and issued new guidelines for contributors. It added a large number of new human moderators, bringing its total of moderators to 10,000. YouTube also removed the extremely popular "Toy Freaks" channel, which users had suggested contained child abuse, after investigating its content.
YouTube wasn't the only entity to act after the worldwide exposure of "Elsagate" videos. Many of these videos originated in China, prompting the Chinese government to block certain search keywords to limit local access to the disturbing content, as well as shuttering at least one company involved in the creation of these videos.
Originally posted to the Trust & Safety Foundation website.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: content moderation, elsagate, kids
Companies: youtube
Reader Comments
Subscribe: RSS
View by: Time | Thread
What always surprised me the most about this stuff was the lack of IP-maximalist content houses failing to go batshit crazy over the use of their copyrighted and trademarked characters and stuff.
I mean, they'll send someone after a guy in a Spiderman outfit at a birthday party.
[ link to this | view in chronology ]
Since Youtube is a common carrier, and all this content is legal, Youtube has no business taking it down, right?
[ link to this | view in chronology ]
Re:
Nice try, John Smith.
[ link to this | view in chronology ]
Re:
Wrong.
[ link to this | view in chronology ]
Re:
"A common carrier offers its services to the general public under license or authority provided by a regulatory body, which has usually been granted "ministerial authority" by the legislation that created it."
What international license is there?
Every nation has its rules, and being able to deal with all of them and require them of Every person on the net??
You wouldnt have an internet.
YOU would have no rights, into another nation using the internet. Build the same walls we have as nations?
[ link to this | view in chronology ]
Are there no parents left?
When I was a kid I watched Rocky and Bullwinkle, where they'd blow up Boris and Natasha on a regular basis. I also watched the coyote fall of cliffs every day. I also watched Mutual of Ohama's Wild Kingdom and I swear I've seen every type of carnivore eat everything with blood and guts everywhere. The best nature photo I've seen recently was of a leopard with its head covered in blood after removing it from the guts of its prey.
Did it turn me into a serial killer? No.
Did it turn me into a recluse, scared of everything? No.
Did it desensitize me to violence? No.
Maybe I was just raised properly.
[ link to this | view in chronology ]
Re: Are there no parents left?
Being "raised properly" is a pretty loaded statement.
I'm 40, so I grew up everything from My little Ponies, Looney Tunes, TMNT, Ren & Stimpy, and Beevis & Butthead in my tweens. My home was mostly stable, but even if it wasn't I doubt cartoons would have been a factor in how I turned out.
But I'm also a parent now. And those thumbnails look just like Little Angel/ Little World, and even similar to Cocomelon musical cartoons that my daughter loves. Now I have to pre-approve every video she can see on her Amazon tablet, but not our computers. Some times, I take advantage of the opportunity to poop alone. Or finish something, anything that I started without 35 interruptions. And I'm going to be hella pissed if I come back to my 3 yr old watching a drunken Elsa injecting Spidey babies with something to get them high, in some disturbing version of pregnancy, with Spiderman slapping her around and humping her. It's a pretty undesirable situation to end up in her imaginative play, though probably harmless on its own, we would still have to address what is wrong in an age appropriate way. But that's not my concern.
When I was little, my mom overhead me playing with a little girl saying we needed to get money to bail dad out of jail. Her home life was a little rough. So was mine, but I didn't know about the multiple times my dad had to be bailed out because he was a drunk, it was hidden from me. My mom didn't want me to play with her after that. Later on, when I turned 8 or 9, we watched Beetlejuice at my slumber party. And another little girl's mom didn't want her to play with me anymore. And as uptight as parents are now a days!?! I don't give a rat's ass what some parent thinks of me, but I don't want my kid to lose out on friendships it.
So where are the parents? Right here, busting out butts, trying not to screw up too badly.
[ link to this | view in chronology ]
Alcohol and pregnancy really got a bad deal being placed in that group.
I remember watching a lot of Happy Tree Friends as a kid. I can't imagine it was particularly good for my brain, but damn watching cartoon violence was fun.
[ link to this | view in chronology ]
Ella and Spidey get married and have ice-spider babies?
That's crazy!
Everyone knows Ella is batting for the other team.
[ link to this | view in chronology ]
Re: Ella and Spidey get married and have ice-spider babies?
your freeze ray has no effect. ice spider babies do 10d4 to your character. are you wearing anything vs cold or venom?
[ link to this | view in chronology ]
Took away a favorite!
Low and behold, not an hour after I read this, my daughter is trying to watch a favorite Little Angel video on her Fire Tablet, and it says that it has been marked private. This is an older video, or collection of them, and the thumbnail looks just like the examples above, except it's not Elsa.
The song is "Oops! what's that sound?", And it's a baby and mom singing out body sounds in a super innocent and cute way. It's not vulgar, it's nothing more than an example of the sound and then a very simple explanation. The worst of it is a short little fart sound, and it's referred to as a poot, explains that it's normal , could mean that you need to go potty. I'm an overgrown child so I giggle about it, but it really is very innocent.
The only other song that might be questionable, or picked up on an algorithm is a song about stinky socks, which is very silly, and very innocent as well. My little one doesn't understand why we can't watch it any more and she is really upset.
Now I had seen some questionable videos similar to those thumbnails, especially when searching for more girl centered content. And I was also surprised that Little Angel and lookalikes got away with using the likenesses of Disney princesses. But sorta seems like they should have left alone known content creators. I bet that horrid Little Baby Bum video that has the song about a kid putting a cat down a well hasn't been yanked.(otherwise I like Little Baby Bum). But content moderation at scale is never easy.
[ link to this | view in chronology ]