Content Moderation Case Study: Chatroulette Leverages New AI To Combat Unwanted Nudity (2020)
from the as-opposed-to-the-wanted-kind dept
Summary: Chatroulette rose to fame shortly after its creation in late 2009. The platform offered a new take on video chat, pairing users with other random users with each spin of the virtual wheel.
The novelty of the experience soon wore off when it became apparent Chatroulette was host to a large assortment of pranksters and exhibitionists. Users hoping to luck into some scintillating video chat were instead greeted with exposed penises and other body parts. (But mostly penises.)
This especially unsavory aspect of the service was assumed to be its legacy -- one that would see it resigned to the junkheap of failed social platforms. Chatroulette attempted to handle its content problem by giving users the power to flag other users and deployed a rudimentary AI to block possibly-offensive users.
The site soldiered on, partially supported by a premium service that paired users with other users in their area or who shared the same interests. Then something unexpected happened that drove a whole new set of users to Chatroulette: the COVID-19 pandemic. More people than ever were trapped at home and starved for human interaction. Very few of those were hoping to see an assortment of penises.
Faced with an influx of users and content to moderate, Chatroulette brought in AI moderation specialist Hive, the same company that currently moderates content on Reddit. With Chatroulette experiencing a resurgence, the company is hoping a system capable of processing millions of frames of chat video will keep its channels clear of unwanted content.
Decisions to be made by Chatroulette:
- Is it possible to filter live content quickly and accurately enough to prevent a return to the "old" Chatroulette?
- Is the cost of moderation AI affordable given the site's seeming inability to attract or sustain a large user base?
- If user growth continues, will it still be possible to backstop AI moderation with human moderators?
- What metrics should Chatroulette consider as measures of success here?
- Is over-moderation a foreseeable problem, given the challenges of moderating live video streams?
- Is it possible to attract a more-dedicated user base while still respecting their apparent desire for anonymity?
- Is it wise to maintain an unmoderated channel given the historical issues the site has had with unsolicited nudity and its exposure to/of minors?
Originally published on the Trust & Safety Foundation website.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: ai, content moderation, live video
Companies: chatroulette, hive
Reader Comments
Subscribe: RSS
View by: Time | Thread
Penii are a goto thing for online 'pranks'... I mean they are right at hand.
CheezBurger at one point had chatroulette memes & funny screen shots...
The Eye of Sauron connecting with Frodo
2 jokers showing up in the same virtual teddy bear masking
Response to someones sign show me your tits & I'll show you my dick being met with someone holding up a penis shaped bong asking if that will do
Steve Kardynal who lipsynced several popular songs with dramatic sets & staging...
https://www.youtube.com/watch?v=InYvRyX2Fu4
As with many things people focused only on a few who ran around showing off their penii & none of the other things happening.
As the meme says...
https://imgur.com/gallery/x5UgAMi?nc=1
[ link to this | view in chronology ]
But the site also points out that only a little more than 3% of its millions of monthly interactions contain explicit material.
Damn, doing the math on that is still tens of thousands of penises every month. I don’t think that’s a very good look for the rebrand.
[ link to this | view in chronology ]
Re:
There is no possible way got technology to catch every penis.
They can't hire half the planet to monitor the other half of the planet to make sure a penis is never seen.
People think technology can do the impossible & refuse to accept it can't be done.
We spent millions training TSA agents to look at people & spot terrorists, not heard of any caught... but we did have a bunch of TSA agents helping smuggle drugs & weapons that went undetected.
The government is 'accidentally' collecting all of our data & despite the claims of 'going dark' they completely dropped the ball on several high profile attacks including Jan 6, despite claiming the had to have the data. (One would mention here that 'Thin Thread' detected the 911 hijackers when fed the data that was available before the attack, the giant machinery we have now can't detect anything.)
If you use Chatroullete you might see a penis/breasts/vagina/butts and 300 other bad things... if you are that upset by the chance of seeing them... don't use the platform.
Do not expect that a platform can protect you, they can't.
Know the risks and act accordingly, but instead we demand a completely risk free world where nothing is ever our fault. (See also: Well if they had used enough tech to detect they were in a moving car & blocked texting then my child speeding down the freeway texting would never have killed that family of 5 so we are suing apple.)
Y'all can hit your head in the shower, fall down face down in like 3/4 inch water and die... should we sue cities for providing water, or tub makers, or god for not giving us gills?? We could pass a law requiring everyone have a nonslip mat in all tubs... it'll work out just as well as the drunk driving laws work.
[ link to this | view in chronology ]
Re: Re:
"There is no possible way got technology to catch every penis.
They can't hire half the planet to monitor the other half of the planet to make sure a penis is never seen."
There's a Cellmate(TM) joke in there somewhere. If you don't want something to be seen, just lock it up and hand the access key to the relevant authorities...
[ link to this | view in chronology ]
Re:
"I don’t think that’s a very good look for the rebrand."
Hence why Chatroulette now wants an AI to cockblock it's user base.
...I think that read better inside my head than it did in the thread...🤔
[ link to this | view in chronology ]
Re:
It also means you have a better than 50% chance of seeing at least one penis after 23 sessions.
[ link to this | view in chronology ]
Re:
That's almost a mile of dicks per million interactions.
[ link to this | view in chronology ]
Unwanted nudity?
Says who?
[ link to this | view in chronology ]
Re: Unwanted nudity?
As an image I keep in my folder of "dick pics" says...
Whoever said,
"All men are created equal",
obviously has never received any dick pics.
[ link to this | view in chronology ]
Re: Re: Unwanted nudity?
"As an image I keep in my folder of "dick pics" says..."
As someone with a folder of "Cthulhu & Kitten images" I shouldn't really comment, but...collectors are a weird bunch.
[ link to this | view in chronology ]