Techdirt's think tank, the Copia Institute, is working with the Trust & Safety Professional Association and its sister organization, the Trust & Safety Foundation, to produce an ongoing series of case studies about content moderation decisions. These case studies are presented in a neutral fashion, not aiming to criticize or applaud any particular decision, but to highlight the many different challenges that content moderators face and the tradeoffs they result in. Find more case studies here on Techdirt and on the TSF website.

Content Moderation Case Study: Chatroulette Leverages New AI To Combat Unwanted Nudity (2020)

from the as-opposed-to-the-wanted-kind dept

Summary: Chatroulette rose to fame shortly after its creation in late 2009. The platform offered a new take on video chat, pairing users with other random users with each spin of the virtual wheel.

The novelty of the experience soon wore off when it became apparent Chatroulette was host to a large assortment of pranksters and exhibitionists. Users hoping to luck into some scintillating video chat were instead greeted with exposed penises and other body parts. (But mostly penises.)

This especially unsavory aspect of the service was assumed to be its legacy -- one that would see it resigned to the junkheap of failed social platforms. Chatroulette attempted to handle its content problem by giving users the power to flag other users and deployed a rudimentary AI to block possibly-offensive users.

The site soldiered on, partially supported by a premium service that paired users with other users in their area or who shared the same interests. Then something unexpected happened that drove a whole new set of users to Chatroulette: the COVID-19 pandemic. More people than ever were trapped at home and starved for human interaction. Very few of those were hoping to see an assortment of penises.

Faced with an influx of users and content to moderate, Chatroulette brought in AI moderation specialist Hive, the same company that currently moderates content on Reddit. With Chatroulette experiencing a resurgence, the company is hoping a system capable of processing millions of frames of chat video will keep its channels clear of unwanted content.

Decisions to be made by Chatroulette:

  • Is it possible to filter live content quickly and accurately enough to prevent a return to the "old" Chatroulette?
  • Is the cost of moderation AI affordable given the site's seeming inability to attract or sustain a large user base?
  • If user growth continues, will it still be possible to backstop AI moderation with human moderators? 
  • What metrics should Chatroulette consider as measures of success here?
Questions and policy implications to consider:
  • Is over-moderation a foreseeable problem, given the challenges of moderating live video streams?
  • Is it possible to attract a more-dedicated user base while still respecting their apparent desire for anonymity?
  • Is it wise to maintain an unmoderated channel given the historical issues the site has had with unsolicited nudity and its exposure to/of minors?
Resolution: Chatroulette's new moderation efforts appear to be successfully distancing it from its inauspicious beginnings. However, the site's operating team still wryly acknowledges the reputation for nudity that defined the service for much of the last decade. But the site also points out that only a little more than 3% of its millions of monthly interactions contain explicit material, indicating Chatroulette feels the way forward is offering something that might have seemed boring ten years ago: a predictable and safe random chat experience.

Originally published on the Trust & Safety Foundation website.

Hide this

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: ai, content moderation, live video
Companies: chatroulette, hive


Reader Comments

Subscribe: RSS

View by: Time | Thread


  1. icon
    That Anonymous Coward (profile), 24 Feb 2021 @ 4:02pm

    Penii are a goto thing for online 'pranks'... I mean they are right at hand.

    CheezBurger at one point had chatroulette memes & funny screen shots...
    The Eye of Sauron connecting with Frodo

    2 jokers showing up in the same virtual teddy bear masking

    Response to someones sign show me your tits & I'll show you my dick being met with someone holding up a penis shaped bong asking if that will do

    Steve Kardynal who lipsynced several popular songs with dramatic sets & staging...

    https://www.youtube.com/watch?v=InYvRyX2Fu4

    As with many things people focused only on a few who ran around showing off their penii & none of the other things happening.

    As the meme says...

    https://imgur.com/gallery/x5UgAMi?nc=1

    link to this | view in thread ]

  2. identicon
    Anonymous Coward, 24 Feb 2021 @ 5:58pm

    But the site also points out that only a little more than 3% of its millions of monthly interactions contain explicit material.

    Damn, doing the math on that is still tens of thousands of penises every month. I don’t think that’s a very good look for the rebrand.

    link to this | view in thread ]

  3. icon
    That Anonymous Coward (profile), 24 Feb 2021 @ 7:06pm

    Re:

    There is no possible way got technology to catch every penis.
    They can't hire half the planet to monitor the other half of the planet to make sure a penis is never seen.

    People think technology can do the impossible & refuse to accept it can't be done.

    We spent millions training TSA agents to look at people & spot terrorists, not heard of any caught... but we did have a bunch of TSA agents helping smuggle drugs & weapons that went undetected.

    The government is 'accidentally' collecting all of our data & despite the claims of 'going dark' they completely dropped the ball on several high profile attacks including Jan 6, despite claiming the had to have the data. (One would mention here that 'Thin Thread' detected the 911 hijackers when fed the data that was available before the attack, the giant machinery we have now can't detect anything.)

    If you use Chatroullete you might see a penis/breasts/vagina/butts and 300 other bad things... if you are that upset by the chance of seeing them... don't use the platform.

    Do not expect that a platform can protect you, they can't.

    Know the risks and act accordingly, but instead we demand a completely risk free world where nothing is ever our fault. (See also: Well if they had used enough tech to detect they were in a moving car & blocked texting then my child speeding down the freeway texting would never have killed that family of 5 so we are suing apple.)

    Y'all can hit your head in the shower, fall down face down in like 3/4 inch water and die... should we sue cities for providing water, or tub makers, or god for not giving us gills?? We could pass a law requiring everyone have a nonslip mat in all tubs... it'll work out just as well as the drunk driving laws work.

    link to this | view in thread ]

  4. identicon
    Pixelation, 24 Feb 2021 @ 7:21pm

    Unwanted nudity?

    Says who?

    link to this | view in thread ]

  5. icon
    That Anonymous Coward (profile), 25 Feb 2021 @ 12:09am

    Re: Unwanted nudity?

    As an image I keep in my folder of "dick pics" says...

    Whoever said,
    "All men are created equal",
    obviously has never received any dick pics.

    link to this | view in thread ]

  6. icon
    Scary Devil Monastery (profile), 25 Feb 2021 @ 3:08am

    Re:

    "I don’t think that’s a very good look for the rebrand."

    Hence why Chatroulette now wants an AI to cockblock it's user base.

    ...I think that read better inside my head than it did in the thread...🤔

    link to this | view in thread ]

  7. icon
    Scary Devil Monastery (profile), 25 Feb 2021 @ 3:12am

    Re: Re: Unwanted nudity?

    "As an image I keep in my folder of "dick pics" says..."

    As someone with a folder of "Cthulhu & Kitten images" I shouldn't really comment, but...collectors are a weird bunch.

    link to this | view in thread ]

  8. icon
    Scary Devil Monastery (profile), 25 Feb 2021 @ 3:14am

    Re: Re:

    "There is no possible way got technology to catch every penis.
    They can't hire half the planet to monitor the other half of the planet to make sure a penis is never seen."

    There's a Cellmate(TM) joke in there somewhere. If you don't want something to be seen, just lock it up and hand the access key to the relevant authorities...

    link to this | view in thread ]

  9. icon
    nasch (profile), 25 Feb 2021 @ 2:55pm

    Re:

    doing the math on that is still tens of thousands of penises every month.

    It also means you have a better than 50% chance of seeing at least one penis after 23 sessions.

    link to this | view in thread ]

  10. identicon
    TheDumberHalf, 26 Feb 2021 @ 11:58am

    Re:

    That's almost a mile of dicks per million interactions.

    link to this | view in thread ]


Follow Techdirt
Essential Reading
Techdirt Insider Discord

The latest chatter on the Techdirt Insider Discord channel...

Loading...
Recent Stories

Close

Email This

This feature is only available to registered users. Register or sign in to use it.