Content Moderation Case Study: Chatroulette Leverages New AI To Combat Unwanted Nudity (2020)

from the as-opposed-to-the-wanted-kind dept

Summary: Chatroulette rose to fame shortly after its creation in late 2009. The platform offered a new take on video chat, pairing users with other random users with each spin of the virtual wheel.

The novelty of the experience soon wore off when it became apparent Chatroulette was host to a large assortment of pranksters and exhibitionists. Users hoping to luck into some scintillating video chat were instead greeted with exposed penises and other body parts. (But mostly penises.)

This especially unsavory aspect of the service was assumed to be its legacy -- one that would see it resigned to the junkheap of failed social platforms. Chatroulette attempted to handle its content problem by giving users the power to flag other users and deployed a rudimentary AI to block possibly-offensive users.

The site soldiered on, partially supported by a premium service that paired users with other users in their area or who shared the same interests. Then something unexpected happened that drove a whole new set of users to Chatroulette: the COVID-19 pandemic. More people than ever were trapped at home and starved for human interaction. Very few of those were hoping to see an assortment of penises.

Faced with an influx of users and content to moderate, Chatroulette brought in AI moderation specialist Hive, the same company that currently moderates content on Reddit. With Chatroulette experiencing a resurgence, the company is hoping a system capable of processing millions of frames of chat video will keep its channels clear of unwanted content.

Decisions to be made by Chatroulette:

• Is it possible to filter live content quickly and accurately enough to prevent a return to the "old" Chatroulette?
• Is the cost of moderation AI affordable given the site's seeming inability to attract or sustain a large user base?
• If user growth continues, will it still be possible to backstop AI moderation with human moderators? 
• What metrics should Chatroulette consider as measures of success here?

Questions and policy implications to consider:

• Is over-moderation a foreseeable problem, given the challenges of moderating live video streams?
• Is it possible to attract a more-dedicated user base while still respecting their apparent desire for anonymity?
• Is it wise to maintain an unmoderated channel given the historical issues the site has had with unsolicited nudity and its exposure to/of minors?

Resolution: Chatroulette's new moderation efforts appear to be successfully distancing it from its inauspicious beginnings. However, the site's operating team still wryly acknowledges the reputation for nudity that defined the service for much of the last decade. But the site also points out that only a little more than 3% of its millions of monthly interactions contain explicit material, indicating Chatroulette feels the way forward is offering something that might have seemed boring ten years ago: a predictable and safe random chat experience.

Originally published on the Trust & Safety Foundation website.

Filed Under: ai, content moderation, live video
Companies: chatroulette, hive