Content Moderation Case Study: Telegram Gains Users But Struggles To Remove Violent Content (2021)

from the influx-of-insurrectionists dept

Summary: After Amazon refused to continue hosting Parler, the Twitter competitor favored by the American far-right, former Parler users looking to communicate with each other -- but dodge strict moderation -- adopted Telegram as their go-to service. Following the attack on the Capitol building in Washington, DC, chat app Telegram added 25 million users in a little over 72 hours.

Telegram has long been home to far-right groups, who often find their communications options limited by moderation policies that, unsurprisingly, remove violent or hateful content. Telegram's moderation is comparatively more lax than several of its social media competitors, making it the app of choice for far right personalities.

But Telegram appears to be attempting to handle the influx of users -- along with an influx of disturbing content. Some channels broadcasting extremist content have been removed by Telegram as the increasingly-popular chat service flexes its (until now rarely used) moderation muscle. According to the service, at least fifteen channels were removed by Telegram moderators, some of which were filled with white supremacist content.

Unfortunately, policing the service remains difficult. While Telegram claims to have blocked "dozens" of channels containing "calls to violence," journalists have had little trouble finding similarly violent content on the service, which either has eluded moderation or is being ignored by Telegram. While Telegram appears responsive to some notifications of potentially-illegal content, it also appears to be inconsistent in applying its own rule against inciting violence.

Decisions to be made by Telegram:

• Should content contained in private chats (rather than public channels) be subjected to the same rules concerning violent content?

• Given that many of its users migrated to Telegram after being banned elsewhere for posting extremist content, should the platform increase its moderation efforts targeting calls for violence?

• Should a process be put in place to help prevent banned users/channels from resurfacing on Telegram under new names?

• Does Telegram's promise of user security and privacy dissuade it from engaging in more active content moderation?

• Is context considered when engaging in moderation to avoid accidentally blocking people sharing content they feel is concerning, rather than promoting the content or endorsing its message?

• Do reports of mass content violations (and lax moderation) draw extremists to Telegram? Does this increase the chance of the moderation problem "snowballing" into something that can no longer be managed effectively?

Originally posted to the Trust & Safety Foundation website.

Filed Under: chat, content moderation, violence
Companies: telegram

Content Moderation Case Study: Scammers Targeting Scrabble Chat (2020)

from the it's-always-scrabble dept

Summary: In the spring of 2020, Mattel and Hasbro announced that the official mobile version of the game Scrabble would no longer be the game produced by Electronic Arts, but rather a new game called Scrabble Go created by a company called Scopely. The change drew the ire of fans (who have even started a petition for the old game to be brought back) for taking what had been a fairly standard mobile version of the popular word game, and introducing a new, flashier version that had some additional “gamification” incentives and put the focus on playing against others, rather than the computer as was typical in the previous game.

This also introduced a new feature: chat. Since players are playing against other human beings, Scopely decided to add a chat feature, but apparently did not consider how such features may be regularly abused. In the months since Scrabble Go launched, there have been many reports of so-called “romance scammers” trying to reach out to people via Scrabble Go’s chat feature.

Multiple reports of these kinds of approaches started appearing in various forums, with some examples of the scammers being quite persistent. At least in Australia, consumer protection officials noted that they have received multiple complaints of romance scammers approaching them via Scrabble Go. One woman in the UK noted that she has been approached by such scammers two to three times every week.

After three months of complaints, Scopely announced that it was rolling out an update that would allow players to “mute” the chat function.

Decisions to be made by Scopely:

• Does a mobile Scrabble game need a chat feature?
• If scammers are bothering players so often is the game better off without it?
• How will chat be monitored? Is there a program in place to catch and stop scammers?
• Are there other tools to limit the abuse of the chat feature?
• Should the default be that chat is open to all or should it be opt-in?

• Any system that allows for person-to-person communication can be abused. How should companies looking to add useful features take this into account?
• How do you weigh the pros and cons of features like chat when comparing their usefulness for engagement against their trust-and-safety risks?

The company has also added the ability to “report” a chat if the user feels it is inappropriate.

Finally, to address the broader complaints about the game, Scopely introduced a “classic mode” to focus more on the traditional game, rather than all the bells and whistles of the full Scrabble Go.

Originally posted at the Trust & Safety Foundation website.

Filed Under: chat, content moderation, scammers, scams, scrabble

Creating Family Friendly Chat More Difficult Than Imagined (1996)

from the the-kids-will-find-a-way dept

Summary: Creating family friendly environments on the internet presents some interesting challenges that highlight the trade-offs in content moderation. One of the founders of Electric Communities, a pioneer in early online communities, gave a detailed overview of the difficulties in trying to build such a virtual world for Disney that included chat functionality. He described being brought in by Disney alongside someone from a kids’ software company, Knowledge Adventure, who had built an online community in the mid-90s called “KA-Worlds.” Disney wanted to build a virtual community space, HercWorld, to go along with the movie Hercules. After reviewing Disney’s requirements for an online community, they realized chat would be next to impossible:

Even in 1996, we knew that text-filters are no good at solving this kind of problem, so I asked for a clarification: "I’m confused. What standard should we use to decide if a message would be a problem for Disney?"

The response was one I will never forget: "Disney’s standard is quite clear:

No kid will be harassed, even if they don’t know they are being harassed."...

"OK. That means Chat Is Out of HercWorld, there is absolutely no way to meet your standard without exorbitantly high moderation costs," we replied.

One of their guys piped up: "Couldn’t we do some kind of sentence constructor, with a limited vocabulary of safe words?"

Before we could give it any serious thought, their own project manager interrupted, "That won’t work. We tried it for KA-Worlds."

"We spent several weeks building a UI that used pop-downs to construct sentences, and only had completely harmless words – the standard parts of grammar and safe nouns like cars, animals, and objects in the world."

"We thought it was the perfect solution, until we set our first 14-year old boy down in front of it. Within minutes he’d created the following sentence:

I want to stick my long-necked Giraffe up your fluffy white bunny.

In that initial 1996 project, chat was abandoned, but as they continued to develop HercWorld, they quickly realized that they still had to worry about chat, even without a chat feature:

It was standard fare: Collect stuff, ride stuff, shoot at stuff, build stuff… Oops, what was that last thing again?

"…kids can push around Roman columns and blocks to solve puzzles, make custom shapes, and buildings.", one of the designers said.

I couldn’t resist, "Umm. Doesn’t that violate the Disney standard? In this chat-free world, people will push the stones around until they spell Hi! or F-U-C-K or their phone number or whatever. You’ve just invented Block-Chat^TM. If you can put down objects, you’ve got chat. We learned this in Habitat and WorldsAway, where people would turn 100 Afro-Heads into a waterbed." We all laughed, but it was that kind of awkward laugh that you know means that we’re all probably just wasting our time.

Decisions for family-friendly community designers:

• Is there a way to build a chat that will not be abused by clever kids to reference forbidden content (e.g., swearing, innuendo, harassment, abuse)?
• Can you build a chat that does not require universal moderation and pre-approval of everything that users will say?
• Are there ways in which kids will still be to communicate with others even without an actual chat feature?
• How much of a “community” do you have with no chat or extremely limited chat?

Questions and policy implications to consider:

• Is it possible to create an online family friendly environment that will work?
  • If so how do you prevent abuse?
  • If not, how do you handle the fact that kids will get online whether they are allowed to or not?
• How do you incentivize companies to create spaces that actually remain as child-friendly as possible?
• If “the kids will always find a way” to get around limitations, does it make sense to hold the companies themselves responsible?
• Should family friendly environments require full-time monitoring, or pre-vetting of any usage?

"The Disney Standard" (now a legend amongst our employees) still held. No harassment, detectable or not, and no heavy moderation overhead.

Brian had an idea though: Fully pre-constructed sentences – dozens of them, easy to access. Specialize them for the activities available in the world. Vaz Douglas, our project manager working with Zoog, liked to call this feature "Chatless Chat." So, we built and launched it for them. Disney was still very tentative about the genre, so they only ran it for about six months; I doubt it was ever very popular.

The same interview notes that Disney tried once again in 2002 with a new world called “ToonTown”, with pulldown menus that allowed you to construct very narrowly tailored speech within the chat to try to avoid anything that violated the rules.

As the story goes, Disney still had problems with this. To make sure people were only communicating with people they knew in real life, one of the restrictions in this new world was that you had to have a secret code from any user you wished to chat with. The thinking was that parents would print these out for kids who could then share them with their friends in real life, and they could link up and “chat” in the online world.

And yet, once again, people figured out how to get around the restrictions:

Sure enough, chatters figured out a few simple protocols to pass their secret code, several variants are of this general form:

User A:"Please be my friend."
User A:"Come to my house?"
User B:"Okay."
A:[Move the picture frames on your wall, or move your furniture on the floor to make the number 4.]
A:"Okay"
B:[Writes down 4 on a piece of paper and says] "Okay."
A:[Move objects to make the next letter/number in the code] "Okay"
B:[Writes…] "Okay"
A:[Remove objects to represent a "space" in the code] "Okay"
[Repeat steps as needed, until…]
A:"Okay"
B:[Enters secret code into Toontown software.]
B:"There, that worked. Hi! I’m Jim 15/M/CA, what’s your A/S/L?"

Incredibly, there was an entire Wiki page on the Disney Online Worlds domain that included a variety of other descriptions on how to exchange your secret number within the game, even as users were not supposed to be doing so:

For example, let's say you have a secret code (1hh 5rj) which you would like to give to a toon named Bob.

First, you should make clear that you want to become their SF.
You: Please be my friend!
You: (random SF chat)
You: I can't understand you
You: Let's work on that
Bob: Yes
Now, start the secret.
You: (Jump 1 time and say OK. Jump 1 time because that is the first thing in your code. Say OK to confirm that was part of your secret.)
Bob: OK (Wait for this, as this means he has written down or otherwise recorded the 1)
You: Hello! OK (Say hello because the first letter of hello is h, which is the second part of your secret.)
Bob: OK (again, wait for confirmation)
Repeat above step, as you have the same letter for the third part of your secret.
Bob: OK (by now you should know to wait for this)
You: (Jump 5 times and say OK. Jump 5 times as this is the 4th part of your secret)
Bob: OK
You: Run! OK (The 5th part of your secret is r, and "Run!" starts with r)
Bob: OK
You: Jump! OK (Say this because j is the last part of your secret.)
Bob: OK
At this point, you have successfully transmitted the code to Bob.
Most likely, Bob will understand, and within seconds, you will be Secret Friends!

So even though Disney eventually did enable a very limited chat, with strict rules to keep people safe, it still left open many challenges for early trust & safety work.

Images from HabitChronicles

Filed Under: chat, content moderation, family friend
Companies: disney