Content Moderation Case Studies: Can Baby Yoda GIFs Defeat The DMCA Force? (2019)

from the copyright-on-the-dark-side dept

Summary: In the fall of 2019, Disney launched its Disney+ streaming service to instant acclaim. While it offered up access to the extensive Disney catalog (including all of its Marvel, Star Wars, and 21st Century Fox archives), the first big new hit for the service was a TV series set in the Star Wars universe called The Mandalorian, which featured a character regularly referred to as “Baby Yoda.”

Baby Yoda was a clear hit online, and people quickly made animated gif images of the character, helping spread more interest in The Mandalorian and the Disney+ service. However, soon after Vulture Magazine put up a story that was all just Baby Yoda GIFs, it was discovered that Giphy, a company that has built a repository of GIFs, had taken all of the Baby Yoda GIFs down. This caused many to complain, blaming Disney, highlighting that such GIFs were clearly fair use.

Many people assumed that Disney was behind the takedown of the Baby Yoda GIFs. This may be a natural assumption since Disney, above and beyond almost any other company, has a decades-long reputation for aggressively enforcing its copyright. The Washington Post even wrote up an entire article scolding Disney for “not understanding fans.”

That article noted that it was possible that Giphy pre-emptively decided to take down the images, but pointed out that this was, in some ways, even worse. This would mean that Disney’s own reputation as an aggressive enforcer of copyrights would lead another company to take action even without an official DMCA takedown notice.

Giphy itself has always lived in something of a gray area regarding copyright, since many of the GIFs are from popular culture, including TVs and movies. While there is a strong argument that these are fair use, the company has claimed that most of its content is licensed, and said that it does not rely on fair use.

Decisions to be made by Giphy:

• Should the company rely on fair use to cover certain GIFs that are uploaded, or should it try to license everything?

• Regarding uploaded GIFs, how aggressive should the company be in searching for and taking down content? Should it only do so after receiving a takedown notice, or should it proactively remove content?

• For popular content, like Baby Yoda images, should Giphy reach out to the copyright holder (in the case Disney) to either get permission or to work out a partnership?

Questions and policy implications to consider:

• Popular culture content is frequently used in memes and GIFs. Is copyright law properly calibrated to allow this kind of activity?

• Fair use is found in only a few countries (most notably, the US). How do differences in international copyright law impact the ability of a service like Giphy to exist?

• Is Disney better off encouraging fans to spread GIFs, such as those of Baby Yoda, than exercising whatever copyright enforcement powers it has?

• If Giphy took down the Baby Yoda images preemptively, does that indicate that fear of copyright litigation is holding back cultural sharing?

Resolution: Soon after the story went viral, Giphy issued an apology to Disney, and to Vulture which had posted the original article full of Baby Yoda GIFs. The apology suggested that Giphy had made the decision to remove the GIFs without any takedown notice or other input from Disney.

"Last week, there was some confusion around certain content uploaded to Giphy and we temporarily removed these Gifs while we reviewed the situation," said the image-hosting website in a statement.

"We apologise to both Disney and Vulture for any inconvenience, and we are happy to report that the Gifs are once again live on Giphy."

In multiple articles about this, it is noted that Disney refused to comment on the issue, leaving some reporters to wonder if Disney had played a role but did not wish to discuss it publicly. Either way, there are now many Baby Yoda GIFs on Giphy.

Originally posted on the Trust & Safety Foundation website.

Filed Under: baby yoda, copyright, disney plus, fair use, pre-emptive takedowns, takedowns
Companies: disney, giphy

Content Moderation Case Study: Google's Ad Policies Inadvertently Block Religious Organizations From Advertising On YouTube (2019)

from the unacceptable-content? dept

Summary: Google's ad service offers purchasers access to millions of users, including those viewing videos on YouTube. But its policies -- meant to prevent abuse, fraud, harassment, or targeting of certain demographics -- sometimes appear to prevent legitimate organizations from doing something as simple as informing others of their existence.

Chad Robichaux, the founder of Christian veterans support nonprofit Mighty Oaks, wanted to reach out to veterans who might need his services. But his attempt to purchase YouTube ads was rejected by Google's Ad service for a seemingly strange reason.

According to a screenshot posted by Robichaux to Twitter, Google forbade the use of "Christian" as a keyword. To Robichaux (and many responders to his tweet), this was evidence of Big Tech's bias against Christians and conservatives.

But the real reason for this block was far less censorial or nefarious, if no more explicable. According to YouTube (which reached out directly to Robicheaux), the aim isn't to keep Christians from advertising, but rather to prevent advertisers from targeting users on the basis of their religion. Unfortunately, Google's policy doesn't exactly make that clear, instead stating that ads cannot contain "religious basis" content if the purchaser is engaging in personalized advertising.

Decisions to be made by Google:

• Does blocking certain keywords make some ads impossible to place, no matter what audience is targeted or where the content may appear?

• Is it ok for advertisers to target these groups if the users have already self-identified as being members of these groups? Would it be ok if users could explicitly opt in to being targeted in this way?

• Is clarification or simplification of the rules needed to ensure accidental blocking or further misunderstandings are avoided?

• Should advertisers be given more guidance on how to craft ads/seek users to prevent violations?

Questions and policy implications to consider:

• Does having control of a majority of the advertising market lower the quality of assistance users receive from Google given the limited options available to them elsewhere?

• Does increasing the number of keyword restrictions result in fewer successful ad placements and lower ad sales?

• Does "protecting" users from personalized ads using certain keywords result in users see more irrelevant ads?

Resolution: The confusion was (somewhat) cleared up by YouTube's direct contact with the concerned ad buyer. But other confusion still remains since the policies guiding ad purchasing/ad construction are far from straightforward. Allegations of bias were off-base. Instead, it was simply Google enforcing its policies, which would have made it equally impossible to use any other religion as a keyword.

Originally posted on the Trust & Safety Foundation website.

Filed Under: ads, content moderation, religious targeting, youtube
Companies: google, youtube

Content Moderation Case Studies: The Challenges In Moderating Information Regarding Eating Disorders (2012)

from the not-as-simple-as-you-might-think dept

Summary: In 2012, the Huffington Post did an exposé on eating disorder blogs, mainly on the site Tumblr. It discussed the world of “thinspo” and “thinspiration” blogs, that focused on building a community around losing unhealthy amounts of weight. In response, Tumblr announced that it was banning “self harm” blogs, and classified eating disorder blogs among those no longer allowed.

Three years later, a study by Munmun De Choudhury discussed how there was still eating disorder information on Tumblr, but that it was mainly split into two different categories: those who were supportive of eating disorders such as anorexia (referred to as “proana”) as well as communities built up around recovering from eating disorders. One interesting finding of the report was that the “recovery” groups often used the same keywords and messaging, in an attempt to permeate among the “proana” groups, in order to try to encourage those with eating disorders to seek support, therapy, and help towards recovery.

That same year, Amanda Hess argued in Slate that the rush to ban content about eating disorders on social media (or, in the case of France, where such things were outlawed) was the wrong approach.

"But while we know anorexia can kill, we’re not quite sure what happens to people who read about it online. In an article published last month, Canadian criminologists Debra Langan and Nicole Schott could find 'no scholarly evidence' that pro-ana blogs pose a threat to their audiences. If they do, there’s no proof that any of our social remedies—censorship, PSAs, or prison time—do anything to help. These campaigns are most obviously effective at flattering the egos of the lawmakers and tech execs who champion them. When a girl searches Tumblr for a pro-ana–adjacent term like #thinspo or #thighgap now, Tumblr intercepts her request with bland concern ('Everything OK?'), then advises her to check out the cutesy motivational messaging on the National Eating Disorders Association’s Tumblr instead. However the girl responds, Tumblr can feel satisfied it’s performed its civic responsibility. The strategy recalls the one favored by a 19th-century doctor who believed that reading novels caused hysteria in women: He counseled men to confiscate their wives’ fiction and replace it with a book on 'some practical subject,' like 'beekeeping.'"

The following year, De Choudhury and other authors released another study detailing how pro-eating disorder groups would get around social media blocks on their content by changing words around, or slightly misspelling them, suggesting that the out-and-out blocking method was likely to continue to be ineffective.

Another article suggested that the blocks almost made it easier to find information about eating disorders, because dozens of new hashtags were created for the community, rather than just a few before social media sites began to ban such content.

A study released in the peer-reviewed New Media & Society journal in 2018 highlighted how easy it appeared to be for users to get around attempts to block content regarding certain eating disorders. The researcher, Ysabel Gerrard, looked mainly at Pinterest, Tumblr, and Instagram, finding that while all three had some policies in place regarding eating disorder information, it was not difficult to find groups or sites dedicated to such information.

As summarized by Wired:

"She immediately found that Instagram’s pro-ED hashtag ban has an easy workaround: You can search for people who have the keywords in their usernames, just not hashtagged in their posts. She identified 74 public accounts that had terms like 'proana,' 'proanorexia,' or 'thighgap' in their names or bios and who also posted pro-ED content. Then, she analyzed 1,612 of their posts—only 561 of which had hashtags—by cataloguing the content of the image and its caption.

"On Tumblr, Gerrard followed a number of terms related to pro-ED content, like 'thinspo,' 'proana,' and 'bulimic.' Tumblr allows you to follow topics without needing to follow specific users. For example, you can simply follow 'movies' without following any specific user who posts about that topic. Through this method, she found 50 pro-ED blogs and analyzed 20 posts from each, or 1,000 posts total. Only 218 of the posts were tagged."

The report also found that the recommendation algorithms often drove users towards more such content. By saving a few 'proana' blogs, Gerrard found that Tumblr began recommending more. While it did also recommend some recovery blogs, Gerrard found them easy to exclude.

"Once I had followed ED-related terms - anorexia, anorexic, bulimia, bulimic, thinspiration, thinspo, proana, purge, purging - the platform delivered this content to me through my dashboard and also via email. Tumblr showed me relevant posts and suggested a list of users whose accounts I should follow. As some of these terms are not straightforwardly pro-ED (unlike, for example, proana), I was presented with blogs identifying as ‘pro-recovery’ in their biographies. But I excluded these blogs from the dataset as they were not the focus of my analysis. Tumblr recommended blogs that were, for example, ‘big in proana’ or ‘like’ other popular blogs. I identified fifty pro-ED users through this method."

Another study from 2014 argued that even the “proana” content represented a “double-edged sword” and might help some of those either with eating disorders, or those at risk, to recognize that what they were exploring was unhealthy.

Decisions to be made by Tumblr/Instagram/Pinterest:

• How do you deal with information about eating disorders? Is it actually possible to ban it?

• How do you distinguish between “proana” and “recovery” content?

• Are there other interventions that can be done, such as putting up warning labels, or directing users towards “recovery” resources when they search on certain terms?

• How should recommendation algorithms handle information about eating disorders? Does it need to be adjusted to avoid sending people towards content that glorifies such disorders?

Questions and policy implications to consider:

• Does banning content of those promoting unhealthy eating disorders actually help prevent eating disorders?

• Does reading about eating disorders function as a how-to guide for the vulnerable, or does it help those at risk recognize that risk? If both, how do you balance these two competing forces?

• Can pointing people towards recovery content or other helpful resources lead to better outcomes?

Resolution: Various websites struggle with how to deal with eating disorder information and communities. Attempts to ban it have continued to fail, as the various communities continue to figure out ways to route around any ban. The research on the impact of this content remains mixed, however, and there have been some concerns that efforts to ban such content on certain platforms only makes it move to others that are less well organized to handle the issue.

The latest is a report that teens who were engaged in eating disorder discussions on Tumblr have now moved to TikTok. However, that same article also notes that, unlike Tumblr, TikTok seems to have a number of users who celebrate healthy eating and living, and that TikTok’s algorithm may be inserting such videos mixed in with those discussing more unhealthy eating behavior.

Originally published on the Trust & Safety Foundation website.

Filed Under: anorexia, content moderation, eating disorders, proana, recovery

Content Moderation Case Study: Bumble Shuts Down Sharon Stone's Account, Not Believing It's Really Her (2019)

from the bees-and-catfish dept

Summary: Almost any platform that allows users to create accounts eventually has to deal with questions of identity and impersonation. Many platforms set up systems like “verified” or “trusted” users for certain recognizable accounts. Others focus on real name policies, or trying to verify all users. But services often discover challenges that come with celebrity users and verification.

While it’s one thing to do verified accounts on platforms like Twitter, Facebook, or Instagram that are often used for promotion and connection, dating site verification is a bit different and more complicated. Setting up fake personas on dating sites to lure people into misleading relationships (for a wide variety of reasons) is so common that it led to the creation of a whole new term: catfishing. Many dating sites now take user verification quite seriously, not just to avoid catfishing issues, but for the safety and protection of their userbase -- who, by definition, are usually trying to meet someone new with the hope of getting together in person.

Bumble is a popular dating app which was built up around the premise of being safer, and more responsive to the needs of female daters. The site includes a verification feature that requests the user upload selfie poses that match poses in photos sent to the user -- which are then reviewed by a team member. The idea is that if a user were faking images by pulling them from online profiles or generating them via AI, it’s much harder to match the pose.

Apparently, however, this form of verification ran into a problem when the actress Sharon Stone decided to use Bumble to meet potential dates. Users who matched with her, perhaps understandably, had difficulty believing that a famous Hollywood star would be using a dating app like Bumble, and they reported the account. Staff reviewers at Bumble were (again, reasonably) equally suspicious of the account, leading them to suspend it.

Bumble quickly restored the account, and did so in a good natured way, wishing her luck in “finding your honey.”

Decisions to be made by Bumble:

• What systems do you use to verify users are who they say they are?

• How much weight should be given to user reports that people they matched with are not real?

• How do you handle celebrities, whose accounts people may not believe are legitimate?

• What appeals process should there be for blocked accounts that were deemed to be fake?

Questions and policy implications to consider:

• On dating apps in particular, user safety is key, so should sites default towards overblocking, rather than being hesitant?

• Are there other forms of verification that would alleviate problems similar to the one Stone faced here?

• Stone was able to get her account reinstated quickly because of her fame; does the existing appeals process work as well for users who don’t have that pull?

Resolution: Bumble was pretty quick to restore Stone’s account after she tweeted about it, and major news organizations picked up the story. A few months later, Stone admitted that she suspected that she was reported by men who were upset she had turned them down on the platform.

“I think that I said no to a couple of people that thought that it would be a nice way to be not-so-kind back,” she explained. “I think some people don’t like to hear, ‘No, no I don’t want to go out with you.'”

She also noted that she has “made some nice friends” on the site.

In the meantime, some have argued that Bumble purposely chose to block Stone in order to generate publicity. Of course, this would only work if the company knew that Stone would complain about the block publicly, which certainly was not guaranteed.

Originally posted to the Trust & Safety Foundation website.

Filed Under: content moderation, dating, identity, sharon stone
Companies: bumble

Content Moderation Case Study: Decentralized Social Media Platform Mastodon Deals With An Influx Of Gab Users (2019)

from the decentralized-content-moderation-challenges dept

Summary: Formed as a more decentralized alternative to Twitter that allowed users to more directly moderate the content they wanted to see, Mastodon has experienced slow, but steady, growth since its inception in 2016.

Unlike other social media networks, Mastodon is built on open-source software and each "instance" (server node) of the network is operated by users. These separate "instances" can be connected with others via Mastodon's interlinked "fediverse." Or they can remain independent, creating a completely siloed version of Mastodon that has no connection with the service's larger "fediverse."

This puts a lot of power in the hands of the individuals who operate each instance: they can set their own rules, moderate content directly, and prevent anything the "instance" and its users find undesirable from appearing on their servers. But the larger "fediverse" -- with its combined user base -- poses moderation problems that can't be handled as easily as those presenting themselves on independent "instances." The connected "fediverse" allows instances to interact with each other, allowing unwanted content to appear on servers that are trying to steer clear of it.

That's where Gab -- another Twitter alternative -- enters the picture. Gab has purposely courted users banned from other social media services. Consequently, the platform has developed a reputation for being a haven for hate speech, racists, and bigots of all varieties. This toxic collection of content/users led to both Apple and Google banning Gab's app from their app stores.

Faced with this app ban, Gab began looking for options. It decided to create its own Mastodon instance. With its server now technically available to everyone in the Mastodon "fediverse," those not explicitly blocking Gab's "instance" could find Gab content available to its users -- and also allow for Gab’s users to direct content to their own users. It also allowed Gab to utilize the many different existing Mastodon apps to sidestep the app bans handed down by Google and Apple.

Decisions to be made by Mastodon:

• Should Gab (and its users) be banned from setting up "instances," given that they likely violate the Mastodon Server Covenant?

• Is it possible to moderate content across a large number of independent nodes?

• Is this even an issue for Mastodon itself to deal with, given that the individuals running different servers can decide for themselves whether or not to allow federation with the Gab instance?

• Given the open source and federated nature of Mastodon, would there reasonably be any way to stop Gab from using Mastodon?

Questions and policy implications to consider:

• Will moderation efforts targeting the "fediverse" undercut the independence granted to "instance" owners?

• Do attempts to attract more users create moderation friction when the newly-arriving users create content Mastodon was created to avoid?

• If Mastodon continues to scale, will it always face challenges as certain instances are created to appeal to audiences that the rest of the “fediverse” is trying to avoid?

• Can a federated system, in which unique instances choose not to federate with another instance, such as Gab, work as a form of “moderation-by-exclusion”?

Resolution: Mastodon's founder, Eugen Rochko, refused to create a blanket ban on Gab, leaving it up to individual "instances" to decide whether or not to interact with the interlopers. As he explained to The Verge, a blanket ban would be almost impossible, given the decentralized nature of the service.

On the other hand, most "fediverse" members would be unlikely to have to deal with Gab or its users, considering the content contained in Gab's "instance" routinely violates the Mastodon "covenant." Violating these rules prevents instances from being listed by Mastodon itself, lowering the chances of other "instance" owners inadvertently adding toxic content and users to their server nodes. And Rochko himself encouraged users to preemptively block Gab's "instance," resulting in ever fewer users being affected by Gab's attempted invasion of the Mastodon fediverse.

But running a decentralized system creates an entirely new set of moderation issues, which has turned Mastodon itself into a moderation target. Roughly a year after the Gab "invasion," Google threatened to pull Mastodon-based apps from its store for promoting hate speech, after users tried to get around the Play Store ban by creating apps that pointed to Mastodon “instances” filled with hateful content. Google ultimately decided to leave Mastodon-based apps up, but appears ready to pull the trigger on a ban in future.

Originally posted to the Trust & Safety Foundation website.

Filed Under: content moderation, decentralized, federated social media, instances
Companies: gab, mastodon

Content Moderation Case Study: Removing Nigerian Police Protest Content Due To Confusion With COVID Misinfo Rules (2020)

from the moderation-confusion dept

Summary: With the beginning of the COVID-19 pandemic, most of the large social media companies very quickly put in place policies to try to handle the flood of disinformation about the disease, responses, and treatments. How successful those new policies have been is subject to debate, but in at least one case, the effort to fact check and moderate COVID information ran into a conflict with people reporting on violent protests (totally unrelated to COVID) in Nigeria.

In Nigeria, there’s a notorious division called the Special Anti-Robbery Squad, known as SARS in the country. For years there have been widespread reports of corruption and violence in the police unit, including stories of how it often robs people itself (despite its name). There have been reports about SARS activities for many years, but in the Fall of 2020 things came to a head as a video was released of SARS officers dragging two men out of a hotel in Lago and shooting one of them in the street.

Protests erupted around Lagos in response to the video, and as the government and police sought to crack down on the protests, violence began, including reports of the police killing multiple protesters. The Nigerian government and military denied this, calling it “fake news.”

Around this time, users on both Instagram and Facebook found that some of their own posts detailing the violence brought by law enforcement on the protesters were being labeled as “False Information” by Facebook’s fact checking system. In particular an image of the Nigerian flag, covered in blood of shot protesters, which had become a symbolic representation of the violence at the protests, was flagged as “false information” multiple times.

Given the government’s own claims of violence against protesters being “fake news” many quickly assumed that the Nigerian government had convinced Facebook fact checkers that the reports of violence at the protests were, themselves, false information.

However, the actual story turned out to be that Facebook’s policies to combat COVID-19 misinformation were the actual problem. At issue: the name of the police division, SARS, is the same as the more technical name of COVID-19: SARS-CoV-2 (itself short for: “severe acute respiratory syndrome coronavirus 2”). Many of the posts from protesters and their supporters in Lagos used the tag #EndSARS, talking about the police division, not the disease. And it appeared that the conflict between those two things, combined with some automated flagging, resulted in the Nigerian protest posts being mislabeled by Facebook’s fact checking system.

Decisions to be made by Facebook:

• How should the company review content that includes specific geographical, regional, or country specific knowledge, especially when it might (accidentally) clash with other regional or global issues?
• In dealing with an issue like COVID misinformation, where there’s an urgency in flagging posts, how should Facebook handle the possibility of over-blocking of unrelated information as happened here?
• What measures can be put in place to prevent mistakes like this from happening again?

Questions and policy implications to consider:

• While large companies like Facebook now go beyond simplistic keyword matching for content moderation, automated systems are always going to make mistakes like this. How can policies be developed to limit the collateral damage and false marking of unrelated information?
• If regulations require removal of misinformation or disinformation, what would likely happen in scenarios like this case study?
• Is there any way to create regulations or policies that would avoid the mistakes described above?

Resolution: After the incorrectly labeled content began to get attention both Instagram and Facebook apologized and took down the “false information” flag on the content.

Yesterday our systems were incorrectly flagging content in support of #EndSARS, and marking posts as false. We are deeply sorry for this. The issue has now been resolved, and we apologize for letting our community down in such a time of need.

Facebook’s head of communications for sub-Saharan Africa, Kezia Anim-Addo, gave Tomiwa Ilori, writing for Slate, some more details on the combination of errors that resulted in this unfortunate situation:

In our efforts to address misinformation, once a post is marked false by a third party face checker, we can use technology to “fan out” and find duplicates of that post so if someone sees an exact match of the debunked post, there will also be a warning label on it that it’s been marked as false.

In this situation, there was a post with a doctored image about the SARS virus that was debunked by a Third-Party Fact Checking partner

The original false image was matched as debunked, and then our systems began fanning out to auto-match to other images

A technical system error occurred where the doctored images was connected to another different image, which then also incorrectly started to be matched as debunked. This created a chain of fan outs pulling in more images and continuing to match them as debunked.

This is why the system error accidentally matched some of the #EndSARS posts as misinformation.

Thus, it seems like a combination of factors was at work here, including a technical error and the similarities in the “SARS” name.

Originally posted to the Trust & Safety Foundation website.

Filed Under: confusion, content moderation, covid, filters, nigeria, sars
Companies: facebook

Content Moderation Case Study: Chatroulette Leverages New AI To Combat Unwanted Nudity (2020)

from the as-opposed-to-the-wanted-kind dept

Summary: Chatroulette rose to fame shortly after its creation in late 2009. The platform offered a new take on video chat, pairing users with other random users with each spin of the virtual wheel.

The novelty of the experience soon wore off when it became apparent Chatroulette was host to a large assortment of pranksters and exhibitionists. Users hoping to luck into some scintillating video chat were instead greeted with exposed penises and other body parts. (But mostly penises.)

This especially unsavory aspect of the service was assumed to be its legacy -- one that would see it resigned to the junkheap of failed social platforms. Chatroulette attempted to handle its content problem by giving users the power to flag other users and deployed a rudimentary AI to block possibly-offensive users.

The site soldiered on, partially supported by a premium service that paired users with other users in their area or who shared the same interests. Then something unexpected happened that drove a whole new set of users to Chatroulette: the COVID-19 pandemic. More people than ever were trapped at home and starved for human interaction. Very few of those were hoping to see an assortment of penises.

Faced with an influx of users and content to moderate, Chatroulette brought in AI moderation specialist Hive, the same company that currently moderates content on Reddit. With Chatroulette experiencing a resurgence, the company is hoping a system capable of processing millions of frames of chat video will keep its channels clear of unwanted content.

Decisions to be made by Chatroulette:

• Is it possible to filter live content quickly and accurately enough to prevent a return to the "old" Chatroulette?
• Is the cost of moderation AI affordable given the site's seeming inability to attract or sustain a large user base?
• If user growth continues, will it still be possible to backstop AI moderation with human moderators? 
• What metrics should Chatroulette consider as measures of success here?

Questions and policy implications to consider:

• Is over-moderation a foreseeable problem, given the challenges of moderating live video streams?
• Is it possible to attract a more-dedicated user base while still respecting their apparent desire for anonymity?
• Is it wise to maintain an unmoderated channel given the historical issues the site has had with unsolicited nudity and its exposure to/of minors?

Resolution: Chatroulette's new moderation efforts appear to be successfully distancing it from its inauspicious beginnings. However, the site's operating team still wryly acknowledges the reputation for nudity that defined the service for much of the last decade. But the site also points out that only a little more than 3% of its millions of monthly interactions contain explicit material, indicating Chatroulette feels the way forward is offering something that might have seemed boring ten years ago: a predictable and safe random chat experience.

Originally published on the Trust & Safety Foundation website.

Filed Under: ai, content moderation, live video
Companies: chatroulette, hive

Content Moderation Case Study: Senator Asks YouTube To Block Al Qaeda Videos (2008)

from the is-that-constitutional? dept

Summary: In 2008, the Senate Homeland Security and Government Affairs Committee, chaired by then Senator Joe Lieberman, produced a report entitled: “Violent Islamist Extremism: The Internet, and the Homegrown Terrorist Threat.” The report mentions a rap video called “Dirty Kuffar” (Kuffar meaning “nonbeliever”) that, according to the report, praises Osama bin Laden and the attacks of 9/11.

A few days after the report came out, Lieberman sent a letter addressed to then Google CEO Eric Schmidt pointing to the report and asking the company to remove terrorist content from the site, including things like the named video.

As some quickly pointed out, the one video named in the report was hardly espousing terrorism or hate speech. It may be mildly offensive, but it was clearly protected political speech.

The letter that Lieberman sent was accompanied by a list of other videos that Lieberman’s staff claimed were promoting terrorism content, and Lieberman asked YouTube to not only remove the specific videos that violated its policies, but to shut down the accounts of those who posted the videos in the first place.

Decisions for YouTube:

• How do you determine which content is actually promoting terrorism compared to those that are just discussing it, reporting on it, or highlighting terrorist attacks?
• How do you distinguish political speech from terrorist content?
• How do you respond to a sitting US Senator demanding the removal of content?
• If an account has violated policies against terrorist content once, should the entire account be shut down?

Questions and policy implications to consider:

• Since an elected US official can potentially change the laws, requests are likely to be taken more seriously.  If the requests are to take down 1st Amendment protected speech, does that raise 1st Amendment issues?
• Is it possible to readily distinguish content that is promoting terrorism from that which is documenting terrorism and war crimes for the historical record?

Resolution: YouTube chose to push back on Senator Lieberman’s request, putting up a blog post saying that it wished to have a dialogue with the Senator. While the company said it did remove some of the videos Lieberman’s staff highlighted, if they were found to violate its policies, it would not remove them all, nor would it shut down all of the accounts mentioned.

Senator Lieberman's staff identified numerous videos that they believed violated YouTube's Community Guidelines. In response to his concerns, we examined and ended up removing a number of videos from the site, primarily because they depicted gratuitous violence, advocated violence, or used hate speech. Most of the videos, which did not contain violent or hate speech content, were not removed because they do not violate our Community Guidelines.

Senator Lieberman stated his belief, in a letter sent today, that all videos mentioning or featuring these groups should be removed from YouTube -- even legal nonviolent or non-hate speech videos. While we respect and understand his views, YouTube encourages free speech and defends everyone's right to express unpopular points of view. We believe that YouTube is a richer and more relevant platform for users precisely because it hosts a diverse range of views, and rather than stifle debate we allow our users to view all acceptable content and make up their own minds. Of course, users are always free to express their disagreement with a particular video on the site, by leaving comments or their own response video. That debate is healthy.

Senator Lieberman continued to pressure Google over these policies over the years, including writing a similar letter to complain about “terrorist content” on the company’s Blogger platform a few years later. Since then, YouTube has ramped up its efforts to block “terrorist” content on the platform, but has also been accused many, many times of going too far and actually deleting content from human rights groups that were trying to document war crimes and other atrocities.

Originally posted to the Trust & Safety Foundation website.

Filed Under: joe lieberman, terrorist videos, threats, videos
Companies: google, youtube

Content Moderation Case Study: Valve Takes A Hands Off Approach To Porn Via Steam (2018)

from the know-it-when-we-see-it dept

Summary: Different platforms have different rules regarding “adult” content, but they often prove difficult to enforce. Even the US judicial system has declared that there is no easy way to define pornography, leading to Justice Potter Stewart’s famous line, “I know it when I see it.”

Many, if not most, internet websites have rules regarding such adult content, and in 2017 Valve’s online game platform, Steam, started trying to get more serious about enforcing its rules, leading to some smaller independent games being banned from the platform. Over the next few months more and more games were removed, though some started pointing out that this policy and the removals were doing the most harm to independent game developers.

In June of 2018, Valve announced that it had listened to various discussions on this and decided that it was going to take a very hands off approach to moderating content, including adult content. After admitting that there are widespread debates over this, the company said that it would basically allow absolutely anything on the platform, with very, very few exceptions:

So we ended up going back to one of the principles in the forefront of our minds when we started Steam, and more recently as we worked on Steam Direct to open up the Store to many more developers: Valve shouldn't be the ones deciding this. If you're a player, we shouldn't be choosing for you what content you can or can't buy. If you're a developer, we shouldn't be choosing what content you're allowed to create. Those choices should be yours to make. Our role should be to provide systems and tools to support your efforts to make these choices for yourself, and to help you do it in a way that makes you feel comfortable.

With that principle in mind, we've decided that the right approach is to allow everything onto the Steam Store, except for things that we decide are illegal, or straight up trolling. Taking this approach allows us to focus less on trying to police what should be on Steam, and more on building those tools to give people control over what kinds of content they see. We already have some tools, but they're too hidden and not nearly comprehensive enough. We are going to enable you to override our recommendation algorithms and hide games containing the topics you're not interested in. So if you don't want to see anime games on your Store, you'll be able to make that choice. If you want more options to control exactly what kinds of games your kids see when they browse the Store, you'll be able to do that. And it's not just players that need better tools either - developers who build controversial content shouldn't have to deal with harassment because their game exists, and we'll be building tools and options to support them too.

The company admitted that it would likely struggle with this plan, especially given different laws around the globe, but that it wanted to put the onus on end users, rather than itself.

Decisions to be made by Valve:

• Is it really possible to allow anything that isn’t “illegal or straight up trolling”?
• How do you define “straight up trolling”?
• How do you make sure that parts of the Steam store are safe for younger users?
• What tools need to be provided to users to set their own filters?

Questions and policy implications to consider:

• With more and more pressure from governments to clean up the internet, will taking a “hands off approach” lead to even more regulatory threats?
• Does taking such a hands off approach create greater legal liability?
• Can a hands off approach make users feel that the company is putting all of the responsibility on them, rather than itself?

Resolution: In the following few months, Valve released more ways to filter content in its store, including an adult filter. It also began approving more explicit games, as suggested by the policy.

At around the same time, it did continue to remove games, supposedly for violating its new “no trolling” policy. The company admitted that the no trolling policy is intentionally vague.

It is vague and we'll tell you why. You're a denizen of the internet so you know that trolls come in all forms. On Steam, some are simply trying to rile people up with something we call "a game shaped object" (ie: a crudely made piece of software that technically and just barely passes our bar as a functioning video game but isn't what 99.9% of folks would say is "good"). Some trolls are trying to scam folks out of their Steam inventory items, others are looking for a way to generate a small amount of money off Steam through a series of schemes that revolve around how we let developers use Steam keys. Others are just trying to incite and sow discord. Trolls are figuring out new ways to be loathsome as we write this. But the thing these folks have in common is that they aren't actually interested in good faith efforts to make and sell games to you or anyone. When a developer's motives aren't that, they're probably a troll.

Our review of something that may be "a troll game" is a deep assessment that actually begins with the developer. We investigate who this developer is, what they've done in the past, their behavior on Steam as a developer, as a customer, their banking information, developers they associate with, and more. All of this is done to answer the question "who are we partnering with and why do they want to sell this game?" We get as much context around the creation and creator of the game and then make an assessment. A trend we're seeing is that we often ban these people from Steam altogether instead of cherry-picking through their individual game submissions. In the words of someone here in the office: "it really does seem like bad games are made by bad people."

This doesn't mean there aren't some crude or lower quality games on Steam, but it does mean we believe the developers behind them aren't out to do anything more than sell a game they hope some folks will want to play.

The company has still faced some criticism over these policies. In 2019 an anti-pornography group complained publicly that it was too easy to find adult content on Steam despite the new filters that were put in place, saying that the filters were “mere speedbumps.”

In late 2020, Steam started to experiment with a revamp of how it organizes content, and that may include an explicit games area.

Originally published to the Trust & Safety Foundation website.

Filed Under: content moderation, porn, steam, trolling, video games
Companies: valve

Content Moderation Case Study: Twitter Attempts To Tackle COVID-Related Vaccine Misinformation (2020)

from the dealing-with-covid-misinfo dept

Summary: Following on its efforts in tamping down on election-related misinformation, Twitter's latest moderation efforts target misleading posts about COVID and the coronavirus, with a specific focus on vaccine related information.

Despite being months into a global pandemic, there has been a lack of clear, consistent communication from all levels of government in the United States, which has given conspiracy theorists and anti-vaccination activists plenty of room to ply their dubious trades. Twitter is hoping to reduce exposure to tweets containing misleading information as the nation continues to deal with multiple COVID outbreaks.

Since early in the pandemic, Twitter had been aggressive in moderating misleading content regarding how the virus spreads, unproven remedies and treatments, and other health related info. Its new policy expands on that, mainly to focus on false information and conspiracy theories regarding vaccines.

Twitter won't be limiting itself to applying warnings to tweets with dubious content. The platform will force users to delete tweets that don't comply with its expanded code of conduct. Added to restrictions on misinformation about the spread of the disease and its morbidity rates are bans on false claims about immunization safety or COVID's dangers.

Decisions for Twitter:

• If there are actual stories of harm or side effects from vaccines, how will Twitter distinguish those reports from the conspiracy theories?
• Who will Twitter consider to be trusted sources for determining which information is factual, and which are conspiracy theories?
• Is there an effective way to teach users that they are sharing false information, rather than simply banning them?

Questions and policy implications to consider:

• Does moderating COVID conspiracy theories inadvertently lend "credence" to conspiracy theorists?
• Does more direct moderation of certain content invite even more Congressional scrutiny and calls for direct government regulation?
• Will human moderation help mitigate possible collateral damage to "good faith" users who aren't aware they're spreading misinformation?

Resolution: Twitter's expanded ban on misinformation is crafted to specifically target conspiracy theories about vaccinations or the disease's existence. The site will use a combination of AI and human moderators to determine whether flagged tweets should be labeled as questionable or removed entirely. Repeat violators can expect to have their accounts suspended or deleted.

The company also suggests that in early 2021 it will likely expand this fight against vaccine misinformation by placing warnings or labels on content that is unsubstantiated, disputed, or without necessary context.

Originally posted to the Trust & Safety Foundation website.

Filed Under: content moderation, covid, disinformation, misinformation, vaccines
Companies: twitter