Content Moderation Case Study: Vimeo Sued For Encouraging Infringement Via 'Lipdubs' (2009)

from the inducement? dept

Summary: Vimeo is a video hosting site that was originally founded in 2004 as an offshoot of CollegeHumor. IAC acquired the company early on and it tried to position itself as an alternative to YouTube that was more for creators. In the early years, one thing that was common on Vimeo was so-called “lipdubs.” These were music videos often made by groups of people lipsync’ing to popular songs.

Vimeo itself effectively launched this trend with its staff doing a lipdub of the Harvey Danger song “Flagpole Sitta.” At the end of the video, you hear one Vimeo employee say “ we just made a million dollars, people!” And, indeed the phenomenon helped establish Vimeo’s place in the market. Soon there were many other lipdubs all over Vimeo.

However, lipdubs also caught the attention of the music industry, which noticed that the songs in these lipdubs had not been licensed. In 2009, EMI subsidiary Capitol Records sued Vimeo for copyright infringement. Like many web services hosting user generated content, Vimeo relied on the safe harbors of the Digital Millennium Copyright Act (DMCA) in the US, which holds that if you meet certain conditions, you cannot be held liable for user uploads of infringing works.

Capitol Records argued that Vimeo was not protected under the DMCA for a variety of reasons, starting with the fact that it uploaded its own lipdub, which then effectively encouraged others to upload similar lipdubs. The complaint noted that the DMCA only protects against user uploads, not ones done by the company itself. It also claimed that since Vimeo employees were seen to have “liked” or commented on many other lipdubs, often speaking approvingly of the videos, it also meant that the company had so-called “red flag knowledge” of the infringing content -- which might also remove the DMCA’s safe harbor protections from the company.

Decisions to be made by Vimeo:

• Should employees have created their own lipdub to kick off this kind of trend? If they did, should they have first licensed the music?
• How should the company handle copyright-covered music?
• Should employees like or comment videos of users if they’re uploading music for which it’s likely they do not hold the rights?
• Should the company continue to rely on the DMCA or seek out licenses?
• Should the company explore alternatives for moderation beyond just taking down videos based on notices?

Questions and policy implications to consider:

• The songs included in lipdubs often ended up getting lots of public recognition and renewed attention. Given the promotional value of these videos, should these kinds of videos be seen as beneficial, rather than infringing?
• Should a lipdub be considered fair use?

Resolution: The Capitol Records v. Vimeo lawsuit went on for many years, bouncing around the courts. In an initial ruling, the district court judge noted that while some videos were likely protected under the DMCA’s safe harbors, many others were not (including those uploaded by the company itself).

On appeal, however, the 2nd circuit found that just because some employees within a company had clearly seen the videos in question, that did not mean the company had “red flag knowledge”, and the DMCA’s safe harbors still applied.

The mere fact that an employee of the service provider has viewed a video posted by a user (absent specific information regarding how much of the video the employee saw or the reason for which it was viewed), and that the video contains all or nearly all of a copyrighted song that is “recognizable,” would be insufficient for many reasons to make infringement obvious to an ordinary reasonable person, who is not an expert in music or the law of copyright.

Originally posted to the Trust & Safety Foundation website.

Filed Under: content moderation, copyright, employees, inducement, lipdubs, moderation
Companies: vimeo

Content Moderation Case Study: Friendster Battles Fakesters (2003)

from the revenge-of-the-fakesters dept

Summary: While the social media/social networking space today is dominated by Facebook, it’s interesting to look at how Facebook’s predecessors dealt with content moderation challenges as well. One of the earliest social networks to reach mainstream recognition was Friendster, founded by Jonathan Abrams in 2002 and launched in early 2003, gaining millions of users who signed up to connect with friends. Originally built as a dating site, it expanded quickly beyond that.

One of the first big content moderation questions that the site faced was whether or not to allow “fakesters.” As the site grew rapidly, one popular usage was to set up fake accounts -- these were accounts for completely made up fictional characters (e.g., Homer Simpson), concepts (e.g., Pure Evil), random objects (e.g., Giant Squid), or places (e.g., New Jersey). Researcher danah boyd catalogued the different types of fakesters and studied the phenomenon of fake accounts on the site.

However, Abrams quickly decided that “fakester” went against the ethos of the site he envisioned. In a 2003 article in SF Weekly that discusses the “fakester” issue, Abrams makes it clear that such accounts do not belong on the site, even if some people find them amusing:

In early July, Friendster's affable chief operating officer, Kent Lindstrom, told me the only fakesters that the company would likely remove would be ones it received complaints about. (On Friendster, users can “flag” somebody's profile for the company to review, and write comments about why it offended them.) But Abrams shakes his head emphatically when I mention this.

“No. They're all going,” he says, his voice steely. “All of them.”

As the article notes, the fakesters were often the most active users on the site, but that did not change Abrams’ mind about whether or not they belonged there:

Though they are some of Friendster's most ardent fans — many spend several hours a day on the site — fakesters do everything they can to create anarchy in the system. They are not interested in finding friends through prosaic personal ads, but through a big, surreal party where Jesus, Chewbacca, and Nitrous are all on the guest list. To fakesters, phony identities don't destroy the social experience of Friendster; they enrich it.

But fakesters aren't hosting this gig. Jonathan Abrams, the 33-year-old software engineer who founded Friendster to improve his own social life, is — and he abhors the phony profiles. He believes they diminish his site's worth as a networking tool and claims that fakesters' pictures — often images ripped off the Web — violate trademark law. Abrams' 10-person Sunnyvale company has begun ruthlessly deleting fakesters and plans to eventually eradicate them completely from the site.

A few months later, an article in Salon laid out the growing conflict between those who found the “fakesters” to be fun, and Abrams who remained adamantly against them.

Giant Squid is not alone: Among the "Fakesters" who've signed up for Friendster are Jackalope, God, Beer, Drunk Squirrel, Hippie Jesus, Malcolm X and more than a dozen Homer Simpsons. Just like regular users, they post their photos, blab on bulletin boards and collect friends like so many baseball cards. Some, staying in character, even write gushing testimonials about their friends: What higher endorsement could there be than a few complimentary words from Homer himself? "Better than a cold can of Duff beer ... "

But while it may be amusing to invite God himself into your pool of friends and get back the message, "God is now your friend," the founder of the site says that such chicanery only distorts his system.

"Fake profiles really defeats the whole point of Friendster," says entrepreneur Abrams, interviewed by cellphone as he waited to catch a plane in Los Angeles. "Some people find it amusing, but some find it annoying. And it doesn't really serve a legitimate purpose. The whole point of Friendster is to see how you're connected to people through your friends," he says.

Decisions to be made by Friendster:

• Should it delete any and all “fakester” profiles? Should it only do so if users complain about a specific profile, or should it be proactive in removing such profiles?
• How should the company deal with nicknames, rather than a person’s real name? How would it distinguish between celebrities and those pretending to be celebrities?
• Does removing “fakesters” harm some of the “fun” aspects that brought people to the site in the first place?

Questions and policy implications to consider:

• Does forcing everyone to be on the site with their real name lead to less participation from more vulnerable and at risk populations, worried about putting their profiles online?
• How much should web sites cater to emergent behavior, like fakesters, which some users enjoyed and resulted in more usage of the site?
• Are there legal risks associated with allowing fake profiles?

Resolution: While Friendster continued its fights against “fakester” profiles, apparently the company’s vehement stance against such profiles did not apply to monetization opportunities. In the summer of 2004, some people noticed an advertising campaign on Friendster for the “Anchorman” movie with Will Ferrell in which his character, Ron Burgandy was suggested as a “friend” to users.

When asked about this, Friendster tried to frame this situation as different than the “fakester” issue, saying that it was a “new paradigm.”

What Friendster is doing with these movie-character profiles is actually a brand-new paradigm in media promotion," Friendster spokeswoman Lisa Kopp said. "We are working directly with a number of production houses and movie studio partners to create film-character profiles, or 'fan' profiles, that allow our users to share their enthusiasm about the film with their friends."

The company also claimed that it wasn’t “fake” because it was done in partnership with the movie studio Dreamworks, which had the rights to the character:

"The issue here is actually about consumer protection," said Kopp. "We do, as a policy, strongly discourage fake profiles. A rogue user hiding behind a Jesus profile, for example, has the potential to abuse the service or users in many ways. In the case of the Anchorman characters, DreamWorks owns the rights to the characters and there is nothing fraudulent about it."

Of course, many of the “fakester” profiles didn’t involve anyone else’s intellectual property, so this excuse wouldn’t apply to accounts like “Pure Evil” and “Giant Squid.”

Friendster struggled to grow, in part because its own success overwhelmed its technical abilities. The site was quickly overtaken by MySpace and then Facebook. Since then, other sites, including Facebook, have struggled with the question of whether or not accounts should have “real names.” While many have argued that such policies discourage bad behavior, studies on this point have suggested otherwise.

Originally posted to the Trust & Safety Foundation website.

Filed Under: content moderation, fakesters, real names
Companies: friendster

Content Moderation Case Study: Lyft Blocks Users From Using Their Real Names To Sign Up (2019)

from the scunthorpe-again? dept

Summary: Users attempting to sign up for a new ride-sharing program ran into a problem from the earliest days of content moderation. The "Scunthorpe problem" dates back to 1996, when AOL refused to let residents of Scunthorpe, England register accounts with the online service. The service's blocklist of "offensive" words picked out four of the first five letters of the town's name and served up a blanket ban to residents.

Flash forward twenty-three years and services still aren't much closer to solving this problem.

Users attempting to sign up for Lyft found themselves booted from the service for "violating community guidelines" simply for attempting to create accounts using their real names. Some of the users affected were Nicole Cumming, Cara Dick, Dick DeBartolo, and Candace Poon.

These users were asked to "update their names," as though such a thing were even possible to do with a service that ties names to payment systems and internal efforts to ensure driver and passenger safety.

Decisions to be made by Lyft:

• Should names triggering Community Guidelines violations be reviewed by human moderators, rather than automatically rejected?
• Is the cross-verification process enough to deter pranksters and trolls from activating accounts with actually offensive names?

Questions and policy implications to consider:

• Considering the identification system is backstopped by credit cards and payment services that require real names, does deploying a blocklist actually serve any useful purpose?
• Given that potential users are likely to abandon a service that generates too much friction at sign up, does a blocklist like this do damage to company growth?
• Does global growth create a larger problem by adding other languages and possible names that will trigger rejections of more potential users? Can this be mitigated by backstopping more automatic processes with human moderators?

Resolution: The users affected by Lyft's blocklist were reinstated. Lyft apologized for the rejections, pointing a finger at automated moderation efforts designed to keep people from creating offensive content using nothing more than the First Name/Last Name fields.

Unfortunately, the problem still hasn't been solved. Candace Poon -- whose first attempt to sign up for Lyft was rejected -- just ran into the same issue attempting to create an account for new social media platform, Clubhouse.

Originally posted to the Trust & Safety Foundation website.

Filed Under: content moderation, filtering, keywords, names, scunthorpe problem
Companies: lyft

Content Moderation Case Study: Dealing With Podcast Copycats (2020)

from the podcast-yourself,-but-not-someone-else dept

Summary: Since the term was first coined in 2004, podcasts have obviously taken off, with reports saying that around 55% of Americans have listened to a podcast as of early 2021. Estimates on a total number of podcasts vary, but some sites estimate the total at 1.75 million podcasts, with about 850,000 of them described as “active.” Still, for many years, actually hosting a podcast remained somewhat complicated.

A few services have been created to try to make it easier, and one of the biggest names was Anchor.fm, which tried to make it extremely easy to create and host a podcast -- including the ability to add in an advertising-based monetization component. In early 2019, as part of its aggressive expansion into podcasts, Spotify purchased Anchor for $150 million.

However, in the summer of 2020, podcasters began calling out Anchor for allowing others to re-upload copies of someone else’s podcasts, claim them as their own, and monetize those other podcasts. Erika Nardini from Barstool Sports called this out on Twitter, after seeing a variety of Barstool podcasts show up on Anchor, despite not being uploaded there by Barstool.

The issue got a lot more attention a month later when podcaster Aaron Mahnke wrote a thread detailing how a variety of popular podcasts were being reuploaded to Anchor and monetized by whoever was uploading them.

After that thread started to go viral, Anchor promised to crackdown on copied/re-uploaded podcasts. The company claimed that it had an existing system in place to detect duplicates, but that those doing the uploading had figured out some sort of workaround, by manually uploading the podcasts, rather than automating the effort:

The copycats, Mignano says, found a workaround in Anchor’s detection system. “This is definitely a new type of attack for Anchor,” he says. The people who uploaded these copycat shows downloaded the audio from another source, manually reuploaded it to Anchor, and filled in the metadata, essentially making it appear to be a new podcast.

This manual process, he says, makes uploading copycat shows more time-intensive and therefore less appealing and only achievable on a small scale. He says the company found “a few dozen” examples out of the more than 650,000 shows uploaded to Anchor this year. (In contrast, people can also upload shows more automatically by pasting an RSS feed link into Anchor, but the company would seemingly detect if someone tried to upload a popular show’s feed and pass it off as their own.)

“The good news is that so many creators are using Anchor, and that growth has been far more than I think we projected, which is great, but I think the downside in this case is that, with any rapidly growing platform, that has brought on some growing pains and we need to do a better job of anticipating things like this,” he says. “We’re working right now to ensure that our copycat detection and creator outreach continues to improve to keep pace.”

Decisions to be made by Anchor/Spotify:

• How do you detect which podcasts are copies from elsewhere, especially when the original versions may not have originated on Anchor?

• People can always work around attempts to block copycats, so what kind of review process can be put in place to prevent abuse?

• Will being too aggressive at preventing abuse potentially lead to taking down too much? For example, what if one podcast uses clips from another for the purpose of commentary?

• Should there be extra validation or other hurdles to turn on monetization?

Questions and policy implications to consider:

• What are the trade-offs in making it especially easy to host, distribute and monetize podcasts? Is it worth making it so easy when that process will likely be abused?

• Is there a middle ground that allows for the easy creation, distribution and monetization of audio content that won’t be as subject to abuse?

• Is there a risk that cracking down on copycat content itself could go too far and silence commentary and criticism?

While Anchor/Spotify continue to update their practices, in the midst of all of this, another story came out (with less attention) showing how being too aggressive in stopping copycats can also backfire. This story actually came out a month before Anchor said it would beef up its process for stopping copycats, and involved a podcaster who wanted to test out Anchor to see if he should recommend it on his own podcast. That podcaster, Jonathan Mendonsa, posted a test episode to Anchor to see how it works, only to have his entire account shut down with no explanation or notice.

Mendonsa was surprised to find out that Anchor was comparing audio he uploaded to Anchor to audio uploaded elsewhere, and felt that the decision to completely shut down his account immediately was perhaps a step too far. From the story at PodNews:

Jonathan told Podnews: “I was testing Anchor to see if I would recommend it to my podcast course students. This ”duplicate content" caused them to not only take down the episode but to actually shut down my account entirely without no recourse or notice. What does that mean when a podcaster wants to republish an old episode? Or use a clip from another episode?"

For this show to have been pulled within two hours of posting must mean that Anchor is automatically comparing audio uploaded to their platform with all audio already available on Spotify - since this audio was identical to an episode already there.

“Anchor needs to clarify the tech they are using, and what triggers this,” Jonathan told us. “I never considered that any podcast platform would be looking for duplicate content, so I just used the same trailer. I wouldn’t be mad if it got flagged or the episode got unpublished - but to delete the entire account?”

It’s interesting to note the challenges on both sides of this issue, with some being upset that Anchor makes it too easy to distribute duplicate content, and others being upset at how quickly and aggressively it responds to duplicate content.

Originally posted to the Trust & Safety Foundation website.

Filed Under: content moderation, copycats, duplicates, podcasts
Companies: anchor.fm, spotify

Content Moderation Case Study: NASA Footage Taken Down By YouTube Moderation (2012)

from the moderating-the-public-domain dept

Summary: NASA's historic landing of a mobile rover on the surface of Mars created many newsworthy moments. Unfortunately, it also generated some embarrassing takedowns of NASA's own footage by YouTube's copyright flagging system, ContentID.

NASA uploaded its own footage of the landing -- footage that was public domain given that it was a work created by the U.S. government. US copyright law says that any work created by the US government cannot be granted a copyright. It is, instead, in the public domain, and can be used by anyone without restriction. Unfortunately, the use of this public domain footage in news broadcasts created by Scripps (owner of multiple TV stations as well as a significant ABC stakeholder) put a chain of events in motion that ultimately saw NASA's footage taken down by YouTube.

The problem was YouTube's ContentID. Once Scripps uploaded its footage, ContentID bots began crawling the site seeking matches. ContentID is structured in such a way that it believes that if content is uploaded by a private entity, that entity has a legitimate copyright over it -- even if the content is in the public domain, shareable freely by all. Shortly thereafter, NASA footage began disappearing while Scripps' broadcasts utilizing NASA footage stayed live.

This unfortunate outcome is common enough NASA reps are used to having to contact YouTube to get their own public domain uploads restored.

Decisions to be made by YouTube:

• Should this collateral damage to footage uploaded by government agencies continue to be an acceptable outcome of automated moderation?

• Should YouTube consider a "three strikes" system that punishes false copyright claims, rather than just those wrongly targeted by ContentID matches?

• Should YouTube require more input and detail from copyright claimants in order to head off errors like these?

• How should ContentID deal with material that is in the public domain?

Questions and policy implications to consider:

• Does erring on the side of copyright holders (even when the rightsholders are wrong) decrease the chance of YouTube being involved in copyright litigation?

• Is it possible to backstop newsworthy events involving government agencies (and their recordings) with more human moderators? Or is it impossible to decide what's newsworthy or not given the immense amount of footage uploaded to the site every day?

• Is overblocking -- including public domain works -- a reasonable price to pay to avoid infringement?

Resolution: While these sorts of mistaken takedowns are less frequent, they still occur. The most recent removal of NASA footage from YouTube was the result of a (likely automated) copyright claim by National Geographic, which had incorporated NASA's recordings into its own production. Given YouTube's dominant position in the video marketplace -- which has resulted in it processing thousands of hours of uploads every day -- it seems unlikely a solution that pleases every competing stakeholder will ever arrive.

Originally published to the Trust & Safety Foundation website.

Filed Under: content moderation, contentid, copyright, mars rover, nasa, public domain, video
Companies: youtube

Content Moderation Case Study: Twitter Briefly Bans Russian Parody Accounts (2016)

from the parody-in-action dept

Summary: Twitter allows parody accounts to remain live (often over the protests of those parodied), provided they follow a narrow set of rules -- rules apparently intended to make sure everyone's in on the joke.

Here's everything Twitter users agree to do when creating a parody account:

• Bio: The bio should clearly indicate that the user is not affiliated with the subject of the account. Non-affiliation can be indicated by incorporating, for example, words such as (but not limited to) "parody," "fake," "fan," or "commentary.” Non-affiliation should be stated in a way that can be understood by the intended audience.

• Account name: The account name (note: this is separate from the username, or @handle) should clearly indicate that the user is not affiliated with the subject of the account. Non-affiliation can be indicated by incorporating, for example, words such as (but not limited to) "parody," "fake," "fan," or "commentary.” Non-affiliation should be stated in a way that can be understood by the intended audience.

Unfortunately for the very popular Vladimir Putin parody account (@DarthPutinKGB), Twitter's moderators decided the account didn't strictly adhere to the "make it obvious" policies covering accounts like these.

In May 2016, Twitter suspended the account for its alleged violations.

This ban immediately resulted in backlash from other Twitter users who were fans of the account -- one that made it clear (albeit without all the specifics demanded by Twitter) that it was a parody. Disappointed fans included Estonian president Toomas Hendrik and Radio Free Europe, which published a collection of the account's best tweets.

While the ban was technically justified by the violation of the specifics of Twitter's rules, the end result was a lot of Twitter users wondering whether Twitter moderators were capable of recognizing obvious parody without accounts bios copying the platform's parody guidelines word-for-word.

Decisions to be made by Twitter:

• Is the banning of harmless parody accounts an acceptable tradeoff for protecting users from impersonation?

• Should the parody guidelines be altered to make it easier to identify parody accounts?

• Should moderators be allowed to make judgment calls if an account is clearly a parody but does not strictly adhere to the parody account guidelines?

Questions and policy implications to consider:

• Should Twitter use more caution when moderating parody accounts whose parodic nature isn't immediately clear?

• Is impersonation too much of a problem on the platform to ever relax the standards governing this kind of humor?

Resolution: Twitter swiftly reinstated the account following the backlash. The account remains active, despite its new bio not explicitly following the Twitter Rules for parody accounts.

But it wasn't the first time Twitter moderated accounts parodying Russian government officials. A similar thing happened roughly a year earlier, when Twitter blocked an account parodying powerful Russian oil executive Igor Sechin, apparently in response to a Russian government complaint the satirical account "violated privacy laws." This happened despite the fact the user's handle was IgorSechinEvilTwin, making it clear it was a parody, rather than an attempt to impersonate the real Igor Sechin.

Originally published on the Trust & Safety Foundation website.

Filed Under: content moderation, darthputin, parody, russia
Companies: twitter

Content Moderation Case Study: Automated Copyright Takedown Bot Goes Haywire (2018)

from the take-it-all-down dept

Summary: For years, Google and YouTube have included a trusted flagger program by which certain entities that have shown they “are particularly effective at notifying YouTube” of content violations are given more powerful tools with which to do so.

This is used often in the copyright context, and companies with a good history may be given access to things like bulk flagging tools and priority review of flagged content. One such trusted flagger for copyright was a company called Topple Track, which offered an automated service for musicians, searching the internet for infringing works and dashing off automated DMCA notices.

In May of 2015, digital music distribution company Symphonic purchased Topple Track, but appeared to keep the service running under its own brand.

In the summer of 2018, some people noticed that Topple Track’s automated DMCA notices appeared to go a bit haywire, sending DMCA notices for all kinds of perfectly legitimate content. Among those targeted with DMCA notices were the Electronic Frontier Foundation (EFF), the American Bar Association, NYU’s Law Review, the Crunchbase article about the company MP3Tunes and many, many more -- including many artists’ own web stores. EFF’s summary of the wild takedowns gives a sample

Among others, these notices improperly target:

• EFF’s case page about EMI v MP3Tunes
• The authorized music store on the official homepage of both Beyonce and Bruno Mars
• A fundraising page on the Minneapolis Foundation’s website
• The Graceland page at Paul Simon’s official website
• A blog post by Professor Eric Goldman about the EMI v MP3Tunes case
• A Citizen Lab report about UC Browser
• A New Yorker article about nationalism and patriotic songs

Other targets include an article about the DMCA in the NYU Law Review, an NBC News article about anti-virus scams, a Variety article about the Drake-Pusha T feud, and the lyrics to ‘Happier’ at Ed Sheeran’s official website. It goes on and on.

EFF published an article about this and noted that it seemed as yet another example of an automated DMCA reporting bot “running amok.” The group also questioned why such a company was in Google’s “trusted flagger” program.

Decisions to be made by Google / YouTube:

• What qualifications are there for a partner to be considered a “trusted flagger”?

• How often are trusted flaggers reviewed to make sure they still belong in the program?

• What does it take to get a trusted flagger removed from the program?

Questions and policy implications to consider:

• With more emphasis on the speed of removals, it is often tempting for regulators to promote “trusted flagging” or “priority” accounts that are able to get content removed at a much quicker pace. What are the benefits and risks of such programs?

• Automated flagging and now AI/Machine Learning flagging are increasingly a part of the content moderation landscape. How are they calibrated? How frequently are they reviewed?

• What should the response be when an automated bot is flagging many accounts mistakenly?

Resolution: After the EFF published its article about Topple Track, the parent company Symphonic Distribution apologized to the organization, saying: “bugs within the system that resulted in many whitelisted domains receiving these notices unintentionally.” As EFF pointed out in response, this seemed difficult to believe, seeing as the problem was not simply a mistake in domains that shouldn’t have been scanned, but simply claiming stuff that had nothing to do with the underlying copyright-covered material.

A few weeks after the article, YouTube also told EFF that Topple Track had been removed from its Trusted Flagger program, “due to a pattern of problematic notices.” Some time after this, Topple Track, as a unique organization appeared to disappear, and the service and technology have apparently been subsumed into Symphonic Distribution’s catalog of services.

Originally published on the Trust & Safety Foundation website.

Filed Under: automated takedowns, content moderation, copyright, dmca, trusted flagger
Companies: google, symphonic, topple track, youtube

Content Moderation Case Study: Facebook Removes A Picture Of A Famous Danish Mermaid Statue (2016)

from the the-nudity-issue dept

Summary: For over a century, Edvard Eriksen’s bronze statue of The Little Mermaid becoming human has been installed on a rock along the water in Copenhagen, Denmark. The statue was designed to represent the Hans Christian Anderson fairy tale, and has become a tourist attraction and landmark.

In 2016, Danish politician Mette Gjerskov used Facebook to post a link to her own blog post on the TV2 website, which included an image of the statue. Facebook automatically displayed the image with the link, leading the company to then take down the link. The explanation provided by Facebook was that the image had "too much bare skin or sexual undertones."

Gjerskov highlighted the absurdity of the situation, calling it “ridiculous” that the image caused Facebook’s moderators to block the link. Many people appeared to agree, and as the story began to get more attention, Facebook quickly backtracked and admitted the removal was in error. It restored the link.

Many of the news reports on the story concluded with Facebook’s reversal, but the image actually did not return to Facebook. Due to copyright law in Denmark, the statue is still considered to be covered by copyright (until 2029, 70 years after Eriksen’s death), and his estate has been fairly aggressive in demanding licensing and royalty payments. Because of that, TV2, which hosted Gjerskov’s blog, chose to remove the image that caused the takedown in the first place -- not to appease Facebook’s moderation, but to avoid a copyright issue from the Erikson estate, even though a copyright on the statue itself is different from copyright on images of the statue.

Decisions to be made by Facebook:

• How do you write rules regarding nudity that take into account art or cultural landmarks?

• Is taking down a link due to images that are automatically embedded via the OpenGraph feature the best solution? Would it make sense to simply remove that image while leaving the link, or have a different image show?

Questions and policy implications to consider:

• The line between artistic works that depict nudity and works that violate a nudity policy or include “sexual overtones” is often a very subjective judgment call. How can companies craft rules that are enforced consistently across a diverse set of moderators, often with different cultural backgrounds and experiences?

• A strict policy against nudity is likely to capture many artistic situations. Is that a reasonable trade-off for websites that seek to be family friendly?

• How can copyright intersect with other types of challenges regarding content moderation?

Resolution: As noted in the case study, the link was restored after Facebook admitted error, but the image was taken off the website (and, thus, the link on Facebook) due to copyright concerns from TV2. Facebook’s policies already allow many forms of artistic nudity, but mistaken removals for nudity still feel common, given the huge scale of review decisions made on a daily basis. The statue has continued to be a cultural landmark in Denmark, and is often used for making political statements, leading to more photographs being shared of it. Just recently it was vandalized to promote democracy in Hong Kong and to protest racism.

Originally posted to the Trust & Safety Foundation website

Filed Under: content moderation, copyright, denmark, evard eriksen, mette gjerskov, the little mermaid
Companies: facebook

Content Moderation Case Study: Huge Surge In Users On One Server Prompts Intercession From Discord (2021)

from the moderating-game-stonks dept

Summary: A wild few days for the stock market resulted in some interesting moderation moves by a handful of communications/social media platforms.

A group of unassociated retail investors (i.e. day traders playing the stock market with the assistance of services like Robin Hood) gathering at the Wall Street Bets subreddit started a mini-revolution by refusing to believe Gamestop stock was worth as little as some hedge funds believed it was.

The initial surge in Gamestop's stock price was soon followed by a runaway escalation, some of it a direct response to a hedge fund's large (and exposed) short position. Melvin Capital -- the hedge fund targeted by Wall Street Bets denizens -- had announced its belief Gamestop stock wasn't worth the price it was at and had put its money where its mouth was by taking a large short position that would only pay off if the stock price continued to drop.

As the stock soared from less than $5/share to over $150/share, people began flooding to r/wallstreetbets. This forced the first moderation move. Moderators briefly took the subreddit private in an attempt to stem the flow of newcomers and get a handle on the issues these sort of influxes bring with them.

Wall Street Bets moved some of the conversation over to Discord, which prompted another set of moderation moves. Discord banned the server, claiming users routinely violated guidelines on hate speech, incitement of violence, and spreading misinformation. This was initially viewed as another attempt to rein in vengeful retail investors who were inflicting pain on hedge funds: the Big Guys making sure the Little Guys weren't allowed on the playing field. (Melvin Capital received a $2.75 billion cash infusion after its Gamestop short was blown up by Gamestop's unprecedented rise in price.)

But it wasn't as conspiratorial as it first appeared. The users who frequented a subreddit that described itself as "4chan with a Bloomberg terminal" were very abrasive and the addition of mics to the mix at the Discord server made things worse by doubling the amount of noise -- noise that often included hate speech and plenty of insensitive language.

The ban was dropped and the server was re-enabled by Discord, which announced it was stepping in to more directly moderate content and users. With over 300,000 users, the server had apparently grown too large, too quickly, making it all but impossible for Wall Street Bets moderators to handle on their own. This partially reversed the earlier narrative, turning Discord into the Big Guy helping out the Little Guy, rather than allowing them to be silenced permanently due to the actions of their worst users.

Decisions to be made by Discord:

• Do temporary bans harm goodwill and chase users from the platform? Is this the expected result when this happens?

• Is participating directly in moderation of heavily-trafficked servers scalable?

• How much moderation should be left in the hands of server moderators? Should they be allowed more flexibility when moderating questionable content that may violate Discord rules but is otherwise still legal?

Questions and policy implications to consider:

• Are temporary bans of servers more effective than other, more scaled escalation efforts? Are changes more immediate?

• Is the fallout from bans offset by the exit of problem users? Or do server bans tend to entrench the worst users to the detriment of new users and moderators who are left to clean up the mess?

• As more users move to Discord, is the platform capable of stepping in earlier to head off developing problems before they reach the point a ban is warranted?

• Does offloading moderation to users of the service increase the possibility of rules violations? If so, should Discord take more direct control earlier when problematic content is reported?

Resolution: The Wall Street Bets Discord server is still up and running. Its core clientele likely hasn't changed much, which means moderation is still a full-time job. An influx of new users following press coverage of this particular group of retail traders may dilute the user base, but it's unlikely to turn WSB into a genteel community of stock market amateurs. Discord's assistance will likely be needed for the foreseeable future

Originally published on the Trust & Safety Foundation website.

Filed Under: content moderation, game stonks, short selling, stock trading, wall street
Companies: discord, gamestop, reddit

Content Moderation Case Study: Telegram Gains Users But Struggles To Remove Violent Content (2021)

from the influx-of-insurrectionists dept

Summary: After Amazon refused to continue hosting Parler, the Twitter competitor favored by the American far-right, former Parler users looking to communicate with each other -- but dodge strict moderation -- adopted Telegram as their go-to service. Following the attack on the Capitol building in Washington, DC, chat app Telegram added 25 million users in a little over 72 hours.

Telegram has long been home to far-right groups, who often find their communications options limited by moderation policies that, unsurprisingly, remove violent or hateful content. Telegram's moderation is comparatively more lax than several of its social media competitors, making it the app of choice for far right personalities.

But Telegram appears to be attempting to handle the influx of users -- along with an influx of disturbing content. Some channels broadcasting extremist content have been removed by Telegram as the increasingly-popular chat service flexes its (until now rarely used) moderation muscle. According to the service, at least fifteen channels were removed by Telegram moderators, some of which were filled with white supremacist content.

Unfortunately, policing the service remains difficult. While Telegram claims to have blocked "dozens" of channels containing "calls to violence," journalists have had little trouble finding similarly violent content on the service, which either has eluded moderation or is being ignored by Telegram. While Telegram appears responsive to some notifications of potentially-illegal content, it also appears to be inconsistent in applying its own rule against inciting violence.

Decisions to be made by Telegram:

• Should content contained in private chats (rather than public channels) be subjected to the same rules concerning violent content?

• Given that many of its users migrated to Telegram after being banned elsewhere for posting extremist content, should the platform increase its moderation efforts targeting calls for violence?

• Should a process be put in place to help prevent banned users/channels from resurfacing on Telegram under new names?

Questions and policy implications to consider:

• Does Telegram's promise of user security and privacy dissuade it from engaging in more active content moderation?

• Is context considered when engaging in moderation to avoid accidentally blocking people sharing content they feel is concerning, rather than promoting the content or endorsing its message?

• Do reports of mass content violations (and lax moderation) draw extremists to Telegram? Does this increase the chance of the moderation problem "snowballing" into something that can no longer be managed effectively?

Resolution: Telegram continues to take a mostly hands-off approach to moderation but appears to be more responsive to complaints about calls to violence than it has been in the past. As it continues to draw more users -- many of whom have been kicked off other platforms -- its existing moderation issues are only going to increase.

Originally posted to the Trust & Safety Foundation website.

Filed Under: chat, content moderation, violence
Companies: telegram