Content Moderation Case Study: Twitter Temporarily Locks Account Of Indian Technology Minister For Copyright Violations (2021)

from the copyright-underpins-so-much dept

Summary: In late June 2021, Twitter briefly suspended the account of Ravi Shankar Prasad, the Indian government's Technology Minister because his content violated copyright. Although Prasad was given a copy of the DMCA (Digital Millennium Copyright Act) takedown request, he posted a short Twitter thread that claimed this moderation action was Twitter silencing him, rather than simply responding to a takedown notice targeting alleged copyright violations.

According to Prasad, the removal of his tweets and brief suspension were the result of him not aligning himself with Twitter's "agenda." He claimed Twitter would "arbitrarily" do the same to others who did not "tow the line that they draw."

Prasad also claimed the takedown via DMCA notice violated Indian law. Specifically, the Technology Minister claimed Twitter violated local law by failing to provide him with prior notice before locking him out of his account. The law cited was Rule 4(8) of the Information Technology Rules 2021.

Prasad's pushback against Twitter's decision to suspend his account based on claims of copyright violations came after weeks of Indian government officials' public criticism of American social media companies. Shortly after this critique, the new Information Technology Rules law went into effect on May 26, 2021.

The new law affects multiple US social media platforms. It requires them to remove certain content within 24 hours of notification and forces them to set up local offices in India where complaints can be handled by newly-created compliance, contact, and grievance officers. While the new law has been in the works since 2018, its passage was accelerated by Twitter's refusal to block tweets/accounts associated with the long-running "Farmer's Protest" that greeted the Indian government's passage of laws that many farmers believe do more to aid corporate farms and large agricultural concerns than the many small, independent farmers that populate the country.

Earlier in June, the Indian government issued a "final warning" to Twitter, ordering it to comply with the new law or "face consequences." Twitter has since temporarily removed nearly 500 tweets associated with the Farmer's Protests. During the same time period, its offices in India were raided by Indian law enforcement after Twitter labelled a tweet from a member of India’s ruling party as "misinformation."

Company Considerations:

• Which country's laws should be applied to social media content? Should it be the country in which the tweet originated (in this case, India)? Or should it be the laws in which the headquarters of a platform is located (in this case, the US?) 
• Should a foreign person be subjected to another country's laws if they are using a platform of that other country?
• Should Twitter process DMCA counternotices (if any) before removing content and/or suspending alleged infringers?
• How should a company handle backlash from a government when moderating content violation issues such as a copyright violation?
• When laws conflict (as in this case, where Indian law requires notice and the opportunity to challenge the claim before content is moderated, whereas US law does not), can platforms find a middle ground that removes some of the friction (for instance, temporarily "hiding" the content while waiting for a counternotice to be served rather than immediately suspending accounts)? 
• Does showing consistency in moderation efforts (like those involving the DMCA process) help head off claims that some of these efforts are "politically-motivated?"

Issue Considerations:

• When it's impossible to please both sides (in this case, foreign government officials and content owners protecting their intellectual property), which party should be prioritized?
• Should pressure from foreign government officials be allowed to alter moderation efforts involving intellectual property owners?

Resolution: This chain of events appears to have resulted in government officials like Prasad believing normal moderation actions by Twitter are actually a form of censorship targeting a government that was recently highly critical of US social media services. This has led to even more problems for Twitter: the Indian government has filed a request with a New Delhi court to strip Twitter of its liability protections for user-generated content. For the time being, Twitter's approach to the handling of DMCA takedown notices appears unchanged, even when it involves the accounts of foreign government officials.

Originally posted to the Trust & Safety Foundation website.

Filed Under: censorship, content moderation, copyright, dmca, india, intermediary liability, it rules 2021, ravi shankar prasad
Companies: twitter

Content Moderation Case Study: Spotify Comes Under Fire For Hosting Joe Rogan's Podcast (2020)

from the podcast-moderation dept

Summary: In May of 2020 Joe Rogan, whose podcast was considered one of the most popular in the world, signed an exclusive deal to host the podcast on Spotify. Rogan, who has been described as “America’s Bro Whisperer” by the AV Club, is a comedian/actor whose podcast is known for extended discussions on a variety of topics. He frequently delves into controversial subjects and explores potentially extreme ideas, conspiracy theories, and hoaxes.

Given that history, it was not surprising that people quickly raised questions about why Spotify would “platform” Rogan amidst various controversies regarding the content on his show (both from some of his guests and from Rogan himself), and about how much responsibility Spotify should take for that content. Spotify already has a history, on the music side of its business, of removing some artists from its platform.

In the Fall of 2020, things came to a head when a group of Spotify employees complained to management about Spotify being a platform for Rogan, and complained about some particular Rogan content. Reports revealed an all-hands meeting in which Spotify founder and CEO Daniel Ek discussed these requests and explained his reasoning for keeping Rogan on the platform.

"In the case of Joe Rogan, a total of 10 meetings have been held with various groups and individuals to hear their respective concerns. And some of them want Rogan removed because of things he's said in the past." -- Spotify CEO, Daniel Ek

Spotify did say that it refused to host some of Rogan’s older episodes, but after reviewing some more recent ones, said that they did not violate the company’s policies. The company overall seems happy with having Rogan on its platform. A NY Times report says that “among top Spotify leadership, people familiar with the company say, the notion that Mr. Rogan presents any kind of regrettable executive headache is laughable.” However, there remain accusations that Spotify is simply profiting from the outrage over Rogan being on the platform. Spotify has said that Rogan is drawing more users than the company expected, and even has a special category for Rogan’s podcast. As the NY Times article notes: “So central is he to the company’s fortunes that the podcast is listed as its own category on the app: Sports. Music. News and Politics. Joe Rogan.”

More recently, concerns were raised after Rogan gave advice that could be seen as discouraging people from getting a COVID-19 vaccine, again leading to questions about how much Spotify should moderate the content of the podcast. Rogan clarified his comments and suggested that no one should listen to him for advice regarding medical issues. “I'm not a doctor," he said. "I'm not a respected source of information, even for me."

However, over the last few months, people have noticed that dozens of Rogan’s older podcast episodes have disappeared from Spotify. Some of these were first noticed in September of 2020, and by April 2021, it was reported that Spotify had deleted 42 older episodes. Another report noted that another 15 to 20 were “earmarked” for deletion but had not been deleted due to concerns about “bad optics.” Many fans of Rogan’s podcast reacted angrily to the news of some episodes being deleted, calling it censorship, and threatened to no longer use Spotify.

Company Considerations:

• How can a company effectively and efficiently moderate very long podcasts, like Rogan's, which can go up to 2-3 hours?
• How does a company draw the line between someone's opinion vs spreading misinformation?
• How should Spotify handle competing demands from two very strong constituencies: Rogan’s supporters and detractors?
• How much (if any) should bottom line financial interests be taken into account when considering content moderation questions?
• How much should internal employee complaints be taken into account in reviewing content moderation policies?
• Is removing older podcasts or podcasts with controversial guests or topics an acceptable strategy?

Issue Considerations:

• Will threats of stronger moderation or removal of podcasts lead to more careful consideration by podcast hosts and guests? Will it lead to important, but still controversial, topics not being discussed at all? Or will it result in a combination of both?
• Should internet platforms hosting podcasts be responsible for the content in those podcasts? Would the answer to this question be different if it’s an openly available podcast distributed via RSS on all platforms, compared to an exclusive podcast only available on a single, closed platform like Spotify?
• Is public pressure more effective at policing controversial content than expecting a platform to moderate the content? In the example of the information on vaccines, public outcry resulted in Rogan walking back his comments soon after.

Resolution: Joe Rogan has said that part of the reason he went to Spotify in the first place was that he was getting increasingly uneasy with YouTube (where he was hosting video versions of his show) after the company removed or demonetized other controversial podcasts and video shows.

And while he pushed back at some of the criticisms of his podcast, he also said that Spotify is not censoring the content of his show. He claims that he knew that Spotify did not want to host certain older episodes, but that the company has given him no rules regarding future content.

"They don't give a f**k man. They haven't given me a hard time at all. There were a few episodes they didn't want on their platform, I was like 'okay, I don't care'.

"But other than that, in terms of what I do in the future, the big test was having Alex Jones on... Alex Jones and Tim Dillon was like one of my favorite podcasts." -- Joe Rogan

He separately made some comments regarding employees at Spotify who were upset about content on his podcast.

“I don’t know what the actual conversation has been from Spotify talking to these employees. But if these employees are listening, I would tell you, emphatically, I am not in any way anti-trans. … I am 100 percent for people being able to do whatever they want, as long as it doesn’t harm other people.

“I’m talking off the top of my head. And a lot of times I’m saying shit that I don’t even mean. Cus I’m saying it because this is a fucking podcast. And if you have a problem with people saying terrible shit and you work for Spotify, maybe you should listen to some of the lyrics. Okay, cus some of the lyrics and some of the fucking music that you guys play over and over and over again makes my shit pale in comparison.

“But I get it, you’re a 23-year-old woke kid and you’re working for this company and you think you’re gonna put your foot down, I get it.” -- Joe Rogan

As for the overall situation, Rogan has suggested that the controversy around all of this has only helped him become more famous, though others have argued that he is having less impact on the wider zeitgeist and a review of how often his name is searched for suggests interest has decreased since the move to Spotify.

Originally published to the Trust & Safety Foundation website.

Filed Under: content moderation, joe rogan, podcasts
Companies: spotify

Content Moderation Case Study: Twitter Experiences Problems Moderating Audio Tweets (2020)

from the content-moderation-in-new-areas dept

Summary: Since its debut in 2007, Twitter hasn't changed much about its formula, except for expanding its character limit from 140 to 280 in 2007 and adding useful features such as lists, trending topics and polls. Twitter has embraced images and videos, adding it to its original text-only formula, but seemed to have little use for audio. That changed in June 2020 when Twitter announced it would allow users to upload audio-only tweets. Remaining true to the original formula, audio tweets were limited to 140 seconds, although Twitter will automatically add new audio tweets to a thread if the user's recording ran long.

With Twitter engaged in day-to-day struggles moderating millions of tweets, critics and analysts expressed concern the platform would be unable to adequately monitor tweets whose content couldn't be immediately discerned by other users. The content would be unable to be pre-screened by moderators -- at least not without significant AI assistance. But that assistance might prove problematic if it caused more problems than it solved by overblocking.

There was also the potential for harassment. Since abusive audio tweets relied heavily on other Twitter users reporting, abusive audio tweets could be posted and remain up until someone noticed and reported it. Another issue audio tweets raised wasn't about proactively flagging and removing unwanted content, but that this new offering excluded certain Twitter users from being a part of the conversation.

“Within hours of the first voice tweets being posted, deaf and hard-of-hearing users began to criticize the tool, saying that Twitter had failed to provide a way to make the audio clips accessible for anyone who can’t physically hear them.”

-- Kiera Frazier, YR Media

The new feature debuted without auto-captioning or any other options that would have made the content more accessible to Deaf or hard of hearing users.

There were other potential problems, such as users being exposed to possibly disturbing content with no heads up from the platform.

“‘You can Tweet a Tweet. But now you can Tweet your voice!’ This was how Twitter introduced last week its new audio-tweet option. In the replies to the announcement [another user asked], “Is this what y’all want?” ... reposting another user’s audio tweet, which used the new feature to record the sounds of… porn.

-- Hanna Kozlowska, OneZero

Unlike other adult content on Twitter, the recording of porn sounds was not labelled as sensitive by Twitter or hidden from users whose account settings requested they not be shown this sort of content.

Company considerations:

• Is it possible to proactively filter audio content to be flagged, prevented from being posted, or quickly removed?
• If an audio tweet was reported, should Twitter remove this feature from the user immediately or wait until it is reviewed? If the tweet violates Twitter's content policy, should they temporarily or permanently take away this feature from the user?
• Should unmoderated audio be labelled as sensitive if they are reported until cleared by moderators/AI?
• Should users be given the option to hide/block all audio tweets?

Issue considerations:

• What makes audio-only moderation different from text, image, or video moderating? Is audio-only moderation more challenging?
• What are other proactive methods of moderating audio content? Would they be more or less effective than relying on users flagging abusive content?
• Is AI reliable enough to handle most instances of unwanted content without the assistance of human moderators?
• How can platforms ensure that audio-only or visual-only content be accessible to those who have a hearing or visual disability or impairment?

Resolution: Twitter responded to the concerns of the Deaf and hard-of-hearing by apologizing for not considering the implications of an audio-only option.

“We're sorry about testing voice Tweets without support for people who are visually impaired, deaf, or hard of hearing. It was a miss to introduce this experiment without this support.

Accessibility should not be an afterthought.”

-- Twitter

The platform fixed some issues with visual accessibility and said it was implementing a combination of auto- and human-captioning to give Deaf persons a way to access this content.

As for the porn-audio tweet, Twitter flagged it after it was reported but did not appear to have any other approach to dealing with questions around adult content in audio tweets. It appears sensitive content is not as easy to detect when it's in audio form, which means that for now, it's up to users to report unwanted or abusive content so that Twitter can take action.

Originally published on the Trust & Safety Foundation website.

Filed Under: audio, audio tweets, content moderation
Companies: twitter

Content Moderation Case Study: Dealing With 'Cheap Fake' Modified Political Videos (2020)

from the political-moderation dept

Summary: For years now there have been concerns raised about the possibility of “deep fake” videos impacting an election. Deep fakes are videos that have been digitally altered, often to insert someone’s face onto another person’s body, to make it appear that they were somewhere they were not or did something they did not. To date, most of the more sophisticated deep fake videos have been mainly for entertainment purposes but there has been a concern that they could lead to faked accusations against politicians or other public figures. However, so far, there has been little evidence of deep fake videos being used in elections. This may be because the technology is not yet good enough or because such videos have been easy to debunk through other evidence.

Meanwhile, there has been increasing concern about something slightly different: cheap fake or shallow fake videos, which are just slight modifications and adjustments to real videos—less technically sophisticated, but also potentially harder to combat.

One of the most high profile examples of this was a series of videos that went viral on social media of House Speaker Nancy Pelosi that were modified by slowing down the video to 75% of the original speed. The modified videos were spread with false claims that they showed Pelosi slurring her words, possibly indicating intoxication. Various media organizations fact checked the claims, noting that the videos were altered and therefore presented a very inaccurate picture of Pelosi and her speech patterns.

Social media companies were urged by some to delete these videos, including by Speaker Pelosi herself, who argued that Facebook in particular should remove them. Both Facebook and Twitter refused to take down the video, saying that it did not violate their policies. YouTube removed the video.

In response to concerns raised by Pelosi, some highlighted that it would be impossible to expect social media to remove every misleading political statement that either took the words of an opponent out of context or presented them in a misleading way, while others suggested that there’s a clear difference when it comes to manipulated video as compared to manipulated text.

Others highlighted that it would be difficult to distinguish manipulated video from satire or other attempts at humor.

Company Considerations:

• Where should companies draw the line between misleading political content and deliberate misinformation?
• Under what conditions would a misleading cheap fake video separately violate other policies, such as harassment?
• What should the standards be for removing political content that could be deemed misleading?
• Does medium matter? Should there be different rules for manipulated videos as compared to other types of content, such as taking statements out of context? 
• Should there be exceptions for parody/satire?
• Are there effective ways for distinguishing videos that are manipulated to mislead vs. those that are manipulated for humor or commentary? 
• Should the company have different standards if the subject of a cheap fake video was not a political or public figure? 
• What are other alternative approaches, beyond blocking, that could be used to address manipulated political videos?

Issue Considerations:

• What are the possible unintended consequences if all “manipulated video” is deemed a policy violation?
• What, if any, is the value of not removing videos of political or public figures that are clearly misleading? Would there be any unintended consequences of such a policy?
• What are the implications for democracy if manipulated political videos are allowed to remain on a platform, where they may spread virally?
• Are misleading “cheap fake” videos about politicians considered political speech?
• Who should decide when “cheap fake” political speech is inaccurate and inappropriate—should it be social media platforms, a general public consensus, or a third body??
• How might “cheap fake” videos be used for harassment and bullying of non-public figures, and what are the potential implications for real life harm? 
• If the cheap fake video didn’t originate from a public source (as in the Pelosi video) but a private video, how could a company determine that those videos were manipulated?   

Resolution: After public demands that Twitter, YouTube and Facebook do something about the modified Pelosi video, the three major social media platforms each responded differently. YouTube took the video down, saying that it violated its policies. YouTube also noted that unlike the other platforms, the modified Pelosi video did not appear to go viral or spread widely on its platform. Facebook kept the video up but, in accordance with its fact-checking policies, once the video was deemed “false” Facebook de-ranked the video in its news feed, limiting its spread.

Twitter left the video up. However, by the time a very similar event happened a few months later, Twitter announced a new plan for dealing with such content, saying that it would begin adding labels to “manipulated media,” offering context for people who came across such videos so they would understand that the video is not being shown in its original context. One of the first examples of Twitter applying this “manipulated media” label was to a comedy segment by late night TV entertainer Jimmy Kimmel who used some manipulated video to make fun of former Vice President Mike Pence.

Originally posted to the Trust & Safety Foundation website.

Filed Under: cheap fakes, content moderation, deep fakes, modified content, nancy pelosi, politics, shallow fakes, social media
Companies: facebook, twitter, youtube

Content Moderation Case Study: Facebook Removes Image Of Two Men Kissing (2011)

from the rights-and-responsibilities dept

Summary: In the spring of 2011, two men were on a first date at the John Snow pub in London. They were apparently thrown out of the pub after another patron at the bar complained that the two men were kissing each other in the corner. The story of being thrown out of the pub for kissing began to go viral on social media, followed by a plan for a protest at the pub in question. In a sign of support for the protest, many people on social media posted images of two men kissing each other as well.

The Dangerous Minds Facebook page wrote about the protest, and included a promotional image from the BBC of two male characters from the popular soap opera EastEnders kissing to illustrate the post. The image had become well known in the UK a few years prior, as the scene in question had kicked off complaints to the BBC which the BBC responded to by noting that the relationship between two men was treated no differently than many other relationships displayed on the show between a man and a woman. Soon after this, Facebook removed the image, telling Dangerous Minds that it “violated Facebook’s Statement of Rights and Responsibilities.”

Soon after this, Facebook removed the image, telling Dangerous Minds that it “violated Facebook’s Statement of Rights and Responsibilities.”

“Hello,

Content that you shared on Facebook has been removed because it violated Facebook’s Statement of Rights and Responsibilities. Shares that contain nudity, or any kind of graphic or sexually suggestive content, are not permitted on Facebook.

This message serves as a warning. Additional violations may result in the termination of your account. Please read the Statement of Rights and Responsibilities carefully and refrain from posting abusive material in the future. Thanks in advance for your understanding and cooperation.

The Facebook Team”

Given that the image itself was to promote a protest over two men being removed from a restaurant for kissing, the removal of the image sparked further anger directed at Facebook. Many pointed out that there were plenty of images of men and women kissing each other all over Facebook, and they were not being removed.

Decisions to be made by Facebook:

• Is a picture of two people kissing “graphic or sexually suggestive content” in violation of rules
• Should the identities of the people within the images matter in making content moderation decisions?
• Should the reason for the posting of an image (e.g., as part of a larger protest) matter in determining if the image violated the rules?
• How well trained is the content moderation staff to recognize relationships that might differ from their own preferred relationships?

Questions and policy implications to consider:

• Should Facebook be deciding which kinds of legal relationships are acceptable to display and which are not?
• As seen with the complaints to the BBC, there remain some segment of the population who will vocally protest non-heterosexual relationships. Are there rules, policies, or training that can be put in place to educate content moderation teams that they may receive different types of reports regarding such content?

Resolution: As the story drew more and more mainstream press attention, Facebook quickly apologized, saying the image does not violate any policies, and the takedown was a mistake.

“The photo in question does not violate our Statement of Rights and Responsibilities and was removed in error. We apologize for the inconvenience”

No further explanation of how the image came to be taken down in the first place was given.

Originally posted to the Trust & Safety Foundation website.

Filed Under: content moderation, kissing, protests, sexuality
Companies: facebook

Content Moderation Case Study: Instagram Takes Down Instagram Account Of Book About Instagram (2020)

from the seems-like-a-problem dept

Summary: Three professors, Tama Leaver, Tim Highfield and Crystal Abidin, wrote a book about culture on Instagram and how it developed. The book, entitled Instagram: Visual Social Media Cultures, was released in February of 2020. Along with the book, the authors set up social media accounts to both promote the book and to continue the discussion about how Instagram culture has developed. Not surprisingly, one of the social media accounts they set up was on Instagram itself.

On Instagram, the account would post images about Instagram (including examples of its content moderation issues). The authors were surprised in mid-September when Instagram shut down their account without any clear reason.

The authors submitted an appeal saying that they believed the takedown was in error, noting the nature of their work, and explaining why they did not believe the account’s reposting of others’ work as part of their research should violate copyright (though, the disabled account notice did not specify that it was for copyright infringement):

Our account has been disabled in error. The ‘polityinstabook’ account was used specifically for research purposes, by three visual social media researchers: Associate Professor Tama Leaver (Curtin University), Dr Tim Highfield (University of Sheffield) and Dr Crystal Abidin (Curtin University). We have collectively all done considerable research on Instagram, evident in our co-authored book ‘Instagram: Visual Social Media Cultures’ published this year by Polity Press. This account was set up to document our continued research on Instagram. It seems likely that the account was mistakenly disabled as some reposted content may appear at a glance to violate copyright. However these images have been reposted under the allowances of FAIR USE, for the purposes of SCHOLARLY RESEARCH. This account is used to highlight platform and cultural changes and impacts of Instagram, documenting them for research purposes (as is stated in our bio).

We respectfully request you review the disabling of our account and would appreciate the account being reactivated at your earliest convenience.

The authors also took to Twitter to generate some attention for the fact that their Instagram account was removed.

Decisions to be made by Instagram:

• How much information should be provided to the operators of disabled accounts about why their account was disabled?
• How should the company review appeals on disabled accounts, when the users are not told why their account was disabled?
• Should academic researchers be treated differently than other types of users?

Questions and policy implications to consider:

• Providing information about why an account was taken down makes it easier for those wrongly taken down to appeal and explain their story, while at the same time potentially making it easier for malicious actors to game the system. How should a company balance those competing goals?
• Instagram is likely more aggressive in taking down accounts that have Instagram in their name, to avoid users believing the account is coming from the company itself. Is there a way to balance those interests with allowing authors to promote their book about the platform?

Resolution: Soon after the appeal was sent in, the account was reinstated, and Instagram sent the authors an apology, saying that disabling the account was a mistake.

It looks like your account was disabled by mistake. Your account has been reactivated, and you should now be able to log in. We’re sorry for any inconvenience.

If you have any issues getting back into your account, please let us know.

Thanks, The Instagram Team.

No further explanation was given to the authors.

Originally published to the Trust & Safety Foundation website.

Filed Under: books, content moderation, trademark
Companies: instagram

Content Moderation Case Study: YouTube Relocates Video Accused Of Inflated Views (2014)

from the movin'-videos dept

Summary: The internet is the way that many new musical artists get discovered these days, with perhaps the most famous story being that of that of Justin Bieber on YouTube. Some of this came from finding undiscovered musicians who had talent, and some of it came from finding otherwise unsigned artists who had managed to build large followings themselves.

Of course, this latter situation also opened up the possibility of gaming the system to appear more popular than you are in reality. Partly in response to this -- and more likely to prevent gaming views in order to gain advertising revenue -- YouTube put in place a policy of removing videos that appeared to use automated systems to game the number of views.

An independent musician by the name of Darnaa sought to gain a following via YouTube, and engaged in a marketing campaign designed to drive traffic and popularity to her videos. In 2012 she had uploaded a video that had received nearly 1.9 million views according to YouTube’s counter. In 2013, another video received over 1.1 million views. In 2014, she uploaded a new video, for a song entitled Cowgirl, which started receiving views as well. Darnaa claimed that these came from a coordinated marketing campaign that cost her hundreds of thousands of dollars.

YouTube, however, believed that the views on the video were inflated through artificial means, violating the terms of service. Rather than simply removing the video, or shutting down Darnaa’s videos, the service simply moved the video to a new URL, resetting the counter (and breaking earlier links to the video). Darnaa sent an email complaining about this, and convinced her marketing partners to restart the marketing campaign, leading to YouTube relocating the video a second time, which again, reset the view counter.

Darnaa’s music label, the conveniently named Darnaa, LLC. then sued YouTube arguing that moving the location of the video was both a breach of contract, and interference with her business dealings.

Decisions to be made by YouTube:

• How should a service like YouTube determine which videos are getting legitimate traffic compared to which are generated traffic through artificial means, such as bots?
• Is it possible to distinguish a heavy marketing campaign to point traffic to a video from methods involving artificially generated views?
• In which cases should a video that has received artificial views be moved to a different location (cutting off old links and restarting a counter) as compared to being removed entirely?

Questions and policy implications to consider:

• Will fighting back against artificially inflated views lead to false accusations?
• Could actions designed to stop artificial view inflation impact a legitimate marketing campaign?
• Should musicians and labels rely heavily on things like “views” to determine the actual popularity of an artist when they might be manipulated?

Resolution: After many twists and turns, the lawsuit Darnaa LLC filed against Google was dismissed at both the district court and the appeals court, though much of the dismissal was due to the case being filed after the statute of limitations had passed. However, the court also rejected the parts of the case that survived the statute of limitations questions, noting that YouTube was effectively entitled to manage its service as it saw fit, including how it treated Darnaa’s videos.

Originally posted to the Trust & Safety Foundation website.

Filed Under: content moderation, darnaa, gaming, inauthentic behavior, marketing campaigns, moving videos, viewer counts
Companies: youtube

Content Moderation Case Study: Pretty Much Every Platform Overreacts To Content Removal Stimuli (2015)

from the overblocking-is-the-norm dept

Summary: Content moderation at scale often involves significant tradeoffs between diverse interests. It is often difficult for those without experience in the field to recognize these competing interests.

Social media services aren't just beholden to their users. They're also at the relative mercy of dozens of competing interests at all times.

Users expect one thing. A bunch of governments expect another. Internal policies and guidelines result in another layer of moderation. Then there are the relatively straightforward obligations platforms must fulfill to retain their safe harbors under the DMCA.
So what happens when all of these competing interests collide? Well, according to multiple studies, the most common side effect is over-moderation: the deletion of content that's not in violation of anything, just in case.

For the past half-decade, Stanford Law School's Daphne Keller has been tracking platforms' responses to external stimuli: the pressures applied by outside interests that -- for good or evil -- want social media services to expand their moderation efforts.

And for most of that half-decade, Keller has seen "good faith" efforts expand past the immediate demands to encompass preemptive removal of content that has yet to offend any one of the hundreds of stakeholders applying legal pressure to US-based tech companies.

The research shows large companies are just as preemptively compliant as smaller companies, even though smaller companies have much more at risk.

The easiest, cheapest, and most risk-avoidant path for any technical intermediary is simply to process a removal request and not question its validity. A company that takes an “if in doubt, take it down” approach to requests may simply be a rational economic actor. Small companies without the budget to hire lawyers, or those operating in legal systems with unclear protections, may be particularly likely to take this route.

Multiple studies are cited, and they appear to reach the same conclusion, whether it involves a platform with millions of users or a small group catering to a niche audience: when it doubt, take it out.

Decisions to be made by platforms:

• Should a premium be placed on protecting user content in the face of vague takedown demands?
• Does protecting users from questionable takedown demands result in anything more quantifiable than "goodwill?" 
• Are efforts being made to fight back against mistargeted or unlawful content removal requests? Is the expense/liability exposure too costly to justify defending users against unlawful demands from outside entities?

Questions and policy implications to consider:

• Do platforms ultimately serve their users' interests or the more powerful interests applying pressure from the outside?
• Is staying alive to "fight another day" ultimately of more use to platform users than taking a stand that might result in being permanently shut down? 
• Is it wise to attempt to satisfy all stakeholders in content moderation issues? Should platforms choose a side (users v. outside complainants) or is it wiser to "play the middle" as much as possible?
• Are there fungible advantages to deciding users are more important than outside entities who may have the power to dismantle services specializing in third-party content?

Resolution: The war between users and outside interests continues. As pressure mounts to moderate more and more content, users are often those who feel the squeeze first. The larger the platform, the higher the demands. But larger platforms are more capable of absorbing the costs of compliance. Smaller ecosystems need more protection but are often incapable of obtaining the funds needed to fight legal battles on the behalf of their users.

True balance is impossible to achieve, as this research shows. Unfortunately, it appears preemptive removal of content remains the most cost effective way of satisfying competing moderation demands, even if it ultimately results in some loss to platforms' user bases.

Originally posted to the Trust & Safety Foundation website.

Filed Under: content moderation, daphne keller, governments, overblocking

Content Moderation Case Study: Roblox Tries To Deal With Adult Content On A Platform Used By Many Kids (2020)

from the for-the-children dept

Summary: Roblox is an incredibly popular online platform for games, especially among younger users. In 2020, it was reported that two-thirds of all US kids between 9 and 12 years old use Roblox, and one-third for all Americans under the age of 16. The games on the platform can be developed by anyone, as Roblox has set up a very easy environment, using the scripting language Lua, so that many of the games themselves are developed by Roblox’s young users.

Given the target market of Roblox, the company has put in place a fairly robust content moderation program designed to stop content that the company deems inappropriate. This includes all kinds of profanity and “inappropriate” language, as well as any talk of “dating,” let alone sexual innuendo. The company also does not allow users to share personal identifiable information.

The content moderation extends not just to players on the Roblox platform, but to the many game developers that create and release games on Roblox as well. Roblox apparently uses AI moderation from a company called Community Sift as well as human moderators from iEnergizer. Recent reports say that Roblox has a team of 2,300 content moderators.

Given the competing interests and incentives, there are both widespread reports of adult content being easily available (including to children) as well as developers complaining about having their content, projects, and accounts shut down over perfectly reasonable content, leading to widespread complaints that the moderation system is completely arbitrary.

Roblox is then left trying to figure out how to better deal with such adult content while simultaneously not upsetting its developers, or angering parents who don’t want their children exposed to adult content while playing games.

Decisions to be made by Roblox:

• How do you monitor so much content to make sure that adult content does not get through? How do you make sure that kids are not exposed to adult content?
• If the moderation systems are too aggressive, will that drive developers (and possibly some users) away?
• Should all games go through a human review process before they can be offered through Roblox?
• Are there better ways to communicate how and why content is moderated?

Questions and policy implications to consider:

• Which is a more important constituency: the kids/families using Roblox or the developers who produce content for it? Is being aggressive in content moderation about finding a balance between those two groups?
• Is it worth “overbanning” if it means families feel safer using Roblox?

Resolution: Roblox has continued to evolve and try to improve its content moderation practices. As this case study was being written, the company announced plans to start a content rating system for games, to better inform parents which games may be more appropriate (or inappropriate) for children. However, the company has been promising to improve its efforts to stop adult content from reaching children for many years -- and every few months more reports pour in. At the same time, developers who feel that their content has been blocked for no clear reason continue to take to forums to complain about the lack of clarity and transparency regarding the moderation system.

Originally published on the Trust & Safety Foundation website.

Filed Under: content moderation, kids
Companies: roblox

Content Moderation Case Study: Twitter Suspends Users Who Tweet The Word 'Memphis' (2021)

from the memphis-memphis-memphis-memphis dept

Summary: Twitter users who made the mistake of tweeting out an innocuous word -- 'Memphis -- found themselves suspended from the service for 12 hours for apparently violating the terms of use.

According to messages sent to suspended users, the use of the Tennessee city name violated prohibitions on posting personal information.

The inadvertent damage quickly spread across Twitter as users trolled each other, trying to get unsuspecting accounts to tweet the suddenly-forbidden word. The apparent flaw in the auto-moderation system went unaddressed for several hours as more and more users found themselves temporarily prevented from using the service. Although some users noticed certain accounts (mainly verified ones) weren't being hit with bans, it affected enough users that the ripple effect was not only noticeable, but covered by many mainstream media outlets.

The bans were lifted several hours later with no explanation from Twitter other than that an unspecified "bug" had resulted in tweets containing the word "Memphis" being removed and features limited for those accounts.

That explanation was not entirely clear. Given the "Memphis" bug's link to alleged violations of Twitter's policies against posting other people's personal information, it was speculated the ban on a single city name may have been the result of an erroneously-completed form on the moderation side. Systems security professional SwiftOnSecurity took a plausible stab at the possible root cause of this improbable series of moderation events.

What's possible is a Twitter staffer tried to block a street address, but the postal syntax acted as an escape sequence, or the original was multi-line and they only pasted the city.

If so, every use of the word "Memphis" was considered to be a post containing a full address Twitter had targeted for removal under its personal information policy.

Decisions to be made by Twitter:

• Should moderation of the posting of personal information be handled with automation, given the potential for errors to scale and compound?
• Should a better stop-gap process be put in place to head off future events like these?
• Should users be given better explanations when moderation-at-scale results in features being limited for users affected by moderation bugs? 

Questions and policy implications to consider:

• Should a stop-gap measure be put in place to prevent errors like this from becoming multi-hour failures?
• Have Twitter's moderation efforts resulted in a noticeable limitation of the spread of personal info? 
• Is all publication of personal information considered a violation of policy? Or are exceptions in place for information considered to be of public interest?

Resolution: Twitter restored accounts and tweets targeted by its "Memphis" bug within hours of its emergence. However, the company's moderation team has yet to explain exactly went wrong or what Twitter has done to prevent its recurrence.

Originally posted to the Trust & Safety Foundation website.

Filed Under: blocklist, content moderation, memphis, memphis memphis, memphis memphis memphis
Companies: twitter