Content Moderation At Scale Remains Impossible: Vaccines Edition

from the the-way-of-the-world dept

Last week a story started to blow up that was used, once again, by the media to beat up on Facebook. The headline, from the Daily Beast, says it all: Facebook Axed Pro-Vaccine Ads From Hospitals and Health Orgs, Let Anti-Vaxxer Ads Slip Through. As the story notes, Facebook has (smartly) decided to not allow anti-vax nonsense advertising. It will, of course, allow important pro-vaccination awareness advertising. It does this for a pretty good reason: anti-vax nonsense is killing people. Vaccinations save lives (and I know some anti-vaxxers reading this are foaming at the mouth to scream at us in the comments, and let's just be clear: you're wrong and you should stop it before you kill more people). Anyway, here's what went down:

This month, the Idaho Department of Health and Welfare, the state’s official health department, bought 14 ads to promote a statewide program providing free pediatric vaccinations. Facebook removed all of them.

During the same time period, Children’s Health Defense, an anti-vaccine nonprofit founded and chaired by the nation’s most prominent vaccine conspiracy theorist, Robert F. Kennedy Jr., successfully placed more than 10 ads stoking unfounded fear about vaccines and other medical conspiracy theories.

I saw some people on Twitter using this to attack Facebook, but actually it just highlights the same point we've been making for a few years now: content moderation at scale is impossible to do well and you will always, always make mistakes. And this is one of many kinds of mistakes that happen all the time. Unless someone is deeply, deeply engaged in these issues, distinguishing between anti-vax anti-science quackery can sometimes be difficult. And if moderators are taught to be wary of "vaccine" advertisements, they may just start to key in on anything that mentions vaccines -- including something from a government Department of Health. In some cases it appears that automated systems are to blame:

“It’s our understanding that auto-blocking software flagged these ads, since the text resembles when ads appear to be spreading vaccine misinformation,” said Emily Lowther, a spokeswoman for the Minnesota Hospital Association, who expressed frustration at the phenomenon.

Of course, perhaps what's more interesting is that part of the reason the Daily Beast was even able to write this story is because of Facebook's transparency on advertisements with its Ad Library.

You could say that Facebook must do a better job at this kind of thing, but that would require focusing even more attention on these ads, which inevitably means some other set of ads will end up getting messed up as well. Content moderation at scale is impossible to do well, and that's not a Facebook issue, it's a societal one. There are some people who are going to be pushing bad information, and they're always going to seek to make it look as legit as possible. That's a problem, but expecting that one company can magically fix it seems like a silly thing to do.

Filed Under: ads, anti-vax, content moderation, vaccines
Companies: facebook

No, Internet Companies Do Not Get A 'Free Pass' Thanks To CDA 230

from the it's-just-not-true dept

There are many critics of CDA 230 these days, and there's a pretty wide range in the quality of their arguments. Law professor Danielle Citron is, for good reason, considered one of the more thoughtful critics of the law. And, to her credit, she actually does understand the law, what it enables, and what the wider impacts of the law might be. Her scholarship tends to be thoughtful and careful as well, and, just recently, she was awarded a MacArthur "genius grant." And that's why I find it frustrating that her presentations before Congress recently seem to miss the mark by a fairly wide margin. Earlier this year, we called out her testimony on "deep fakes" because she falsely suggested that internet platforms have "no incentive to address destructive deepfake content."

As we explained at the time, nothing could be further from the truth. The companies have been facing tremendous pressure from the media, the public, politicians, and (importantly) advertisers to clean up junk on their networks, or they risk losing users and revenue. The incentive is the desire not to have their platforms turn into complete garbage dumps.

Unfortunately, Citron is continuing to spread this misleading idea to Congress, as she did last week, when the House held a hearing on Section 230. Citron's opening statement has also been posted to Slate with the title: Tech Companies Get a Free Pass on Moderating Content, and the subtitle: "It's time to change that." Once again the premise is simply false. What is accurate is that platforms do have a legal free pass to decide what level of moderation is appropriate, but that level of appropriateness is very much driven by the concerns of all of the stakeholders mentioned above, with users and advertisers topping the list (for fairly obvious reasons).

It is misleading in the extreme to suggest that a lack of legal incentive somehow means no incentive at all. It is a kind of faith in legal systems (and an ignorance of markets) that is disconnected from reality. Citron's piece does mention market power, but only to brush it away with a blind insistence that it couldn't possibly work:

The market is unlikely to turn this tide. Content that attracts likes, clicks, and shares generates advertising income or a cut of the profits in the case of online firearm marketplaces. Salacious, negative, and novel content is far more likely to attract eyeballs than vanilla, accurate stories. Market pressure is not enough, and it should not have to be.

We need legal reform to ensure that platforms wield their power responsibly.

Again, I do appreciate that Citron -- unlike many other 230 critics -- recognizes the benefits of 230 and that there are inevitable tradeoffs to modifying it. We just disagree on the scope of the downsides to the modifications she suggests. For example:

Another approach would be to adopt the proposal that Benjamin Wittes and I have suggested: to condition the immunity on reasonable content moderation practices rather than the free pass that exists today. If adopted, when the courts consider a motion to dismiss on Section 230 grounds, the question would not be whether a platform acted reasonably with regard to a specific use of the service. For instance, if Grindr is sued for negligently enabling criminal impersonation on its dating app, the legal shield would not depend upon whether the company did the right thing in the plaintiff’s case. Instead, the court would ask whether the provider or user of a service engaged in reasonable content moderation practices writ large with regard to unlawful uses that clearly create serious harm to others. Thus, in the hypothetical case of Grindr, the court would assess whether the dating app had reasonable processes in place to deal with obvious misuses of its service, including criminal impersonation. If Grindr could point to such reasonable practices, like having a functioning reporting system and the ability to ban IP addresses, then the lawsuit should be dismissed even if that system fell short in the plaintiff’s case.

This is the kind of idea that sounds good in theory, but would inevitably be a disaster in practice. A "reasonable" standard is extremely ambiguous until after a whole bunch of expensive case law is established, and would certainly create massive costs, especially for platforms that try to be creative or different in how they approach content moderation. And, then, once the case law is established, it will effectively "lock in" certain approaches, even if they are not the best or don't apply appropriately to other forms of content. If you're a smaller or up-and-coming platform and you want to avoid potentially company-destroying litigation, you are simply going to mimic the models of other companies that have already gone through the litigation gauntlet.

Indeed, such an approach inevitably favors the largest platforms -- the Googles, the Facebooks, etc -- in multiple ways. First off, they can afford to fight the expensive court battles, but that also means that they're the ones to effectively set the standards that the rest of the internet has to use -- and they can set those standards in a manner that only they can afford.

This is not theory. We already have real world examples of this, most obviously in the copyright space. Given all of the litigation around copyright, YouTube spent over $100 million developing ContentID to help it identify potentially infringing material. And now, at least in Europe (and, if industry lobbyists get their way, elsewhere as well), other companies are effectively being told that they need to implement such a filtering solution, with people pointing to Google's setup as evidence that it can be done. Except almost no one else can afford to spend $100 million to build their own system, and while there may be a market for third party services, they tend to be expensive and limiting. At the very least, they shut off all alternative paths to innovation, and lock in the large company's choice.

What's funny is that Citron concludes her testimony by advocating for variety:

There is no one size fits all approach to responsible content moderation. Unlawful activity changes and morphs quickly online, and the strategies for addressing unlawful activity clearly causing serious harm should change as well. A reasonableness standard would adapt and evolve to address those changes.

Except that her own proposal would lead to the exact opposite of that. "Reasonableness" is a vague and unworkable policy that will lead to a one size fits all approach, dominated and controlled by the largest players in the field. It's the wrong approach, driven by the false belief that there is some sort of "free ride" provided by CDA 230, which ignores the reality of public and market pressure, which has shown -- time and time again -- to be quite effective in creating change on these platforms.

Filed Under: cda 230, content moderation, danielle citron, free speech, market power, market pressure, reasonableness, section 230
Companies: facebook, google

Top Myths About Content Moderation

from the so-many-myths-to-debunk dept

How Internet companies decide which user-submitted content to keep and which to remove—a process called “content moderation”—is getting lots of attention lately, for good reason. Under-moderation can lead to major social problems, like foreign agents manipulating our elections. Over-moderation can suppress socially beneficial content, like negative but true reviews by consumers.

Due to these high stakes, regulators across the globe increasingly seek to tell Internet companies how to moderate content. European regulators are requiring Internet services to remove extremist content within an hour and to install upload filters to prospectively block copyright infringement; and U.S. legislators have proposed to ban Internet services from moderating content at all.

Unfortunately, many of these regulatory efforts are predicated on myths about content moderation, such as:

Myth: Content moderation can be done perfectly.

Reality: Regulators routinely assume Internet services can remove all bad content without suppressing any good content. Unfortunately, they can’t. First, mistakes occur when the service lacks key contextual information about the content—such as details about the author’s identity, other online and offline activities, and cultural references. Second, any line-drawing exercise creates mistake-prone border cases because users routinely submit “edgy” content. Third, a high-volume service will make many mistakes, even if it’s highly accurate—1 billion submissions a day at 99.9% accuracy still yields a million mistakes a day.

Myth: Bad content is easy to find and remove.

Reality: Regulators often assume every item of bad content has an impossible-to-miss flashing neon sign saying “REMOVE THIS CONTENT,” but that’s rare. Content is often obviously bad only in hindsight or with context unavailable to the service. Regulators’ cherry-picked anecdotes don’t prove otherwise.

Myth: Technologists just need to “nerd harder.”

Reality: Filtering and artificial intelligence play important roles in content moderation. However, technology alone cannot magically solve the problem. “Edgy” and contextless content vexes the machines, too.

Myth: Internet services should hire more humans to review content.

Reality: Humans have biases and make mistakes too, so adding human reviewers won’t lead to perfection. Furthermore, human reviewers sometimes experience an unrelenting onslaught of horrible content to protect the rest of us.

Myth: Internet companies have no incentive to moderate content.

Reality: In 1996, Congress passed 47 U.S.C. 230, which says Internet services generally aren’t liable for third-party content. Due to this legal protection, critics often assume Internet services won’t invest in content moderation; and some companies have stoked that perception by publicly positioning themselves as “neutral” technology platforms. Yet, virtually every Internet service moderates content, and major services like Facebook and YouTube employ many thousands of content reviewers. Why? The services have their own reputation to manage, and they care about how content can affect their users (e.g., Pinterest combats content that promotes eating disorders). Furthermore, advertisers won’t let their ads appear on bad content, which provides additional financial incentives to moderate.

Myth: Content moderation, if done right, will make everyone happy.

Reality: By definition, content moderation is a zero-sum game. Someone gets their desired outcome, and someone else doesn’t—and those folks won’t be happy with the result.

Myth: There is a one-size-fits-all approach to content moderation.

Reality: Internet services cater to diverse audiences that have different moderation needs. For example, an online crowdsourced encyclopedia like Wikipedia, an open-source software repository like GitHub, and a payment service for content publishers like Patreon all solve different problems for their communities. These services shouldn’t have identical content moderation rules.

Myth: Imposing content moderation requirements will stick it to Google and Facebook.

Reality: Google and Facebook have enough money to handle virtually any requirement imposed by regulators. Startup enterprises do not. Increased content moderation burdens are more likely to block new entrants than to punish Google and Facebook.

Myth: Poor content moderation causes anti-social behavior.

Reality: Poorly executed content moderation can accelerate bad behavior, but often the Internet simply mirrors existing anti-social behavior or tendencies. Better content moderation can’t fix problems that are endemic in the human condition.

Regulators are right to identify content moderation as a critically important topic. However, until regulators overcome these myths, regulatory interventions will cause more problems than they solve.

Reposted from Eric Goldman's Technology & Marketing Law Blog.

Filed Under: cda 230, congress, content moderation, section 230

Elizabeth Warren's Feud With Facebook Over 'False' Ads Just Highlights The Impossibility Of Content Moderation At Scale

from the damned-if-you-do dept

You may have heard over the past few days about a bit of a feud between Presidential candidates -- mainly Elizabeth Warren -- and Facebook about how the company handles political ads with false information. It began a week or so ago when the Trump campaign started running a bunch of Facebook ads around impeachment, some of which were blatantly false, based on totally debunked claims. Facebook, however, just recently clarified its policy, noting that while it will block ads that its partner fact-checkers have determined to be untrue, that does not apply directly to political candidate ads themselves:

We rely on third-party fact-checkers to help reduce the spread of false news and other types of viral misinformation, like memes or manipulated photos and videos. We don’t believe, however, that it’s an appropriate role for us to referee political debates and prevent a politician’s speech from reaching its audience and being subject to public debate and scrutiny. That’s why Facebook exempts politicians from our third-party fact-checking program. We have had this policy on the books for over a year now, posted publicly on our site under our eligibility guidelines. This means that we will not send organic content or ads from politicians to our third-party fact-checking partners for review.

In response to this, Warren decided to hit back by deliberately posting a "false" ad to call out Facebook's policy (in this case, her ad declared that "Mark Zuckerberg and Facebook just endorsed Donald Trump for re-election." The ad then calls out Facebook's policy and attacks the company for its stance. Here's how it read:

Breaking News: Mark Zuckerberg and Facebook just endorsed Donald Trump for re-election.

You're probably shocked, and you might be thinking, "how could this possibly be true?"

Well, it's not. (Sorry.) But what Zuckerberg *has* done is given Donald Trump free rein to lie on his platform -- and then to pay Facebook gobs of money to push out their lies to American voters.

If Trump tries to lie in a TV ad, most networks will refuse to air it. But Facebook just cashes Trump's checks.

Facebook already helped elect Donald Trump once. Now, they're deliberately allowing a candidate to intentionally lie to the American people. It's time to hold Mark Zuckerberg accountable—add your name if you agree.

Except... it turns out that the latter part of Warren's "fake news ad" -- the part which she presents as truthful -- is actually false as well. As Facebook itself noted in a tweet (yes, in a tweet, from Facebook) directed at Warren, Trump is airing TV commercials with the same debunked claims in them on TV stations, and it's one of the most viewed ads:

In other words, the central claim from Warren's campaign -- that other providers, such as TV stations, would not allow such blatant lies in ads -- is pretty clearly untrue. And while some are arguing that Facebook pointing this out "backfired spectacularly", it didn't. It is true that Warren is now claiming that Facebook's argument supports her point, but I don't see that at all.

If anything, it shows the damned if you do/damned if you don't nature of content moderation.

Let's take a second to remember that it was just a few months ago that Warren got really angry at Facebook for temporarily blocking one of her ads. And that time Warren responded by arguing that Facebook should not be able to "shut down a debate" over a political topic. And, yet, in this case, when they won't shut down Trump's posts, suddenly she's mad about that too?

As Julian Sanchez rightly points out, no matter which way Facebook goes, it's going to get yelled at.

And this is the point that lots of us have been trying to make regarding Facebook and content moderation. If you're screaming about all the wrong choices you think it makes to leave stuff up, recognize that you're also going to pretty pissed off when the company also decides to take stuff down that you think should be left up.

Now, there is a consistent way to interpret Warren's position -- which is that in either case, Facebook is "too big" because (as she suggested earlier this year) it can "shut down" the debate on this topic. Except, that's also pretty obviously bullshit, because much of this debate actually happened... on Twitter. And also across lots and lots of media sites.

There are good reasons to be worried about Facebook's market position. There are good reasons to be concerned about the amount of lies that get flung around by political campaigns. But this whole debate over lies in ads is not a Facebook problem. If you want to get at the fact that politicians will lie, well, then you're going to need to take that up with the 1st Amendment. Blaming Facebook for taking the reasonable stance that it doesn't want to be the one directly judging whether or not a politician is being truthful is silly. If it had to review every single political ad for truthfulness, I doubt Elizabeth Warren would be very happy about it either. Hell, part of the problem is that "truth" is not an exact standard either. Some things are misleading, but "technically" true. Some things could count as "little white lies." Some could be blatant falsehoods. But where anything falls is often a matter of opinion, and Warren should actually be happy that Facebook is refraining from making a judgment call on the truthiness of political ads.

Filed Under: advertising, campaign ads, content moderation, content moderation at scale, donald trump, elizabeth warren, truth
Companies: facebook

The DOJ Is Conflating The Content Moderation Debate With The Encryption Debate: Don't Let Them

from the it's-not-the-same dept

As we've detailed a lot over the last week, the DOJ has decided that after years of failing to get backdoors mandated by warning about the "terrorism" bogeyman, it's decided to pick up the FOSTA playbook, and instead start focusing on child porn -- or what "serious people" now refer to as Child Sexual Abuse Material (CSAM). It did this last week with an assist from the NY Times, who published an article with (legitimately) scary stories, but somehow blaming the internet companies... because they actually report it when they find such content on their networks. I've seen more than a few people, even those who generally have been strong voices on the encryption debate and against backdoors, waver a bit on this particular subject, and note that maybe there shouldn't be encryption on social media networks, because it might (as the narrative says) help awful people hide their child porn.

Except... that's confusing a few different things. Mainly, it's mixing up the content moderation debate with the "lawful access" or "backdoors" debate. Yes, encryption makes it harder for the police to get in and see certain things, but that's by design. We live in a country with the 4th Amendment, in which we believe that it should be difficult for law enforcement to snoop deeply into our lives -- and that's always meant that some people will do and plot bad stuff out of the sight and hearing of law enforcement. Yet, if you were to look at law enforcement over the past 100 years, you can bet that they have many times more access to information about people today than they have in the past. The claim of "going dark" is laughable when you compare the information that law enforcement can get today even to what it could get 15 or 30 years ago.

But, importantly, bringing CSAM into the debate muddies the water by pretending -- incorrectly -- that in an end-to-end encrypted world you can't do any content moderation, and there's simply no way for platforms to block or report certain kinds of content. Yet, as Princeton professor Jonathan Mayer highlights in a new paper, content moderation is not impossible in an encrypted system. It may be different than it is today, but it's still very much possible:

Much of the public discussion about content moderation and end-to-end encryption over the past week has appeared to reflect two common technical assumptions:

1. Content moderation is fundamentally incompatible with end-to-end encrypted messaging.
   
2. Enabling content moderation for end-to-end encrypted messaging fundamentally poses the same challenges as enabling law enforcement access to message content.

In a new discussion paper, I provide a technical clarification for each of these points.

1. Forms of content moderation may be compatible with end-to-end encrypted messaging, without compromising important security principles or undermining policy values.
   
2. Enabling content moderation for end-to-end encrypted messaging is a different problem from enabling law enforcement access to message content. The problems involve different technical properties, different spaces of possible designs, and different information security and public policy implications.

You can read the whole thing, but as the paper notes, user reporting of such content still works in an end-to-end encrypted world, as does hash matching if done at the client end. There's a lot more in there as well, but what you realize in reading the paper is that while law enforcement has now latched onto the CSAM issue as its hook to break encryption (in part, I've been told by someone working with the DOJ, because they found it "polled well"), it's an entirely different problem. This is yet another "but think of the children" argument, which ignores the technical and societal realities.

Filed Under: chris wray, content moderation, doj, encryption, going dark, jonathan mayer, william barr
Companies: facebook

China-Based TikTok Actively Banning The Gay Away When Not Helping Authoritarians Be More Authoritarian

from the when-your-core-users-don't-actually-use-the-app dept

TikTok is the new Vine. (I apologize if this sentence is incomprehensible. If I wasn't raising teens, it would be pretty much incomprehensible to me too.) The social media platform that allows users to upload short video and encourages remixing via duets or reaction videos has roped in a billion users in a surprisingly short amount of time.

It's fun for (almost) all ages. The younger you skew, the more popular it is. Fun for all ages, but not all sexualities apparently. The app was created by Chinese company ByteDance, so it's more than receptive than most to calls to censor content based on countries' desire to mute those who aren't heterosexual. (h/t K'Tetch)

The Guardian has been reporting on ByteDance's censorial efforts on behalf of governments around the world. Unsurprisingly, censorship begins at home.

The documents, revealed by the Guardian for the first time, lay out how ByteDance, the Beijing-headquartered technology company that owns TikTok, is advancing Chinese foreign policy aims abroad through the app.

The revelations come amid rising suspicion that discussion of the Hong Kong protests on TikTok is being censored for political reasons: a Washington Post report earlier this month noted that a search on the site for the city-state revealed “barely a hint of unrest in sight”.

The proxy censorship is applied in two ways. ByteDance will delete content it finds unacceptable -- including content it finds unacceptable on behalf of others. It also bans users for repeatedly posting content it later deletes. In other cases, the content stays up, but ByteDance severely reduces its visibility. The end result is a video-sharing platform that contains almost no videos highlighting the protests in Hong Kong.

But it's not just political, even if TikTok's content-burying efforts come at the behest of political parties. The follow-up report from The Guardian notes TikTok has been instrumental in excising plenty of content that fails to denigrate non-hetero relationships and lifestyles.

TikTok’s efforts to provide locally sensitive moderation have resulted in it banning any content that could be seen as positive to gay people or gay rights, down to same-sex couples holding hands, even in countries where homosexuality has never been illegal, the Guardian can reveal.

[...]

[An] entire section of the rules was devoted to censoring depictions of homosexuality. “Intimate activities (holding hands, touching, kissing) between homosexual lovers” were censored, as were “reports of homosexual groups, including news, characters, music, tv show, pictures”. Similarly blocked was content about “protecting rights of homosexuals (parade, slogan, etc.)” and “promotion of homosexuality”. In all those guidelines, TikTok went substantially further than required by law.

TikTok's overzealous content moderation goes further than just muting pro-gay content. It also covers a ton of content various governments find somewhat offensive, but not enough to actually outlaw it. TikTok prevents users in Turkey from mocking the ultra-mockable Recep Erdogan, as well as posting depictions of "non-Islamic gods."

ByteDance calls its content-blocking efforts a "localised approach." But it's really just an extension of something that's all too familiar to citizens of ByteDance's home country: the elevation of governments above citizens, even though its citizens that use the app, not governments.

Weirdly, ByteDance is far more permissive than many social media apps when it comes to questionable content involving children. Sexualized content featuring children tends to avoid moderating efforts -- something TikTok encourages by telling moderators to treat every video subject as over the age of 18 unless they have proof otherwise.

As a private company, TikTok is not a censorship body on its own. It can moderate content however it sees fit without trampling on people's rights. The problem is TikTok has become an unofficial extension of multiple governments by adopting their rules for acceptable content, rather than applying its moderation consistently across the board.

This doesn't necessarily make ByteDance worse than US-based social media platforms. The only difference is its country of origin, where broad censorship is considered normal, rather than an aberration. But US social media companies do the same thing. To avoid being blocked in more authoritarian countries, platforms aid and abet the oppression of citizens by agreeing to follow whatever censorial laws these governments have put in place, even if it means suppressing dissent and making things more dangerous (and less appealing) for their users.

Filed Under: china, content moderation, tiktok
Companies: bytedance, tiktok

Jerks 'Reporting' Women Who Swipe Left On Them In Tinder, Once Again Highlighting How Content Moderation Gets Abused

from the always-another-thing dept

We keep trying to highlight (over and over and over again) how content moderation at scale is impossible to do well for a variety of reasons -- and one big one is the fact that assholes and trolls will game whatever system you put in place -- often in truly absurd ways. The latest example of this is that guys who are pissed off about women who reject them after meeting through Tinder are "reporting" the women in the app, trying to get their accounts shut down.

I had been banned from Tinder. It turns out, though, I’m far from the only woman to have been kicked off the app for no other reason than I rejected the wrong guy. Indeed, without the need for any apparent proof of wrongdoing, a new breed of scorned men have stumbled upon a particularly passive-aggressive way to say, “If I can’t have her, no one can” — tapping the report button.

Case in point: Last year, 33-year-old Amy declined to go out with a man she’d been messaging with when he started insulting her. The insults, of course, only intensified from there — with him telling her she was shaped like Slimer from Ghostbusters and that her fertility was declining. Stunned, she put her phone away. After taking a moment, she went to block him, but when she opened Tinder, her account had been banned.

Of course, as some of the article highlights, Tinder itself seems woefully (ridiculously) unprepared to deal with even the most basic instances of this kind of abuse. Tinder apparently bans accounts based on a single report and the company states that it does "not offer an appeals process at this time." The article highlights a bunch of tweets from women who all seem to have gone through a similar experience. They met a dude on Twitter, date doesn't go well, she says she'd rather not go on another date... the guy flips out, acts like an asshole, and minutes later, she's banned from the app.

The author of the article, who herself was banned from Tinder right after such an experience, found a guy who admitted to doing this.

I did, though, find one man — 26-year-old Brian — who admitted to reporting women who were unresponsive to his messages. “I’ve done this,” he confides. “It’s a huge waste of time for girls to match with you and then not reply. Like what’s the point?”

He then goes on to spout a bunch of misogynistic nonsense, apparently believing that women do this on purpose to be mean to men like him, which apparently is what he needs to convince himself that getting their accounts shut down okay.

Of course, what's left out of this discussion is a bit of the flipside. You can kind of understand why Tinder is so aggressive in banning people, because if people actually are violating its rules, the consequences could be a lot more serious, especially given that the entire point of the app is to get people to meet up in real life. If they mess that up, there will be all sorts of bad press about how Tinder failed to take down an account or something. Hell, as we've detailed, Grindr effectively got sued over this exact scenario and the plaintiff in that case recently asked the Supreme Court to hear his appeal.

That's not to say the companies can't do a better job -- they can. Having an appeals process seems like a no-brainer. But an appeals process can be gamed as well. And this is the point that we keep trying to make: it's literally impossible to do content moderation well at this kind of scale. There will always be problems and judgment calls people disagree with -- and outright abuse. In both directions. People abusing the system to take down content they don't like, and others abusing the system to keep up content or profiles that probably should be taken down. It's easy for someone to say "oh, they shouldn't do that," but no one has yet come up with a system that always gets it right and stops any such abuse. Because it's literally impossible.

Filed Under: appeals, assholes, content moderation, content moderation at scale, dating, reporting, trolls
Companies: tinder

Another Day, Another Major Disinformation Effort Facebook Thinks Is Ok

from the hoodwinked-patriots-dot-come dept

Our disinformation problem simply isn't going away. While the Russian "Internet Research Agency" has received most of the attention for filling the internet with bullshit and bile, the problem is routinely disclosed to be far larger than that. Take for example the thriving "fake news" efforts coming out of Macedonia, where (with the help of US allies) filling the internet and Facebook with disinformation has become a cottage industry. And while Facebook spends a lot of time insisting they're taking radical steps to police the problem in the wake of genocide in Myanmar, it remains fairly clear they still don't have a handle on the problem.

For example yet another disinformation operation uncovered this week involves a Ukrainian-run "I Love America" Facebook group with more than a million members. In concert with other similar pages like "God bless Donald and Melania Trump and God bless America," the effort lures boomers in with cute kittens and patriotic memes before getting to the real meat and potatoes of the effort:

Many of the posts are just repurposed versions previously pushed by the IRA in a bid to try and stoke existing racial and political tensions in the United States:

And while the usual suspects will likely try to downplay this as "just a few harmless memes," the outfit's engagement surpasses many of the biggest, actual news organizations on Facebook, meaning there's still an awful lot of folks having their patriotism and military respect exploited and their heads filled with fluff and nonsense so some Ukrainian nitwits can make a buck:

Collectively this effort is far larger than the IRA-linked Facebook groups highlighted in the Mueller report, none of which had more than 390,000 members. The report doesn't think this effort is covert or sophisticated enough to be a government-backed effort (in large part because the Ukrainian backers aren't trying to hide who they are), and is likely just some "entrepreneurs" using pro-Trump propaganda and kittens to make money. Facebook, for its part, doesn't think this rises to the level of "coordinated inauthentic behavior" because the Ukranians aren't hiding their identity (read: it's profitable to Facebook):

"A Facebook spokesperson told Popular Information that the company does not believe any of the Facebook pages discussed in this article violate its policies, including the policy against "coordinated inauthentic behavior." Facebook defines "coordinated inauthentic behavior" as "when groups of pages or people work together to mislead others about who they are or what they are doing."

While it may not be part of a foreign-government backed campaign, the end result is the same. Facebook users are being exploited and having their heads filled with rocks for profit. And (at least until this story gets some traction), Facebook's cool with it.

Update: That didn't take long, and Facebook appears to have seen the problem with a bunch of Ukrainians misleading US Facebook users to make a quick buck:

Filed Under: content moderation, coordinated inauthentic behavior, disinformation, elections, fake news
Companies: facebook

Content Moderation At Scale Especially Doesn't Work When You Hide All The Rules

from the getting-twitchy dept

For quite a while now, we've pointed out that doing any serious content moderation on major internet sites is laughably difficult, if not impossible. Whether done in a purely automated format, or with real human oversight, everything ends up boiling down to just how much collateral damage are we all willing to accept when sites attempt to enforce moderation rules. Even when sites attempt to communicate the rules to the public in a somewhat transparent fashion, such as Facebook, it all inevitably goes to the kind of hell that includes nixing accounts for sharing what is purely art.

But when sites don't bother to tell their users what the rules are, even after exacting punishment for violation of those rules? Well, that's when you have a bunch of Twitch streamers wondering what the hell is going on.

Earlier this week, Twitch suspended a streamer named Quqco for wearing a cosplay of Street Fighter heroine Chun-Li on stream, deeming her outfit “sexually suggestive.” This took Quqco—and many others—by surprise, given that the outfit was not overtly risqué. But this was not an isolated incident. In the past few days, Twitch has been cracking down on so-called “sexually suggestive” content more aggressively than usual, and picking some questionable targets for its crusade.

The whole post is long and very detailed and should absolutely be read in full. The article lists numerous examples of Twitch streamers being handed out warnings and suspensions under any number of circumstances that range from cosplaying as Street Fighter characters, to wearing a sports bra and baggy shorts on a stream, to drawing clothed cartoon characters. All those interviewed for the post that were on the receiving end of these warnings or suspensions were completely flummoxed.

Especially when their own inquiries into their punishment were being responded to like this.

For example, Fareeha’s warning specifically accused her of wearing “underwear or lingerie,” when she was, in reality, wearing baggy gym clothes in a setting where you’d expect to see them. Twitch’s guidelines around what streamers can and cannot wear are vague and contextual; the sort of attire Fareeha was wearing might not have cut it if a streamer was broadcasting from their bedroom, but streamers regularly wear gym clothes in the gym. Some men even go shirtless. It’s not clear why Twitch singled out Fareeha.

Fareeha specifically believes she may be the victim of another aspect of the internet that makes content moderation at scale impossible: trolls. It turns out that there are dedicated Discord channels out there where people get off on doing mass-reports of Twitch streamers they don't like for any number of reasons. Obviously, those many numbers of reasons often fall into categories that include "they're women doing things we don't like" and "they're people that don't look like us doing things we don't like." The theory goes that whatever automation Twitch has built into its moderation system -- and there surely must be something of that sort of thing -- it's likely to be sensitive to mass reporting of ToS violations. This is being gamed by trolls to get the system to punish Twitch streamers who otherwise would never have been punished.

Is that what's happening? Nobody knows, mostly because Twitch is being frustratingly opaque on the matter.

Saruei, who was suspended for drawing “nude” characters, declined to speak to Kotaku out of concern that she could face further repercussions from Twitch. However, prior to her recent suspension, she spoke out against what she feels is “hypocrisy” on Twitch’s part. Suggestive poses apparently aren’t allowed in her drawings, she said, but it’s fine when some people do them IRL.

“These are suggestive poses, right?” she said of her own art while discussing her Twitch warning during a recent (now-unavailable) stream. “We agree with that, right? Why I can’t draw waifus like this when there is fucking Twitch girls that can do it?” She went on to express frustration about the lack of clarity that she, like others, has had to deal with. “I hope it won’t happen again, because I asked them ‘What is the problem with these drawings?’” she said. “Is it the clothing or the pose? What is against Twitch guidelines? I need to know.”

Those last two sentences are all you need to hear to recognize the massive problem that is Twitch attempting to do content moderation at a scale that is largely impossible while not being transparent as to how to stay on the right side of that moderation. There is entirely too much gray area, too much room for gaming the moderation, and too much stupid collateral damage to make any of this worth it.

Meanwhile, the Twitch streamers who make Twitch's product attractive are stuck trying to figure out what the rules are. Until they decide to go somewhere else, that is.

Filed Under: content moderation, content moderation at scale, quqco, rules, transparency
Companies: twitch