Mother's Lawsuit Attempts To Hold Snapchat, Instagram Responsible For Her Daughter's Suicide

from the tragic,-but-not-actionable dept

In the wake of a tragedy, it's human nature to seek some form of justice or closure. The feeling is that someone should be held accountable for a senseless death, even when there's no one to blame directly. This tends to result in misguided lawsuits, like the multiple suits filed by (far too opportunistic) law firms that seek to hold social media platforms accountable for the actions of mass shooters and terrorists.

The desire to respond with litigation remains even when there's a single victim -- one who has taken their own life. That's the case here in this lawsuit, coming to us via Courthouse News Service. Plaintiff Tammy Rodriguez's eleven-year-old daughter committed suicide. Her daughter was allegedly a heavy user of both Snapchat and Instagram. The connection between the platforms and her daughter's suicide is alluded to and alleged, but nothing in the lawsuit [PDF] shows how either of the companies are directly responsible for the suicide.

Here's how the complaint hopes to achieve these questionable ends:

Defendants have designed Instagram and Snapchat to allow minor users to use, become addicted to, and abuse their products without the consent of the users’ parents, like Tammy Rodriguez.

Defendants have specifically designed Instagram and Snapchat to be attractive nuisances to underage users but failed to exercise ordinary care owed to underage business invitees to prevent the rampant solicitation of underage girls by anonymous older users who do not disclose their real identities, and mass message underage users with the goal of grooming and sexually exploiting minors.

Defendants not only failed to warn Tammy and Selena Rodriguez of the dangers of addiction, sleep deprivation, and problematic use of their applications, but misrepresented the safety, utility, and addictive properties of their products. For example, the head of Instagram falsely testified under oath at a December 8, 2021 Senate Committee hearing that Instagram does not addict its users.

As a result of Selena Rodriguez’s addictive and problematic use of Instagram and Snapchat, she developed numerous mental health conditions including multiple inpatient psychiatric admissions, an eating disorder, self-harm, and physically and mentally abusive behaviors toward her mother and siblings.

As a proximate result of her addiction to Instagram and Snapchat, Selena Rodriguez committed suicide on July 21, 2021. She was eleven years old at the time.

There's the first problem with the lawsuit: "proximate result." While it's possible to show an indirect connection between a person's act and the actions of the services they use, you really can't argue "proximate cause" while also arguing "strict product liability." Either this suicide was the direct result of the design flaws or warning failures or it wasn't. It can't be both strict and proximate.

Inside those claims are further problems. What's stated in the product liability arguments is sometimes directly contradicted by the narrative of the lawsuit. For instance, under the claims about violations of California's Unfair Competition Law, the plaintiff says this:

Defendants engaged in fraudulent and deceptive business practices in violation of the UCL by promoting products to underage users, including Selena Rodriguez, while concealing critical information regarding the addictive nature and risk of harm these products pose. Defendants knew and should have known that their statements and omissions regarding the addictive and harmful nature of their products were misleading and therefore likely to deceive the members of the public who use Defendants’ products and who permit their underage children to use Defendants’ products. Had Plaintiff known of the dangerous nature of Defendants’ products, she would have taken early and aggressive steps to stop or limit her daughter’s use of Defendants’ products.

But the plaintiff did know, as is clearly stated earlier in the lawsuit:

Plaintiff Tammy Rodriguez, Selena’s mother, attempted multiple times to reduce or limit her daughter’s use of social media, which caused a severe reaction by Selena due to her addiction to Defendants’ products. Because Defendants’ products do not permit parental controls, the only way for Tammy Rodriguez to effectively limit access to Defendants’ products would be to physically confiscate Selena’s internet-enabled devices, which simply caused Selena to run away in order to access her social media accounts on other devices.

Plaintiff Tammy Rodriguez attempted to get Selena mental health treatment on multiple occasions. An outpatient therapist who evaluated Selena remarked that she had never seen a patient as addicted to social media as Selena. In the months leading up to Selena’s suicide, she was experiencing severe sleep deprivation that was caused and aggravated by her addiction to Instagram and Snapchat, and the constant 24-hour stream of notifications and alerts Defendants sent to Selena Rodriguez.

So, the plaintiff was aware -- not only by what she had personally observed but by what she had been told by her daughter's therapist. That not only undercuts the arguments made in the state law claims, but the very next paragraph from the narrative of the lawsuit.

Throughout the period of Selena’s use of social media, Tammy Rodriguez was unaware of the clinically addictive and mentally harmful effects of Instagram and Snapchat.

Now, I'm not saying any of this should excuse the slot machine feedback loops that permeate so many social media services. But it's a huge stretch to say social media platforms are directly (or proximately) responsible for violent acts by users, whether they're mass shootings or singular suicides. This lawsuit seems to admit there's no direct link (even though it definitely doesn't want to) by undercutting its own claims with factual assertions about the last several months of this child's life.

While it's true the average person may not understand the personal and psychological aspects of social media addiction, most people are familiar with the symptoms of addiction and, if they're a concerned parent like the one filing this suit, act accordingly. The narrative in this lawsuit attempts to show how dangerous these platforms are, but by doing so, show the plaintiff was well aware of the negative side effects. Just because that knowledge failed to prevent a tragedy does not automatically transfer full responsibility to the internet services her daughter used.

A lawsuit like this may prompt the creation of better parental controls or age verification procedures, but it's unlikely to result in Instagram and Snapchat being ordered to compensate this mother for the tragic death of her child. The connections are too tenuous and the allegations too conclusory to survive a motion to dismiss. The lawsuit's internal contradictions aren't going to help. And, despite some concerning comments from the Ninth Circuit of the supposed overbreadth of Section 230 immunity in recent months, this suicide was ultimately the act of someone who used these services, rather than an act encouraged or abetted by the services being sued.

Filed Under: intermediary liability, section 230, selena rodriguez, suicide, tammy rodriguez
Companies: instagram, snapchat

Devin Nunes, CEO Of Trump's TRUTH Social, Confirms That 'Free Speech' Social Media Will Be HEAVILY Moderated

from the thank-you-section-230 dept

It's never been a secret that for all of the public claims about how Donald Trump's upcoming social network "TRUTH Social" will be for "free speech" that this was never actually the plan. We noted right up front that its terms of service appeared to be way more restrictive than all the competitors it was criticizing -- and even said it would be a violation of terms to "annoy" anyone working for the site. When Rep. Devin Nunes -- who has a long history of suing people for criticizing and mocking him (i.e., no friend of free speech) -- announced he was retiring from Congress to become CEO of Trump's social network, we noted that he'd be quick to ban people on the site.

And, so it's no surprise that Fox Business is reporting that the site will have "strict" content moderation tools in place, in order to make the site "family friendly."

Fox Business has learned that TMTG is partnering with Hive—a San Francisco-based Series D start-up that provides automated solutions through cloud-based artificial intelligence for understanding images, videos, and text content. Hive’s technology provides automated content moderation across video, image, text and audio.

"We want to be very family-friendly, we want this to be a very safe place, and we are focused on making sure any illegal content is not on the site," Nunes said in an exclusive interview with Fox Business.

"Hive has a great track record in this, and they have been good to work with," Nunes continued. "They are very helpful for our team and because of their experience, I think they’re helping to craft the right spot for us."

He added: "We want to be the most family-friendly site."

Hive is, indeed, pretty widely used by various social media platforms struggling to keep craziness in check -- including Parler and Chatroulette.

But, it's pretty incredible to see Nunes saying this without any self-awareness at all. Considering how much he has criticized Twitter, YouTube and other platforms for their moderation practices -- which were also very much based on the belief of making their platforms "family friendly" for users -- it's pretty ridiculous to pretend that TRUTH Social is somehow different.

Also, the reason that TRUTH Social can work with Hive and make these decisions about how to moderate their platform in a "family friendly" manner is because of Section 230 giving them the freedom to do so without liability -- the very law that Trump himself wanted repealed.

Of course, the real irony is that along with that article on Fox Business is a video clip of Maria Bartiromo, whining to GETTR CEO Jason Miller, about how terrible existing social media companies are because they "censor" content, and talking about how great it is that TRUTH Social will be different. She quotes Trump talking about this "new age of censorship." There seems to be no recognition at all that TRUTH Social will moderate just like every other site does -- and, in fact, this video is over an article saying exactly that. Even more ridiculous: the example that Baritromo uses of "censorship" from the big existing platforms... was not censorship at all. She posted a photo of a sunrise to Instagram, and listed upcoming guests as well as "topics" that they would be talking about, including "vaccines."

It appears this triggered an automated context system on Instagram that when someone tried to share the image, Instagram popped up a note that said "Make sure the information is reliable before sharing" and added a link to a CDC website with information about vaccines. That is not even remotely "censorship." Indeed, it's exactly the opposite. It is the quintessential "more information" -- the very thing that Baritromo, quoting Trump, seemed to suggest we need more of. But because it's about her photo, she claims its "censorship."

Perhaps just as laughable, in that interview, Miller claims "fact is, we're living in the worst period of political censorship in American history." That's absolute hogwash. The opposite is objectively true. Until recently the vast majority of Americans had nowhere to speak publicly about politics. Now they all have places they can speak. It's just that not everyone gets to speak on everyone else's property.

The whole thing is crazy, and it would be nice if Baritromo, Trump, Miller, and Nunes just out and out admitted the reality: they just don't like it when their own nonsense is moderated, and they want separate rules for themselves than for anyone else.

Filed Under: ai, content moderation, devin nunes, donald trump, family friendly, free speech, maria bartiromo, section 230
Companies: fox news, gettr, hive, instagram, tmtg, truth social, twitter

Oversight Board Overturning Instagram Takedown Of Ayahuasca Post Demonstrates The Impossibility Of Content Moderation

from the this-is-not-easy dept

Congress has been holding lots of "but think of the children online!" hearings over the past couple of months, and one prominent topic that comes up over and over again is the fact that people can find "drug" information online. Fears about kids and drugs goes back decades, but politicians love it, because it always works. And, of course, the media loves to run these overhyped stories. A quick search finds dozens of stories like the following in just the last month or so:

• Teens have easier access to drugs as illegal trade booms on social media
• Instagram pushes drug content to teens
• Social media platforms becoming sites of illegal fentanyl drug sales targeting teens, L.A. officials warn
• WARNING: Dealers tempt kids with edible drugs via social media
• Record number of drug overdose deaths brings new scrutiny to social media apps
• Instagram offers 'drug pipeline' to kids, tech advocacy group claims

There are many more such stories, but you get the idea. The Instagram stories, in particular, were targeted by the recent Senate hearing which was focused on allowing Senators to grandstand on how bad Instagram supposedly is for kids. Instagram responded to the report by pointing out that it's doing a ton of stuff to try to block such content:

"We removed 1.8 million pieces of content related to drug sales in the last quarter alone, and due to our improving detection technology, the prevalence of such content is about 0.05 percent of content viewed, or about 5 views per every 10,000. We'll continue to improve in this area in our ongoing efforts to keep Instagram safe, particularly for our youngest community members."

But, within a day of that report getting so much attention, you had the Oversight Board step in to demonstrate how much more difficult an issue this is than it sounds to grandstanding politicians or narrative-driving journalists. The Oversight Board overturned Instagram's decision to take down a post that was promoting ayahuasca as medicine. Instagram took down the post (which had just started trending) because it was promoting it as medical treatment:

In July 2021, an Instagram account for a spiritual school based in Brazil posted a picture of a dark brown liquid in a jar and two bottles, described as ayahuasca in the accompanying text in Portuguese. Ayahuasca is a plant-based brew with psychoactive properties that has deeply-rooted religious and ceremonial uses among Indigenous and other groups in some South American countries, and related communities elsewhere. Ayahuasca contains plants which are sources of dimethyltryptamine (DMT), a substance that is prohibited under Schedule I of the 1971 UN Convention on Psychotropic Substances and the law of many countries, though there are relevant exceptions under international law for substances containing DMT such as ayahuasca, and exceptions under some national laws, including in Brazil, for other uses such as religious and Indigenous use.

The text states that “AYAHUASCA IS FOR THOSE WHO HAVE THE COURAGE TO FACE THEMSELVES” and includes statements that ayahuasca is for those who want to “correct themselves,” “enlighten,” “overcome fear” and “break free.” It further states ayahuasca is a “remedy” and “can help you” if one has humility and respect. It ends with “Ayahuasca, Ayahuasca!/Gratitude, Queen of the Forest!”

The content was viewed over 15,500 times and no users reported it. The post was flagged for review by Meta’s automated systems because it had received around 4,000 views and was “trending.” Meta specified neither the image nor the text triggered automatic review. The post was subsequently reviewed by a human moderator and removed. Meta told the Board that it was removed for violating Facebook’s Community Standard on Regulated Goods, but later stated it removed the content for “violating the Instagram Community Guidelines, which include a link to the Facebook Community Standard on Regulated Goods.” Meta notified the user that the post went against Instagram’s Community Guidelines, stating “post removed for sale of illegal or regulated goods.” The messaging also noted that Meta removes “posts promoting the use of hard drugs.”

So, there are a few interesting tidbits there worth pointing out. Contrary to the fear-mongering and grandstanding, this notes that Instagram has a system to proactively flag this kind of content, and that some of the priority of such reviews is based on whether or not the content is getting a lot of attention and going viral (these are both things that many critics of Facebook/Instagram/Meta insist the company does not do). And, if you just listened to the Senators at hearings, or just read all those news articles above, you might not realize th.

But, the more important thing that this case showcases, is just how difficult it is to make these kinds of calls at scale. What seems obvious to some is not so obvious, and there's a lot of subjectivity involved. This can be seen in the way the two sides (the poster of the original content and the company) explained their thinking. The organization that made the original post about ayahuasca says that it was just an image of a regulated and legal ceremony:

The user stated in their appeal that they are certain the post does not violate Instagram’s Community Guidelines, as their page is informative and never encouraged or recommended the purchase or sale of any product prohibited by the Community Guidelines. They said that they took the photo at one of their ceremonies, which are regulated and legal. According to the user, the account aims to demystify the sacred ayahuasca drink. They said that there is a great lack of knowledge about ayahuasca. The user stated that it brings spiritual comfort to people and their ceremonies can improve societal wellbeing. They further state that they have posted the same content previously on their account and that post remains online.

Instagram, for its part, takes the more hardline "drugs bad, mmkay" line that Congress and the press seems to want them to take:

Meta stated that the content violated Facebook’s Community Standards because “the user described ayahuasca with a heart emoji, referred to it as ‘medicine,’ and stated that it ‘can help you.’” Following a question from the Board about whether the content was removed for violating Instagram’s Community Guidelines or Facebook’s Community Standards, Meta responded the content was removed “for violating the Instagram Community Guidelines, which include a link to the Facebook Community Standards on Regulated Goods.” Specifically, the user violated Instagram’s prohibition on content “buying or selling illegal or prescription drugs (even if legal in your region).” Meta also cited another line of the Community Guidelines which links to the Community Standard on Regulated Goods, which “clarifies that Facebook prohibits content that ‘[c]oordinates or promotes (by which we mean speaks positively about [...] non-medical drugs.’”

Referring to Meta’s values, the company stated that “Safety” displaced “Voice.” Meta noted that users are permitted to advocate for the legalization of non-medical drugs and to discuss the medical and scientific benefits of non-medical drugs, but that there is no religious or traditional use allowance. Meta argued that this rule strikes the correct balance between “Voice” and “Safety.”

Meta also stated that prohibiting this content follows human rights principles. It stated that it considered the right to freedom of expression under Article 19 ICCPR and the right to freedom of religion or belief under Article 18 ICCPR, and argued that its decision satisfied the conditions required to restrict these rights.

According to Meta, Facebook’s Community Standard is easily accessible and its non-public definition of non-medical drug as a “substance that causes ‘a marked change in consciousness’” is apparent. It further argued that the decision sought to protect public health. It stated that dimethyltryptamine (DMT), one of the active hallucinogenic substances in ayahuasca, poses significant safety risks, citing the 1971 UN Convention on Psychotropic Substances.

The final ruling by the Oversight Board overturn's Instagram's decision to take it down, citing that Instagram's own community guidelines were somewhat confusing and focused on the buying and selling of illegal drugs (though it notes that the rules are confusing, because they said "illegal or prescription drugs (even if legal in your region)" which is, as the Board notes "confusing and contradictory." There was also some level of conflict between Instagram's rules and Facebook's rules, and an unclear delineation of which one should apply where.

But where the impossibility of content moderation on topics like this really shines through is in the discussion about "voice" vs. "safety," which was one of the key points that Instagram made in defending its decision. The Board says it values things differently than the company:

The Board concludes that Meta’s decision to remove the content was not consistent with the company’s values. In this case, as in many, Meta's values of “Voice,” “Safety,” and “Dignity” point in different directions. Meta's decision to take down the post weighted “Safety” over “Voice.” The Board would balance the values differently, believing that the genuine but not particularly strong interests in “Safety” are outweighed in this context by the value of “Voice” and the importance of recognizing the “Dignity” of those engaging in traditional or religious uses where there is historic evidence of such use, including by Indigenous and religious communities. Scientific research indicates that the use of ayahuasca in a controlled context in traditional and religious ceremonies is not linked to a serious risk of harm. Meta cited the European Court of Human Rights case of Franklin-Beentjes and Ceflu-Luz da Floresta v. The Netherlands (Case No. 28167/07, European Court of Human Rights, May 6, 2014) to demonstrate the risks of ayahuasca use. Other courts, however, have reached different conclusions – for example, the United States Supreme Court, in considering the risk of harm from “the circumscribed, sacramental use of hoasca” by members of an ayahuasca based religion, found that the government had not put forward sufficient evidence of harm from religious use, which was its burden, to justify the prohibition in these circumstances. Meta’s rationale does not appear to have taken into account controlled uses of ayahuasca which aim to mitigate health risks. In light of scientific research, Meta’s rationale did not demonstrate the danger of this post to the value of “Safety” in a manner sufficient to displace “Voice” and “Dignity” to the extent to justify removal of the post.

There should be no way to look at that paragraph and not realize just how subjective all of this is. Every one of these involves judgment calls and people weigh different values in very different ways. Even people who work at the same company. Figuring out how to (1) write rules that take all this into account, (2) express those rules in AI moderation systems, (3) express those rules clearly to tens of thousands of human moderators across hundreds of languages and cultures, (4) and expect it to work at a scale of billions of users and millions of posts per day... is literally impossible. It cannot be done well.

That doesn't mean that everyone should give up, but there should at least be some level of humility in recognizing the level of this task, and highlighting individual decisions that we disagree with is not evidence that people or the company "don't care" about this issue, but rather evidence of just how impossible a task this truly is.

Filed Under: ayahuasca, brazil, content moderation, drugs, health misinformation
Companies: facebook, instagram, meta, oversight board

The Senate's Finsta Problem

from the that's-now-how-it-works dept

Earlier this year we wrote about Senator Richard Blumenthal's viral "finsta" debacle in which he asked a Facebook executive to "end Finsta" as if (1) it was a product, or (2) it was "endable." Some people pushed back on the mocking of Blumenthal, noting that he (or at least his staffers) actually understood what "finsta" meant, since he had given an accurate description earlier, and his staff had set up a fake Instagram account, pretending to be a young girl to see what happened. However, as we and others noted, if he actually understood "finsta" that made his demand to "end Finsta" even worse, because it meant he was calling for an end to anonymity on Instagram, which would have huge problems -- especially for more marginalized people or those at risk, who have very good reasons for not using their real names on social media.

Of course, rather than learning from that debacle, Senators across the board seemed to see it as an opportunity to score some media headlines when they dragged in Instagram CEO Adam Mosseri to testify in yet another one of the Senate's pointless, grandstanding "big tech bad" hearings last week. Senator Blumenthal brought back his finsta. On the other side of the aisle, both Senators Marsha Blackburn and Mike Lee rolled out their own finstas. And all of them were shocked, shocked, shocked, to find that sometimes they didn't like what the algorithm recommended to them.

Except, as an article at the National Journal highlights, this little game of gotcha doesn't make much sense and is not how good research is actually done.

Joshua Tucker, a co-director of the NYU Center for Social Media and Politics, said Senate staff don’t have the time or training to make broad inferences about how teens experience Instagram.

“In a sense these [fake accounts] are useful, because they are surfacing things that can happen on the platform,” Tucker said. “But it’s this leap from ‘it can happen’ to ‘this is prevalent, this is common’—that’s when you need the more sophisticated research design.”

Tucker called it “ironic” that lawmakers are deploying these fake accounts in hearings, given ongoing concerns from Blumenthal and others that “finstas” are a menace and the fact that creating such accounts violates Instagram’s terms of service.

“These are exactly the types of protections that we are asking Congress to provide to researchers—to protect researchers who want to do research, who want to go about and find out how these platforms work, to protect them from being attacked by companies for violating terms of service,” Tucker said.

But, rather than handing access to actual researchers who can determine whether or not there's a problem -- and if there is a problem how big or how prevalent it is -- Senators are instead pushing out garbage science. That is, they're pushing anecdotal examples based on a single case set up in a way specifically designed to try to get the outcome they want.

Not surprisingly, when journalist Brendan Bordelon pointed all this out to Blumenthal's office, it was dismissed in Blumenthal's typically dismissive manner:

The Connecticut Democrat shrugged off the idea that there’s something ironic about his creation of a fake account, given his concerns about “finstas” generally.

“I think it’s justified to be illustrative and then disclosed, and nothing done with it,” Blumenthal said. “It’s not like we set it up to exist for any length of time. We set it up to see what happened, and then we shut it down.

“If I were analogizing it to law enforcement, it’s a little bit like having somebody work undercover,” he said.

It's a misleading anecdote, and for someone who claims to be so concerned about misinformation, you'd think that Blumenthal would be careful not to create more of it.

And... that's not even getting into the point that all of these Senators broke Instagram's terms of service, and according to some interpretations of the law, that means they violated the CFAA. It seems like it would be a hell of a lot more productive to fix that, rather than freaking out about the other stuff.

Filed Under: anonymity, finsta, kids, marsha blackburn, mike lee, richard blumenthal, terms of service
Companies: instagram

CNN Goes Full Moral Panic About Kids And Social Media

from the have-we-no-balance? dept

CNN, the news organization that, until recently, employed Chris Cuomo, and still employs Jeffrey Toobin, and is (for the moment at least) owned by AT&T which funded an entire extremist propaganda TV network just to appease President Trump (not to mention being absolutely terrible on privacy issues), wants you to hate social media. There may be reasons to hate on social media, but it's difficult to take CNN seriously when it presents itself (1) as some unbiased party in this discussion, and (2) puts forth an article that is nothing more than blatant moral panic propaganda about kids and social media.

Are there dangers to kids on social media? Maybe! Are there benefits for kids on social media? Maybe! Does the article only present one side full of anecdotes without any actual data? You bet. The article presents a couple of anecdotes about teens with depression, and then just insists that it's because of social media. Apparently it may surprise CNN's reporters to learn this, but teenagers (and adults) have been dealing with depression for a long, long time, including before social media existed. Again, it's entirely possible that social media creates image problems for teens, but the article repeatedly just insists its true without evidence. It opens with a pure anecdote that is designed to pull at the emotional heart strings.

Last September, just a few weeks into the school year, Sabine Polak got a call from the guidance counselor. Her 14-year-old daughter was struggling with depression and had contemplated suicide.

"I was completely floored," said Polak, 45, who lives in Valley Forge, Pennsylvania. "I had no clue she was even feeling remotely down at all. When I asked her about it, she just kept saying she wanted to get away from it all ... but I didn't know what that meant."

After taking her to a crisis center, which banned phone use for anyone checking in, Polak learned from her daughter that the pressures of social media were driving her increased anxiety. The main source of stress: waiting for her friends to open and respond to messages and photos on Snapchat.

Okay. But people had anxiety pre-internet as well. It's easy to blame it on these services, but what actual evidence is there? The article presents literally none.

And the only attempts at pointing to "evidence"... doesn't say what it seems to think it says. There are just some of the documents Frances Haugen revealed, which showed a study that Facebook/Instagram conducted showing that some teens self-reported that Instagram makes them feel worse about themselves. But what nearly all of the reporting left out -- and CNN of course ignores -- is that same study showed a much larger percentage said that Instagram made them feel better about themselves.

That doesn't mean that anyone should discount those who are depressed, or those who claim that Instagram and social media make them feel worse about themselves. Because anything that can be done to help should be explored. But presenting a totally one-sided article that suggests social media is all bad for kids and dangerous ignores all the people (again, a much larger percentage) who found that social media helps them.

It seems most likely that much of it depends on existing personality types and how people use social media. If you use it to make connections and maintain friendships, it can be super helpful. If you use it as a barometer of how much better other people's lives are, it's easy to see how it can be damaging. But how do you allow one and prevent the other?

Unfortunately, the CNN piece explores exactly none of this nuance. Instead, it just runs the party line: "social media = bad for teens." And in doing so it ignores all the many teens who are able to connect with people, or find their communities, or explore interests that their local schoolmates might not have, all thanks to the internet's ability to bring them together.

Any serious look at the challenges and negative impacts of social media has to take into account the flipside as well, and come up with some sort of plan to balance the two, and explore how do you help encourage more of the good kind of connectivity, and minimize the problematic kinds. But that's not what CNN gives you. If you read the CNN article you'd naturally assume that all social media is bad for all teens, and the only solution is to take the phone away from your kid.

Later in the article, there's a second bit of "evidence," but it comes from Bark, a company that sells parental snooping software and has a huge economic incentive to scare parents into believing their kids are all being lead astray by social media. Again, a thoughtful exploration of the challenges of teens and social media might look for a more credible source, but that's not what CNN's reporters were doing. They had a narrative, and they needed to support it, so this is what they got.

Some data also support that mental health issues among young people on social media are on the rise. Bark, a paid monitoring service that screens social media apps, personal messages and emails for terms and phrases that could indicate concerns, said it saw a 143% increase in alerts sent around self-harm and suicidal ideation during the first three months of 2021 compared to the year prior. (Parents receive alerts when Bark detects potential issues, along with expert recommendations from child psychologists for how to address them.)

"Our children's lives are buried deep within their phones and the problems live within their digital signal in places that parents don't go," said Titania Jordan, chief marketing officer of Bark. "If you're not spending time in the places where your children are online, how can you be educated and then how can you give them guidance?"

Yeah, quote the person trying to sell subscriptions to the service, because that's reliable and credible.

And the article does basically nothing in exploring the challenges of what you might actually do about this if it were true. There are suggestions of taking away these services from kids, which again kind of ignores that many more people find these services helpful and useful than find them harmful. Or there's the suggestion -- I kid you not -- of forcing kids to watch The Social Dilemma.

Polak, the mother whose daughter had suicidal thoughts, has proposed a Mental Health Awareness Week at her daughter's school that would include screenings of Childhood 2.0 and The Social Dilemma -- two documentaries that touch on how platforms are impacting the well-being of its users.

As we've explained, that documentary is almost entirely misinformation about how social media applications work. So the "solution" to the bad things that social media does to kids (unsupported by any facts) is to... share with them a propaganda film full of disinformation? Genius.

In 20 years or so, we're going to look back on these moral panics and laugh, just as we did with moral panics in the past. Rock and roll, Dungeons & Dragons, TV, radio, pinball, etc., all promised to rot our brains. This time around it's social media.

Filed Under: anxiety, depression, evidence, moral panic, social media, teenagers
Companies: cnn, instagram

Instagram Founder's Instagram Locked When One Person Convinced Instagram He Had Died

from the i-feel-happyyyyy dept

We have talked a long, long time about how the concept of content moderation at the kind of scale of the largest internet and social media platforms is essentially impossible. But it's not just content moderation that is proving difficult for those platforms. Policing those platforms for anything that relies on user-based input is difficult as well. For instance, Instagram recently found out that its process for locking up the accounts of the deceased may need some work, as one person was able to get Instagram founder Adam Mosseri's Instagram account locked.

Mosseri’s Instagram account was locked because of Instagram’s memorialization feature, where users can report to the company that an Instagram account holder has died. In response, Instagram will block anyone from logging into the account, and stop the ability to make any changes to already uploaded content.

“I find it ridiculous how Instagram lets such things even happen on their platform in the first place,” the scammer who claimed responsibility for the lock on Mosseri’s account, and who used the handle Syenrai, told Motherboard in an online chat. “The entire banning community needs to be discovered and reported to Instagram so they can put an end to this—it’s basically the dark side of Instagram.”

So, what is that memorialization feature? Well, an email goes into Instagram's team to trigger the account change. Instagram then requests proof that the person in question is now deceased. That admittedly morbid request takes the form of a request for either a death certificate or obituary. While that seems reasonable, it should be obvious where the flaw in this system is.

“I have a method which is as simple as finding an online obituary of a person who recently passed away,” they explained. “I then submit a memorialize request for the victim’s account using the random obituary I found, and it takes 1-2 days for support to process.”

“As long as the obituary is recent (within same week) the target will be memorialized,” Syenrai said. “It works 98% of the time.”

While, sure, it's fairly funny to see Instagram lock up and memorialize the account for its own boss, who is very much alive, this all becomes less funny when you understand how this goes for the average, every day user of Instagram. It's also worth noting that this is one of many vulnerabilities that nefarious actors use to harass targets on Instagram.

As for the memorialization process specifically, Instagram's comments have all been canned-style responses indicating that big platforms occasionally have tools like this abused and that Instagram has hires "investigators and cybersecurity specialists" to keep nefarious actors at bay. But it's obviously not working. That Mosseri's account was locked down pretty much proves that.

If the head of Instagram can fall victim to this, what hope is there for the rest of Instagram's usership?

Filed Under: adam mosseri, content moderation, memorial
Companies: instagram

Facebook Banning & Threatening People For Making Facebook Better Is Everything That's Wrong With Facebook

from the such-nonsense dept

Regular readers know that I'm a believer in trying to get the big internet companies to embrace a more protocols over platforms approach, in which they're building something that others can then build on as well, and improve in their own ways (without fear of having the rug pulled out from under them). It's why I'm hopeful about Twitter working on just such a plan with its Bluesky project. Facebook, unfortunately, takes a very different view of the world.

While I understand that some of Facebook's thinking around this is a reaction to what happened when it had created a more open platform for developers... and thenCambirdge Analytica happened, which has been an ongoing (if somewhat confusingly understood) black eye for the company. But Facebook has always been a bit skittish about how open it has wanted to be. Famously, it killed Power.com with an unfortunate reading of the CFAA when that company tried to create a universal login for various social media sites, and to help people not be locked in to just one social media site.

But the latest example is really horrible. Louis Barclay has a write up in Slate about how Facebook banned him for life and threatened him with a lawsuit, because he created a tool to make everyone's Facebook experience better (though, less profitable for Facebook). The tool actually sounds quite nifty:

The tool I created, a browser extension called Unfollow Everything, allowed users to delete their News Feed by unfollowing their friends, groups, and pages. The News Feed, as users of Facebook know, is that never-ending page that greets you when you log in. It’s the central hub of Facebook. It’s also a major source of revenue.

Note that the tool doesn't unfriend your friends, family and groups, it just unfollows them. To be honest, until reading about this, I didn't quite realize there was a difference. The key is that if you unfollow (but keep them as friends, or stay in the groups) you can still see what's happening, it's just that their content doesn't show up in your newsfeed. And if you "unfollow everything" then your newsfeed ends up blank. You can still poke around and see what your friends are posting or what's happening in the groups you're interested in, but it's a proactive decision by you, rather than being pushed to you in the news feed. That's... kinda neat.

I still remember the feeling of unfollowing everything for the first time. It was near-miraculous. I had lost nothing, since I could still see my favorite friends and groups by going to them directly. But I had gained a staggering amount of control. I was no longer tempted to scroll down an infinite feed of content. The time I spent on Facebook decreased dramatically. Overnight, my Facebook addiction became manageable.

When I unfollowed everything for the first time, I did it manually. I spent hours using a Facebook-provided feature to click unfollow on each of my friends, groups, and pages. I quickly realized that very few people would go to the same trouble, so I coded a simple tool that would automate the process. In July 2020, I published it to the Chrome store, where people could download it for free.

As I said... kinda neat.

But, of course, Facebook didn't feel that way. It freaked out.

Then, a few months ago, Facebook sent me a cease-and-desist letter. The company demanded that I take down the tool. It also told me that it had permanently disabled my Facebook account—an account that I’d had for more than 15 years, and that was my primary way of staying in touch with family and friends around the world. Pointing to a provision in its terms of service that purports to bind even former users of Facebook, Facebook also demanded that I never again create a tool that interacts with Facebook or its many other services in any way.

That's fucked up on multiple levels. First off, Louis wasn't doing anything that users couldn't all do themselves. He was just automating it and making it easier. So how could that possibly violate anything? Furthermore, he was simply utilizing the features Facebook gives everyone. How could that possibly deserve this kind of reaction?

Yes, obviously, Facebook doesn't like the idea of people no longer using the news feed, but it seems likely that it's only a very small percentage of folks actually doing this kind of thing. Suck it up and deal with it. In the long run, it could actually be better if it helps people like Louis, who are concerned about how much they use Facebook, to keep using it rather than going away from Facebook (well, not Louis any more, since he's banned).

Second, it's always a bad look when you threaten and punish people who are simply trying to build on your service to make it better for some users. It reminds me when Craigslist sued a company for also making Craigslist better. No matter how you spin it, it's a bad look.

Third, giving him a lifetime ban? Come on. That's just being punitive for the sake of being punitive. It seems clear that Louis likes Facebook in general and the ability to connect with his friends and family -- he just didn't like the news feed. Giving him a lifetime ban just seems ridiculously excessive, and insanely petty. Yes, I know that Facebook is constantly on the lookout for scams and hacks that might impact site integrity, but this tool is not that.

Fourth, telling him that he can "never again create a tool that interacts with Facebook"? That's just beyond overkill. It's also unnecessarily punitive, and also seems like an incredibly broad reading of the company's terms of service.

In a more ideal world, we'd see companies like Facebook encouraging more app builders to create tools that make Facebook's experience better for everyone -- and allowing differentiated services like this. Some people would feel better off without the news feed. Louis was creating a tool to make it easier for people to do that so they don't have to do it manually. He didn't break any thing. He didn't harm the site's integrity. And Facebook flipped out.

As I was thinking about this story, I kept thinking about Block Party, a very cool app I've talked about in the past, which makes it easier for people on Twitter to deal with abuse and harassment, by having it automute certain types of users (there's a lot more to it). In that case, it seems like Twitter is happy to have Block Party create tools that make their users' lives better, even though it might lead to less interaction on Twitter itself. But the end result is, hopefully, a healthier experience. Facebook could do the same and could encourage developers to build better tools for users to create more differentiated experiences.

Instead, with its single-minded focus on revenue and growth, it punishes the guy who was simply making use of Facebook's own features to make lives better.

Oh, yeah, there's one more element to this story which also might explain Facebook's reaction -- not that it makes Facebook look any better. It makes it look worse.

A few months after I published Unfollow Everything, academics at the University of Neuchâtel, in Switzerland, expressed interest in using it to study the News Feed’s impact on the amount of time spent on Facebook and the happiness of the platform’s users. We began working together. The university recruited people to join two study groups: one where participants deleted their News Feeds using Unfollow Everything, and a control group where participants left their feeds intact. Participants agreed to share limited and anonymous information—specifically, the amount of time they spent on Facebook, the number of times they visited the site, and the number of friends, groups, and pages they were following and not following, both in total and broken down by category. (For regular Unfollow Everything users, the only Facebook-related data shared was the ratio of followed profiles to total profiles, a metric that helped me ensure the tool was working.)

Facebook has a really, really contentious relationship with academics trying to study the platform. You'll remember how it recently shut down the accounts of NYU researchers for building a tool to get access to useful data Facebook refuses to share. And, of course, the recent whistleblower revelations show that the company isn't always keen to see research that makes the company's products look bad get out into the world.

If the reason that Facebook freaked out about Unfollow Everything was the fact that it might allow academics to study the impact of the news feed, then that's even more messed up. Let them study your damn service, Facebook.

Filed Under: browser extension, extensions, louis barclay, platforms, threats, unfollow everything
Companies: facebook, instagram

Blumenthal's Finsta Debacle: It Remains Unacceptable That Our Politicians Are So Clueless About The Internet

from the this-is-just-embarrassing dept

Fifteen years ago, the best example of how out of touch elected officials were regarding the internet was Senator Ted Stevens' infamous "it's a series of tubes" speech (which started out "I just the other day got, an internet was sent by my staff at 10 o'clock in the morning on Friday and I just got it yesterday.") Over the years, this unwillingness of those who put themselves in the position to regulate the internet to actually bother to understand it has become something of an unfortunate running joke. A decade ago, in the midst of the fight over SOPA/PIPA, we pointed out that it's no longer okay for Congress to not know how the internet works. And yet, a decade has passed and things have not gotten much better. Senator Ron Johnson tried to compare the internet to a bridge into a small creek. Senator Orrin Hatch has no clue how Facebook makes money.

And now there's a new addition to the list of examples of totally clueless Senators seeking to regulate something they clearly don't understand. This time it's Senator Richard Blumenthal, who has been grandstanding about how he wants to take on the internet since long before he was elected to the Senate. He created the most cringe-worthy media clip of a politician in a while while trying to press Facebook's head of safety Antigone Davis during a Senate hearing on "grandstanding about how we all hate Facebook" (not the actual subject matter, but close enough).

I heard about the exchange before I saw it, but watching the whole thing is even worse than just hearing that Blumenthal seemed to think "finsta" is a Facebook "product or service." If you're somehow as ignorant as Blumenthal, "finsta" is just a slang term for a "fake" Instagram account, where someone doesn't want to be posting under their real name. This can be used to have a more focused conversation, to discuss things you don't want to discuss under your real name, to hide from some people, or for plenty of other reasons. But Blumenthal seemed to think this "product or service" was bad and Facebook should stop it.

Blumenthal (giving his stern politician face): Will you commit to ending Finsta.

Davis (long pause and a very confused look): Senator, let me again explain... we don't actually "do" finsta. What "finsta" refers to is young people setting up accounts where they want to have more privacy. You refer to it as having privacy from their parents, but in my interaction with teens, what I found is they sometimes they like to have an account where they can interact just with a smaller group of friends...

This is a nice diplomatic response to a monumentally confused question. Some people might take that answer and realize that maybe they didn't quite understand what they were asking. But not Senator Blumenthal. Senator Blumenthal doubled down, interrupting Davis:

Blumenthal: Finsta is one of your products or services! [smug face] We're not talking about Google or Apple. It's Facebook! Correct?

Davis (another long pause and another very confused look): F-F-Finsta is slang for a type of account. It's not...

Blumenthal (interrupting again): Okay, will you end that type of account?

I mean, what? How the hell does Blumenthal think that Facebook can magically make people's alias accounts go away? Hell, earlier in the very same hearing... Blumenthal more or less admitted that he had created a finsta account for himself to examine Instagram's friend recommendation engine. And, obviously, there are plenty of good questions to be asked about the recommendation engine and how it works and what it promotes. But the "will you end alias accounts" line of questioning is just bizarrely disconnected from reality.

Even if we were to give Blumenthal the benefit of the doubt, and believe he is honestly concerned that people are able to sign up for accounts under non-real names, that also suggests a serious misunderstanding of a bunch of things about the internet. Facebook actually has long had a "real names" policy, and academic research going back a decade has shown just how damaging that is, especially for marginalized groups and those at risk for various reasons. Forcing everyone to use their real names damages privacy, can lead to stalking, can harm those who wish to explore more controversial topics, and a variety of other things. Automatically assuming that forcing everyone to have a profile under their real name is the best solution is its own kind of ignorance regarding the internet and internet culture. EFF's Jillian York rightly refers to this demand as "the White Man's Gambit," in that it is most often suggested as a solution by white men who don't realize how their privileged position blinds to them to the many, many reasons why people might want to use a "finsta" account for their own privacy and safety.

And this is obviously not the first time Blumenthal has made it clear that he has no interest in understanding the ins-and-outs or details of the internet he so desperately wants to regulate. He was the lead Democratic sponsor of FOSTA and took glee in the idea that the law might cause small internet companies to go out of business. He's also behind the EARN IT Act, which would be a disaster for the internet and encryption, but has hilariously also gotten mad that Zoom's encryption wasn't fully end-to-end even though his own bill would undermine the ability of anyone to offer end-to-end encryption.

So, no, Senator Blumenthal does not get the benefit of the doubt regarding his knowledge of the internet. He has a long history of being confused about how it all works and the impacts of his policy proposals. And maybe, just maybe, before rushing in to sternly judge these internet companies, he could take the time to learn what a "finsta" is.

Filed Under: antigone davis, congress, finsta, ignorance, internet policy, real names, richard blumenthal
Companies: facebook, instagram

Court To Sheriff: Sending An Officer To Tell A Teen To Delete Instagram Posts Is So Very Obviously A Rights Violation

from the [tapping-top-of-house-visit]-you-can-fit-so-many-rights-violations-in-here dept

Wisconsin is apparently America's Karen.

Public schools in Wisconsin referred students to police twice as often as schools nationwide in 2017-18 — nine students were referred to police for every 1,000 students enrolled compared to the national rate of 4.5, a Center for Public Integrity analysis of U.S. Department of Education data found.

Bring cops into schools and schools start thinking every disciplinary problem is a criminal problem. Far too many police departments lending officers to schools agree. Things may have improved over the last couple of years, but this is the latest data from Wisconsin's public schools and it's a pretty grim picture of school administrators abdicating their responsibilities and allowing law enforcement to pick up the slack.

Last year, a Wisconsin student sued the local sheriff and one of his officers for turning an Instagram post of hers into a criminal matter. The student, Amyiah Cohoon, had returned from a spring break trip with her school band and been admitted to a hospital with COVID-like symptoms. Her Instagram photos of her in the hospital were accompanied by statements that she had the coronavirus and wouldn't be back to school anytime soon.

It turned out Cohoon did not have COVID. School administrators decided Cohoon was trying to provoke a panic by falsely claiming she did. They sent out a statement basically calling Cohoon a liar. Then they sent the local sheriff's department after her. This is from her lawsuit:

During the evening on March 27, Defendant Patrol Sergeant Cameron Klump from the Marquette County Sheriff’s office came to the Cohoons’ home. Amyiah answered the door, and Sergeant Klump said he needed to speak with her father.

After Mr. Cohoon came outside, Sergeant Klump explained that the school “superintendent” had complained to Defendant Sheriff Joseph Konrath about one of Amyiah’s Instagram posts. Sergeant Klump showed Mr. Cohoon a screenshot of Amyiah’s third Instagram post. A true and accurate copy of the cropped screenshot Sergeant Klump showed Mr. Cohoon is attached hereto as Exhibit 5.

Sergeant Klump stated that he had direct orders from Sheriff Konrath to demand that Amyiah delete this post, and, if she did not, to cite Amyiah and/or her parents for disorderly conduct and to “start taking people to jail.”

Faced with possibly having her parents arrested, the student deleted her posts. Then she sued. And the court has now told Sheriff Joseph Konrath what he already likely knew: threatening her (and her parents) with arrest over Instagram posts was a clear violation of her First Amendment rights. (h/t Gabriel Malor)

The opening of the ruling [PDF] makes it clear where this is headed:

The SARS-CoV-2 virus and COVID-19 have had a tremendous impact on American society. But, as this case makes clear, that impact has its limits and, more specifically, does not extend to overriding the protections of the First Amendment.

And the Sheriff was limited to making arguments about why this wasn't a rights violations, rather than quibbling about contradictory representations of the incident that forms the basis for this lawsuit. There's no argument to be had here because the whole thing was caught on tape.

The parties agree that the facts are not disputed. Indeed, the entirety of Sergeant Klump’s encounter with Amyiah and her parents is captured in a dash-cam video. That video, along with other record evidence, establishes that Defendants violated Amyiah’s free-speech rights by demanding that she take down her social media posts or face criminal citation or arrest.

Continuing in the spirit of things that cannot be questioned, the court says there's not even the slightest shade of gray in this series of events. This was absolutely a violation of rights.

Even if short and often grammatically scurrilous, social media posts do not fall outside the ambit of the First Amendment. To the contrary, they are exactly what the First Amendment seeks to protect.

The best defense the Sheriff offered was an inadvertent punchline -- one that demonstrates the person raising the argument understands neither the First Amendment nor the precedent they're citing incorrectly.

But Defendants disagree. In their view, Amyiah forfeited her constitutional protection when she published a post that caused concern in the community and led to an influx of phone calls to the Westfield School District and Marquette County Health Department. According to Sheriff Konrath, this was akin to “screaming fire in a crowded movie theater.” Even setting aside that the popular movie theater analogy actually referred to “falsely shouting fire in a theater and causing a panic,” Schenck v. United States, 249 U.S. 47, 52 (1919) (emphasis added), Defendants’ argument still fails.

Even if the government has an interest in protecting the public and heading off potential panics, its interests do not automatically override the rights guaranteed to citizens. The balancing test doesn't even need to be deployed here because the government's actions were so clearly in the wrong.

Labeling censorship societally beneficial does not render it lawful. If it did, nearly all censorship would evade First Amendment scrutiny. [...] Amyiah’s post is not captured by any of the categorical exceptions to the First Amendment, so this Court will not balance the social utility of curtailing it against its government-assigned value.

The correct response in situations like these is more speech. It certainly isn't the dispatching of officers to a teen's house to deliver threats of arrest.

The government here had every opportunity to counter Amyiah’s speech, but it opted instead to engage in the objectionable practice of censorship.

Every argument raised by the government is immediately shut down. Wrong all the way down.

The Marquette County Sheriff had no more ability to silence Amyiah’s posts than it would to silence the many talking heads on cable news, who routinely pronounce one-sided hot takes on the issues of the day, purposefully ignoring any inconvenient facts that might disrupt their preferred narratives. Indeed, even if Amyiah’s posts had been untruthful, no court has ever suggested that noncommercial false speech is exempt from First Amendment scrutiny.

Furthermore, the demand was accompanied by threats to Cohoon and her parents. Even if she wasn't present for the entire conversation, she inferred the intent of the visit correctly. She deleted the posts while the deputy was still speaking to her father and she later deleted another post discussing her hospital stay, fearing officers would return again if that one wasn't removed. And yet the Sheriff claimed this was all voluntary because no officer ever ordered her to remove the posts. Wrong, oh so very wrong again, says the court:

Defendants ask the Court to lump Sergeant Klump’s efforts into the “attempts to convince” basket. Amyiah agreed to delete her Instagram post prior to learning of Sergeant Klump’s threats. How then, Defendants ask, can she possibly claim coercion? This argument ignores the inherently chilling and coercive nature of a uniformed police officer showing up at a teenager’s home and demanding that she cease otherwise protected speech.

Sergeant Klump’s dash-cam footage shows that it was not his persuasive rhetoric that led Amyiah to delete her social media post, but rather his demand made under the auspices of the Sheriff’s Department: “[W]e need to get it taken down.” That was coercion by any metric. The state cannot dispatch a law enforcement officer to the home of a teenager to demand that she remove an Instagram post that government officials disagree with and then claim the officials were only engaging in the Socratic method. It is possible that a Westfield administrator or Marquette County Health Department employee could have engaged in a mutually-respectful discussion with Amyiah to try to convince her to retract her post voluntarily, but that is not the method they chose. They elected, instead, to rely on the coercive power of the Sheriff’s Department, and any attempt to obfuscate that fact by casting Sergeant Klump as an earnest public relations expert must fail.

Imagine your tax dollars being spent to pay the government and its lawyers to argue this badly against your own interests. This is money being burnt by people too obtuse and too shortsighted to notice they've set something on fire. Every defensive effort in this suit is a last-gasp effort.

Here's the court dismantling the defense that the Sheriff's office truly believed a law was being broken, thus excusing every subsequent rights violation.

Defendants’ probable-cause argument dramatically understates the probable-cause analysis for disorderly conduct. If accepted, Defendants’ position would largely gut the First Amendment’s protection for free speech, allowing police officers a free hand to wrongfully arrest anyone engaging in protected speech so long as the offending officer could point to a possible disturbance or perceived anxiety among those who opposed the speech. Accordingly, the Wisconsin Supreme Court has held that speech that “falls within the protection of the First Amendment . . . may not be punished as disorderly conduct.” In re Douglas D., 2001 WI 47, ¶47, 243 Wis. 2d 204, 239-40, 626 N.W.2d 725, 743. Defendants offer no answer to this precedent, which removes any basis for probable cause.

The government loses. It was a loss its representatives -- both from the school and from the Sheriff's office -- should have seen coming. None of this needed to happen. But no one with any power had the self-awareness to be the adult in the room nor the overall awareness that sending law enforcement out with demands to remove social media posts was exactly the sort of thing the First Amendment was erected to guard against.

Filed Under: amyiah cohoon, cameron klump, covid, free speech, marquette county sheriff, police, wisconsin
Companies: instagram

Content Moderation At Scale Is Impossible To Do Well: Series About Antisemitism Removed By Instagram For Being Antisemetic

from the hate-vs.-reporting-on-hate dept

I've written a lot about the impossibility of doing content moderation well at scale, and there are lots of reasons for that. But one of the most common is the difficulty both AI and human beings have in distinguishing hateful/trollish/harassing behavior from those reporting on that behavior. We've pointed this out over and over again in a variety of contexts. One classic example is social media websites pulling down human rights activists highlighting war crimes by saying it's "terrorist content." Another were the many examples of people on social media talking about racism and how they're victims of racist attacks having their accounts and posts shut down over claims of racism.

And now we have another similar example. A new video series about antisemitism posted its trailer to Instagram... where it was removed for violating community guidelines.

You can see the video on YouTube, and it's not difficult to figure out how this happened. The message from Instagram says it violates that organization's community guidelines against "violence or dangerous organizations." The video in question, all about antisemitism, does include some Nazi imagery, obviously to make the point that in its extreme form, antisemitism can lead to the murder of Jews. But, Instagram has banned all Nazi content, in part due to those who complained about antisemitism on Instagram.

And that leads to a dilemma. If you're banning Nazi content, you also have to realize how that might lead to content about Nazis (to criticize them and to warn about what they might do) also getting banned. And, again, this isn't new. Earlier this year we had a case study on how YouTube's similar ban took down historical and educational videos about the Holocaust.

The point here is that there is no easy answer. You can say that it should be obvious to anyone reviewing this that trailer (highlighting how bad antisemitism is) is different from actual antisemitism, but it's a lot harder in practice at massive scale. First you need people who actually understand the difference, and you have to be able to write rules that can go out to thousands of moderators in a simple enough manner that explicitly makes clear the differences. And, you also need to give reviewers enough time to actually understand the context, which is kind of impossible given the scale of the content that needs to be reviewed. In such situations the "simpler" versions of the rules often are what get written: "No Nazi content." That's clear and scalable, but leads to these kinds of "mistakes."

Filed Under: antisemitism, community standards, content moderation, hate speech, nazi content, scale
Companies: facebook, instagram