Oversight Board Overturning Instagram Takedown Of Ayahuasca Post Demonstrates The Impossibility Of Content Moderation

from the this-is-not-easy dept

Congress has been holding lots of "but think of the children online!" hearings over the past couple of months, and one prominent topic that comes up over and over again is the fact that people can find "drug" information online. Fears about kids and drugs goes back decades, but politicians love it, because it always works. And, of course, the media loves to run these overhyped stories. A quick search finds dozens of stories like the following in just the last month or so:

• Teens have easier access to drugs as illegal trade booms on social media
• Instagram pushes drug content to teens
• Social media platforms becoming sites of illegal fentanyl drug sales targeting teens, L.A. officials warn
• WARNING: Dealers tempt kids with edible drugs via social media
• Record number of drug overdose deaths brings new scrutiny to social media apps
• Instagram offers 'drug pipeline' to kids, tech advocacy group claims

There are many more such stories, but you get the idea. The Instagram stories, in particular, were targeted by the recent Senate hearing which was focused on allowing Senators to grandstand on how bad Instagram supposedly is for kids. Instagram responded to the report by pointing out that it's doing a ton of stuff to try to block such content:

"We removed 1.8 million pieces of content related to drug sales in the last quarter alone, and due to our improving detection technology, the prevalence of such content is about 0.05 percent of content viewed, or about 5 views per every 10,000. We'll continue to improve in this area in our ongoing efforts to keep Instagram safe, particularly for our youngest community members."

But, within a day of that report getting so much attention, you had the Oversight Board step in to demonstrate how much more difficult an issue this is than it sounds to grandstanding politicians or narrative-driving journalists. The Oversight Board overturned Instagram's decision to take down a post that was promoting ayahuasca as medicine. Instagram took down the post (which had just started trending) because it was promoting it as medical treatment:

In July 2021, an Instagram account for a spiritual school based in Brazil posted a picture of a dark brown liquid in a jar and two bottles, described as ayahuasca in the accompanying text in Portuguese. Ayahuasca is a plant-based brew with psychoactive properties that has deeply-rooted religious and ceremonial uses among Indigenous and other groups in some South American countries, and related communities elsewhere. Ayahuasca contains plants which are sources of dimethyltryptamine (DMT), a substance that is prohibited under Schedule I of the 1971 UN Convention on Psychotropic Substances and the law of many countries, though there are relevant exceptions under international law for substances containing DMT such as ayahuasca, and exceptions under some national laws, including in Brazil, for other uses such as religious and Indigenous use.

The text states that “AYAHUASCA IS FOR THOSE WHO HAVE THE COURAGE TO FACE THEMSELVES” and includes statements that ayahuasca is for those who want to “correct themselves,” “enlighten,” “overcome fear” and “break free.” It further states ayahuasca is a “remedy” and “can help you” if one has humility and respect. It ends with “Ayahuasca, Ayahuasca!/Gratitude, Queen of the Forest!”

The content was viewed over 15,500 times and no users reported it. The post was flagged for review by Meta’s automated systems because it had received around 4,000 views and was “trending.” Meta specified neither the image nor the text triggered automatic review. The post was subsequently reviewed by a human moderator and removed. Meta told the Board that it was removed for violating Facebook’s Community Standard on Regulated Goods, but later stated it removed the content for “violating the Instagram Community Guidelines, which include a link to the Facebook Community Standard on Regulated Goods.” Meta notified the user that the post went against Instagram’s Community Guidelines, stating “post removed for sale of illegal or regulated goods.” The messaging also noted that Meta removes “posts promoting the use of hard drugs.”

So, there are a few interesting tidbits there worth pointing out. Contrary to the fear-mongering and grandstanding, this notes that Instagram has a system to proactively flag this kind of content, and that some of the priority of such reviews is based on whether or not the content is getting a lot of attention and going viral (these are both things that many critics of Facebook/Instagram/Meta insist the company does not do). And, if you just listened to the Senators at hearings, or just read all those news articles above, you might not realize th.

But, the more important thing that this case showcases, is just how difficult it is to make these kinds of calls at scale. What seems obvious to some is not so obvious, and there's a lot of subjectivity involved. This can be seen in the way the two sides (the poster of the original content and the company) explained their thinking. The organization that made the original post about ayahuasca says that it was just an image of a regulated and legal ceremony:

The user stated in their appeal that they are certain the post does not violate Instagram’s Community Guidelines, as their page is informative and never encouraged or recommended the purchase or sale of any product prohibited by the Community Guidelines. They said that they took the photo at one of their ceremonies, which are regulated and legal. According to the user, the account aims to demystify the sacred ayahuasca drink. They said that there is a great lack of knowledge about ayahuasca. The user stated that it brings spiritual comfort to people and their ceremonies can improve societal wellbeing. They further state that they have posted the same content previously on their account and that post remains online.

Instagram, for its part, takes the more hardline "drugs bad, mmkay" line that Congress and the press seems to want them to take:

Meta stated that the content violated Facebook’s Community Standards because “the user described ayahuasca with a heart emoji, referred to it as ‘medicine,’ and stated that it ‘can help you.’” Following a question from the Board about whether the content was removed for violating Instagram’s Community Guidelines or Facebook’s Community Standards, Meta responded the content was removed “for violating the Instagram Community Guidelines, which include a link to the Facebook Community Standards on Regulated Goods.” Specifically, the user violated Instagram’s prohibition on content “buying or selling illegal or prescription drugs (even if legal in your region).” Meta also cited another line of the Community Guidelines which links to the Community Standard on Regulated Goods, which “clarifies that Facebook prohibits content that ‘[c]oordinates or promotes (by which we mean speaks positively about [...] non-medical drugs.’”

Referring to Meta’s values, the company stated that “Safety” displaced “Voice.” Meta noted that users are permitted to advocate for the legalization of non-medical drugs and to discuss the medical and scientific benefits of non-medical drugs, but that there is no religious or traditional use allowance. Meta argued that this rule strikes the correct balance between “Voice” and “Safety.”

Meta also stated that prohibiting this content follows human rights principles. It stated that it considered the right to freedom of expression under Article 19 ICCPR and the right to freedom of religion or belief under Article 18 ICCPR, and argued that its decision satisfied the conditions required to restrict these rights.

According to Meta, Facebook’s Community Standard is easily accessible and its non-public definition of non-medical drug as a “substance that causes ‘a marked change in consciousness’” is apparent. It further argued that the decision sought to protect public health. It stated that dimethyltryptamine (DMT), one of the active hallucinogenic substances in ayahuasca, poses significant safety risks, citing the 1971 UN Convention on Psychotropic Substances.

The final ruling by the Oversight Board overturn's Instagram's decision to take it down, citing that Instagram's own community guidelines were somewhat confusing and focused on the buying and selling of illegal drugs (though it notes that the rules are confusing, because they said "illegal or prescription drugs (even if legal in your region)" which is, as the Board notes "confusing and contradictory." There was also some level of conflict between Instagram's rules and Facebook's rules, and an unclear delineation of which one should apply where.

But where the impossibility of content moderation on topics like this really shines through is in the discussion about "voice" vs. "safety," which was one of the key points that Instagram made in defending its decision. The Board says it values things differently than the company:

The Board concludes that Meta’s decision to remove the content was not consistent with the company’s values. In this case, as in many, Meta's values of “Voice,” “Safety,” and “Dignity” point in different directions. Meta's decision to take down the post weighted “Safety” over “Voice.” The Board would balance the values differently, believing that the genuine but not particularly strong interests in “Safety” are outweighed in this context by the value of “Voice” and the importance of recognizing the “Dignity” of those engaging in traditional or religious uses where there is historic evidence of such use, including by Indigenous and religious communities. Scientific research indicates that the use of ayahuasca in a controlled context in traditional and religious ceremonies is not linked to a serious risk of harm. Meta cited the European Court of Human Rights case of Franklin-Beentjes and Ceflu-Luz da Floresta v. The Netherlands (Case No. 28167/07, European Court of Human Rights, May 6, 2014) to demonstrate the risks of ayahuasca use. Other courts, however, have reached different conclusions – for example, the United States Supreme Court, in considering the risk of harm from “the circumscribed, sacramental use of hoasca” by members of an ayahuasca based religion, found that the government had not put forward sufficient evidence of harm from religious use, which was its burden, to justify the prohibition in these circumstances. Meta’s rationale does not appear to have taken into account controlled uses of ayahuasca which aim to mitigate health risks. In light of scientific research, Meta’s rationale did not demonstrate the danger of this post to the value of “Safety” in a manner sufficient to displace “Voice” and “Dignity” to the extent to justify removal of the post.

There should be no way to look at that paragraph and not realize just how subjective all of this is. Every one of these involves judgment calls and people weigh different values in very different ways. Even people who work at the same company. Figuring out how to (1) write rules that take all this into account, (2) express those rules in AI moderation systems, (3) express those rules clearly to tens of thousands of human moderators across hundreds of languages and cultures, (4) and expect it to work at a scale of billions of users and millions of posts per day... is literally impossible. It cannot be done well.

That doesn't mean that everyone should give up, but there should at least be some level of humility in recognizing the level of this task, and highlighting individual decisions that we disagree with is not evidence that people or the company "don't care" about this issue, but rather evidence of just how impossible a task this truly is.

Filed Under: ayahuasca, brazil, content moderation, drugs, health misinformation
Companies: facebook, instagram, meta, oversight board

Facebook Promises To Distinguish Takedowns From Governments; Whether They're For Illegal Content, Or Merely Site Rules Violations

from the that's-a-start dept

There's an interesting discussion that happens in content moderation circles with regards to government requests for takedowns: are those requests about content that violates local laws or about the content violating website policies? Many people lump these two things together, but they're actually pretty different in practice. Obviously, if a government comes across content that violates the law, it seems reasonable for them to alert the platform to it and expect that the content will be removed (though, there may be some questions about jurisdiction and such). However, when it's just content that may violate site policy, there are some pretty big questions raised. This actually gets to the "jawboning" question we've been discussing a lot lately, and exploring where the line is between a politician persuading a website to take something down and compelling them to do so.

There is one argument that suggests that a government pointing out content that might violate site policies is simply helping out. The website doesn't want content that violates its standards, and the government reporting it is just like any other user reporting it (though, backed up by whatever credibility the government may or may not have). But, the flip side of that is, if the content is perfectly legal, this often starts to feel like a loophole through which government actors can engage in wink-wink-nudge-nudge censorship -- just send a notification to the site that this particular content may not break the law, but hey, doesn't it violate your policies?

A recent decision by the Oversight Board, and a corresponding statement from Facebook, suggests that Facebook is going to be clearer about when this situation happens. The case the Oversight Board reviewed was an interesting one. Here's how it summarized the situation:

In January 2021, an Instagram user in the United States posted a picture of Abdullah Öcalan, one of the founding members of the Kurdistan Workers’ Party (PKK). The picture included the words “y’all ready for this conversation.” Underneath the picture the user wrote that it was time to talk about ending Öcalan’s isolation in prison on Imrali Island. They encouraged readers to engage in conversation about his imprisonment and the inhumane nature of solitary confinement.

Facebook removed the content for violating Instagram’s Community Guidelines after the post was automatically flagged for review (at this stage, the Board does not know if the content was removed by an automated system or through human review). These Guidelines under the heading “follow the law,” set out that “Instagram is not a place to support or praise terrorism, organized crime, or hate groups.” The Guidelines link to Facebook’s Community Standard on Dangerous Individuals and Organizations. These rules clarify that Facebook also prohibits any support or praise for groups, leaders, or individuals involved in terrorist activity or other serious crimes committed by these groups. The PKK has been designated a terrorist organization by multiple countries, including Turkey, the United States, and the EU.

The user states in their appeal that Öcalan has been a political prisoner for decades and that banning any reference to him prevents discussions that could advance the position of the Kurdish people. They argue that Öcalan’s philosophy is peaceful and that his writings are widely available in bookshops and online. The user compares Öcalan’s imprisonment to that of former South African President Nelson Mandela, noting that discussion of Öcalan’s imprisonment should be allowed and encouraged.

In July, the Oversight Board overturned Facebook's initial decision (even though once the case was taken Facebook admitted it had taken down the content in error before the Oversight Board even ruled). A key part of the discussion was that Facebook claimed it had "misplaced" an "internal policy exception" for the "Dangerous Individuals and Organizations" policy, as that misplaced policy included an exception for the discussion of human rights issues, even among those designated as "dangerous." As with many other Oversight Board rulings, there were a bunch of other recommendations as well.

In early August, Facebook responded to the recommendations, and, as Evelyn Douek noted, buried deep within them (recommendation 11) was an admission that Facebook would start revealing information and data on requests from governments based on "Community Standards" violations, rather than legal violations. Here's what the Oversight Board asked for:

Include information on the number of requests Facebook receives for content removals from governments that are based on Community Standards violations (as opposed to violations of national law), and the outcome of those requests.

While this may have been a minor part of this particular situation, it's important information that has previously not be available. And, as Douek also notes, human rights activists have been begging Facebook for this data for ages, with no luck. But the Oversight Board seems to have made it happen. Here's what Facebook had to say about it:

Our commitment: We are actively working to provide additional transparency when we remove content under our Community Standards following a formal report by a government, including the total number of requests we receive.

Considerations: As noted in our response to Recommendation 9 above, when we receive a formal government report about content that may violate local law, we first review it against our global Community Standards, just as we would review a report from any other source. If the content violates our Community Standards, we will remove it and count it in our Community Standards Enforcement Report. These reports are reviewed under a standardized process in the same way and against the same policies as reports from any other source. As a result, we are not currently able to report when we remove content based on a report by a government or from a Facebook user. In addition, we may receive reports of a piece of content that may violate our policies from multiple sources at the same time—for example, from a government and from user reports on Facebook. Such situations create additional challenges in determining whether content should be considered as removed in response to a government report.

We have been exploring ways to increase the level of transparency we provide to users and the public about requests we receive from governments, in line with best practices laid out by civil society efforts like the Santa Clara Principles and the Ranking Digital Rights project. We are prioritizing that work in response to this recommendation.

Next steps: We are planning work that will enable us to include information on content removed for violating our Community Standards following a formal report by a government, including the number of requests we receive, as a distinct category in our Transparency Center.

Again, as Douek points out, there is some wiggle room here in that Facebook committed to counting "formal" requests from government, which means that totally informal requests and suggestions -- pure "jawboning" situations -- might not be counted. But, still, this is a step forward in getting more transparency about how often governments push Facebook and Instagram to take down content that is legal, even if it may violate site policies.

Filed Under: community standards, content moderation, government requests, illegal content, jawboning, lawful but awful
Companies: facebook, oversight board

The Externalization Of Content Moderation: Facebook Explores 'Election Commission'

from the maybe-that's-not-a-bad-thing dept

We've covered plenty of stuff about the Oversight Board that Facebook set up. While we recognize the cynicism towards it and still think it's somewhat early to judge, for many of us following the Board's decisions closely, it's been really kind of eye opening how much the Board has really pushed Facebook to make real changes. Of course, there are many real structural issues with the way the Oversight Board is set up -- but the initial results have been fascinating.

And that's why it's interesting to hear that the company is considering creating another oversight board like setup for dealing with the challenges of moderation and elections, according to a report in the NY Times.

Facebook has approached academics and policy experts about forming a commission to advise it on global election-related matters, said five people with knowledge of the discussions, a move that would allow the social network to shift some of its political decision-making to an advisory body.

The proposed commission could decide on matters such as the viability of political ads and what to do about election-related misinformation, said the people, who spoke on the condition of anonymity because the discussions were confidential. Facebook is expected to announce the commission this fall in preparation for the 2022 midterm elections, they said, though the effort is preliminary and could still fall apart.

I already know (and have seen) some people responding cynically, that this is, once again, Zuckerberg trying to avoid responsibility by fobbing off the tough choices on other people. And, honestly, given Zuckerberg's own past comments that might be exactly a key motivation for him.

But, here's the crazy thing: this could be good even if it's done for cynical, responsibility-avoidance reasons.

First, even if it's Facebook trying to dodge responsibility, um, since no one trusts Facebook at all, shouldn't we want the responsibility taken away from Facebook? Lots of companies that don't want to take responsibility just throw up their hands and do nothing. Here, at least, Facebook is looking to hand it off to others to actually do the hard work that the company itself doesn't want to do. And, even better, it wants to hand it off to a bunch of actual subject matter experts rather than a stressed out, overworked internal team that gets no respect in the first place and has to deal with all sorts of competing interests and incentives.

Second, a huge part of the problem that everyone is facing regarding the internet giants and their moderation choices is that there is zero transparency involved, and that's a big part of what makes things so uncomfortable for people. From the outside, the big decisions (especially on the content moderation policy side) are suddenly announced by the company, with no insight into who is making these decisions or how. Indeed, the few leaks from behind the blue curtain have not been pretty. Letting actual experts figure out better ways forward, where the public knows who they are, and (hopefully) gets to see some of the reasoning and thought process, might not only come up with better overall policies, but also create more transparency and understanding about the policymaking process itself.

So, yes, sure, this very well may be a cynical move. And maybe Zuckerberg gets to laugh at people for letting him off the hook, but frankly, I don't want Zuckerberg personally to be on the hook for these kinds of decisions. Hand it off to experts, make the process public, let us learn about it and see what happens, rather than trusting Mark or his band of former political operatives to make the decisions.

Filed Under: content moderation, election commission, elections, mark zuckerberg, responsibility, transparency
Companies: facebook, oversight board

Facebook Says Trump's 'Indefinite' Suspension Is Now Two Years Off The Platform, And Then It'll See If He'll Behave

from the i-mean,-it's-something dept

In case you've been living under a rock for all of 2021, following the January 6th mob attack at the Capitol, where then President Donald Trump went to social media and posted things that could be read as egging on his insurrectionist followers, Twitter and Facebook suspended Trump's accounts. A few weeks later, the still relatively new and untested Oversight Board that will review a few Facebook decisions agreed to review the Trump decision. In late April, it upheld the removal, but said that Facebook's decision being for an "indefinite" length violated the company's own policies, and told the company it needed to either put a time limit on it, or come up with an actual rationale for a permanent suspension.

Last week, Facebook announced its response: the suspension would now be officially for two years -- but that doesn't mean Trump will automatically get his account back (just in time to ramp up his 2024 campaign...).

We are today announcing new enforcement protocols to be applied in exceptional cases such as this, and we are confirming the time-bound penalty consistent with those protocols which we are applying to Mr. Trump’s accounts. Given the gravity of the circumstances that led to Mr. Trump’s suspension, we believe his actions constituted a severe violation of our rules which merit the highest penalty available under the new enforcement protocols. We are suspending his accounts for two years, effective from the date of the initial suspension on January 7 this year.

At the end of this period, we will look to experts to assess whether the risk to public safety has receded. We will evaluate external factors, including instances of violence, restrictions on peaceful assembly and other markers of civil unrest. If we determine that there is still a serious risk to public safety, we will extend the restriction for a set period of time and continue to re-evaluate until that risk has receded.

When the suspension is eventually lifted, there will be a strict set of rapidly escalating sanctions that will be triggered if Mr. Trump commits further violations in future, up to and including permanent removal of his pages and accounts.

As is alluded in that snippet, the company also announced an official policy that applies "heightened penalties for public figures during times of civil unrest and ongoing violence." Just the fact that this needs to be its own category of content moderation policies should tell you something about the complexity of coming up with policies that can be equally applied across all situations. Every situation is different, and no policy is going to take into account all context. Two years is the new maximum suspension under this policy:

In establishing the two year sanction for severe violations, we considered the need for it to be long enough to allow a safe period of time after the acts of incitement, to be significant enough to be a deterrent to Mr. Trump and others from committing such severe violations in future, and to be proportionate to the gravity of the violation itself.

Of course, the Oversight Board tasked Facebook with much more than just putting a timeline on Trump's suspension and Facebook buried all the details beneath the headline grabbing "two years!" announcement.

Most of those policies focus on how the company deals with high profile and influential users who are violating its policies, and how it handles whether or not certain user speech is "newsworthy" or not. In particular, politicians will no longer receive an automatic assumption that all their speech is newsworthy. Instead, that speech will be judged in the same way as anyone else's speech for newsworthiness -- or so Facebook claims. In all reality, the actual lesson in all of this is that it's not possible to judge things equally. There is always additional context that makes each situation somewhat unique.

So, honestly, the only real end result here is that politicians won't receive the assumption of newsworthiness and seem more likely to face suspensions under these new policies.

For another view on the details of how Facebook dealt with this, read Evelyn Douek's analysis, in which she suggests that Facebook is basically doing the least possible that it can, which is allowed under the somewhat weak charter it put on the Oversight Board. Many people have highlighted how weak that charter is, but this may have been the first time it's been put to the test, and shows that when Facebook really doesn't want to commit to serious change, it has ways to avoid it. I'm less convinced that's the true outcome here, and honestly given how the Oversight Board has acted so far, I suspect that this weakish response from Facebook is only likely to embolden the Oversight Board to continue to hammer the company over its failures, even if Facebook wants to avoid committing to real change in response.

Filed Under: content moderation, donald trump, newsworthy, world leaders
Companies: facebook, oversight board

Techdirt Podcast Episode 282: The Facebook Oversight Board's Trump Decision

from the high-profile dept

Last week, the Oversight Board made its highest profile decision yet: upholding Facebook's suspension of Donald Trump, though with the caveat that it needs clearer policy reasons to make the suspension indefinite. Unsurprisingly, a whole lot of people have a whole lot of opinions on this, and we wanted to learn more about the decision from the source. Julie Owono is an Oversight Board member and the Executive Director of Internet Sans Frontières, and she joins us on this week's episode to discuss how this decision was reached and what it means for Facebook.

Follow the Techdirt Podcast on Soundcloud, subscribe via Apple Podcasts, or grab the RSS feed. You can also keep up with all the latest episodes right here on Techdirt.

Filed Under: content moderation, donald trump
Companies: facebook, oversight board

The Oversight Board's Decision On Facebook's Trump Ban Is Just Not That Important

from the undue-ado dept

Today is Facebook Oversight Board Hysteria Day, because today is the day that the Facebook Oversight Board has rendered its decision about Facebook's suspension of Donald Trump. And it has met the moment with an appropriately dull decision, dripping in pedantic reasonableness, that is largely consistent with our Copia Institute recommendation.

If you remember, we were hesitant about submitting a comment at all. And the reaction to the Board's decision bears out why. People keep reacting as though it is some big, monumental, important decision, when, in actual fact, it isn't at all. In the big scheme of things, it's still just a private company being advised by its private advisory board on how to run its business, nothing more. As it is, Trump himself is still on the Internet – it's not like Facebook actually had the power to silence him. We need to be worried about when there actually is power to silence people, and undue concern about Facebook's moderation practices only distracts us from them. Or, worse, leads people to try to create actual law that will end up having the effect of giving others the legal power to suppress expressive freedom.

So our pride here is necessarily muted, because ultimately this decision just isn't that big a deal. Still, as a purely internal advisory decision, one intended to help the company act more consistently in the interests of its potential user base, it does seem to be a good one given how it hews to our key points.

First, we made the observation that then-President Trump's use of his Facebook account threatened real, imminent harm. We did, however, emphasize the point that it was generally better to try not to delete speech (or speakers). Nevertheless, sometimes it might need to be done, and in those cases it should be done "with reluctance and only limited, specific, identifiable, and objective criteria to justify the exception." There might not ultimately be a single correct decision, we wrote, for whether speech should be left up or taken down. "[I]n the end the best decision may have little to do with the actual choice that results but rather the process used to get there."

And this sort of reasoning is basically at the heart of the Board's decision: Trump's posts were serious enough to justify a sanction, including a suspension, but imposing the indefinite suspension appeared to be unacceptably arbitrary. Per the Board, Facebook needs to make these sorts of decisions consistently and transparently from here on out.

On January 6, Facebook’s decision to impose restrictions on Mr. Trump’s accounts was justified. The posts in question violated the rules of Facebook and Instagram that prohibit support or praise of violating events, including the riot that was then underway at the U.S. Capitol. Given the seriousness of the violations and the ongoing risk of violence, Facebook was justified in imposing account-level restrictions and extending those restrictions on January 7. However, it was not appropriate for Facebook to impose an indefinite suspension. Facebook did not follow a clear published procedure in this case. Facebook’s normal account-level penalties for violations of its rules are to impose either a time-limited suspension or to permanently disable the user’s account. The Board finds that it is not permissible for Facebook to keep a user off the platform for an undefined period, with no criteria for when or whether the account will be restored.

The Board has given Facebook six months to re-evaluate the suspension in accordance with clear rules.

If Facebook determines that Mr. Trump’s accounts should be restored, Facebook should apply its rules to that decision, including any modifications made pursuant to the policy recommendations below. Also, if Facebook determines to return him to the platform, it must address any further violations promptly and in accordance with its established content policies.

As for what those rules should be, the Board also made a few recommendations. First, it noted that "political leader" versus "influential user" is not always a meaningful distinction. Indeed, we had noted that Trump's position cut both ways: as a political leader, there was public benefit to knowing what he had to say. On the other hand, that position also gave his posts greater ability to do harm. The Board for its part noted that context will matter; while the rules should ideally be the same for everyone, since the impact won't be, it is ok for Facebook to take into account the specific probability of imminent harm in making its decisions.

The Board believes that it is not always useful to draw a firm distinction between political leaders and other influential users. It is important to recognize that other users with large audiences can also contribute to serious risks of harm. The same rules should apply to all users of the platform; but context matters when assessing issues of causality and the probability and imminence of harm. What is important is the degree of influence that a user has over other users.

In general, the Board cited to general principles of human rights law, and specifically the Rabat Plan of Action "to assess the capacity of speech to create a serious risk of inciting discrimination, violence, or other lawless action." As for how long suspensions should generally last, they should be long enough to "deter misconduct and may, in appropriate cases, include account or page deletion." Facebook is therefore free to re-impose Trump's suspension as it re-evaluates it, if it feels it remains warranted. It just needs to do so in a more transparent way that would be scalable to other similar situations. As it summarized:

Facebook should publicly explain the rules that it uses when it imposes account-level sanctions against influential users. These rules should ensure that when Facebook imposes a time-limited suspension on the account of an influential user to reduce the risk of significant harm, it will assess whether the risk has receded before the suspension ends. If Facebook identifies that the user poses a serious risk of inciting imminent violence, discrimination or other lawless action at that time, another time-bound suspension should be imposed when such measures are necessary to protect public safety and proportionate to the risk. The Board noted that heads of state and other high officials of government can have a greater power to cause harm than other people. If a head of state or high government official has repeatedly posted messages that pose a risk of harm under international human rights norms, Facebook should suspend the account for a period sufficient to protect against imminent harm. Suspension periods should be long enough to deter misconduct and may, in appropriate cases, include account or page deletion.

As we suggested in our comment, the right policy choices for Facebook to make boil down to the ones that best make Facebook the community it wants to be. At its core, that's what the Board's decision is intended to help with: point out where it appears Facebook has fallen short of its own espoused ideals, and help it get back on track in the future.

Which is, overall, a good thing. It just isn't, as so many critics keep complaining, everything. The Internet is far more than just Facebook, no matter what Trump or his friends think. And there are far more important things for those of us who care about preserving online expression to give our attention to than this.

Filed Under: appeals, donald trump, permanent suspension, policies, rules, suspension
Companies: facebook, oversight board

Oversight Board Tells Facebook It Needs To Shape Up And Be More Careful About Silencing Minorities Seeking To Criticize The Powerful

from the pay-attention-to-this dept

Tomorrow, the Oversight Board is set to reveal its opinion on whether Facebook made the right decision in banning former President Trump. And that will get tons of attention. But the Board came out with an interesting decision last week regarding a content takedown in India, that got almost no attention at all.

Just last week, we wrote about an ongoing issue in India, where the government of Prime Minister Narendra Modi has failed in almost every way possible in dealing with the COVID pandemic, but has decided the best thing to focus on right now is silencing critics on Twitter. That backdrop is pretty important considering that the very next day, the Oversight Board scolded Facebook for taking down content criticizing Modi's government.

That takedown was somewhat different and the context was very different. Also, it should be noted that as soon as the Oversight Board agreed to take the case, Facebook admitted it had made a mistake and reinstated the content. However, this case demonstrates something important that often gets lost in all of the evidence free hand-wringing about "anti-conservative bias" from people who wrongly insist that Facebook and Twitter only moderate the accounts of their friends. The truth is that content all across the board gets moderated -- and often the impact is strongest on the least powerful groups. But, of course, part of their lack of power is that they're unable to rush onto Fox News and whine about how they're being "censored."

The details here are worth understanding, not because there was some difficult decision to make. Indeed, as noted already, Facebook realized it made a mistake almost immediately after the Oversight Board decided to look into this, and when asked why the content was taken down, basically admitted that it had no idea and that it was a complete and total mistake. Here was the content, as described by the Oversight Board ruling:

The content touched on allegations of discrimination against minorities and silencing of the opposition in India by “Rashtriya Swayamsevak Sangh” (RSS) and the Bharatiya Janata Party (BJP). RSS is a Hindu nationalist organization that has allegedly been involved in violence against religious minorities in India. “BJP” is India’s ruling party to which the current Indian Prime Minister Narendra Modi belongs, and has close ties with RSS.

In November 2020, a user shared a video post from Punjabi-language online media Global Punjab TV and an accompanying text. The post featured a 17-minute interview with Professor Manjit Singh, described as “a social activist and supporter of the Punjabi culture.” In its post, Global Punjab TV included the caption “RSS is the new threat. Ram Naam Satya Hai. The BJP moved towards extremism.” The media company also included an additional description “New Threat. Ram Naam Satya Hai! The BJP has moved towards extremism. Scholars directly challenge Modi!” The content was posted during India’s mass farmer protests and briefly touched on the reasons behind the protests and praised them.

The user added accompanying text when sharing Global Punjab TV’s post in which they stated that the CIA designated the RSS a “fanatic Hindu terrorist organization” and that Indian Prime Minister Narendra Modi was once its president. The user wrote that the RSS was threatening to kill Sikhs, a minority religious group in India, and to repeat the “deadly saga” of 1984 when Hindu mobs attacked Sikhs. They stated that “The RSS used the Death Phrase ‘Ram naam sat hai’.” The Board understands the phrase "Ram Naam Satya Hai" to be a funeral chant that has allegedly been used as a threat by some Hindu nationalists. The user alleged that Prime Minister Modi himself is formulating the threat of “Genocide of the Sikhs” on advice of the RSS President, Mohan Bhagwat. The accompanying text ends with a claim that Sikhs in India should be on high alert and that Sikh regiments in the army have warned Prime Minister Modi of their willingness to die to protect the Sikh farmers and their land in Punjab.

The post was up for 14 days and viewed fewer than 500 times before it was reported by another user for “terrorism.” A human reviewer determined that the post violated the Community Standard on Dangerous Individuals and Organizations and took down the content, which also triggered an automatic restriction on the use of the account for a fixed period of time. In its notification to the user, Facebook noted that its decision was final and could not be reviewed due to a temporary reduction in its review capacity due to COVID-19. For this reason, the user appealed to the Oversight Board.

So, you had an ethnic minority -- one who had been attacked in the past -- warning about those currently in power. And Facebook took it down, refused to review the appeal... until the Oversight Board turned its eye on it, and then admitted it was a mistake, and basically threw its hands in the air and said it had no idea why it had been taken down in the first place.

According to Facebook, following a single report against the post, the person who reviewed the content wrongly found a violation of the of the Dangerous Individuals and Organizations Community Standard. Facebook informed the Board that the user’s post included no reference to individuals or organizations designated as dangerous. It followed that the post contained no violating praise.

Facebook explained that the error was due to the length of the video (17 minutes), the number of speakers (two), the complexity of the content, and its claims about various political groups. The company added that content reviewers look at thousands of pieces of content every day and mistakes happen during that process. Due to the volume of content, Facebook stated that content reviewers are not always able to watch videos in full. Facebook was unable to specify the part of the content the reviewer found to violate the company’s rules.

Got that? Facebook is basically saying "yeah, it was a mistake, but that was because it was a long video, and we just had one person reviewing who probably didn't watch the whole video."

Here's the thing that the "oh no, Facebook is censoring people" don't get. This happens all the time. And none of us hear about it because the people it happens to often are unable to make themselves heard. They don't get to run to Fox News or Parler or some other place and yell and scream. And, this kind of "accidental" moderation especially happens to the marginalized. Reviewers may not fully understand what's going on, or not really understand the overall context, and may take the "report" claim at face value, rather than having the ability or time to fully investigate.

In the end, the Oversight Board told Facebook to put back the content, which was a no-brainer since Facebook had already done so. However, more interesting were its policy recommendations (which, again, are not binding on Facebook, but which the company promises to respond to). Here, the Oversight Board said that Facebook should make its community standards much more accessible and understandable, including translating the rules into more languages.

However, the more interesting bit was that it said that Facebook "should restore human review and access to a human appeals process to pre-pandemic levels as soon as possible while fully protecting the health of Facebook’s staff and contractors." There were some concerns, early in the pandemic, about how well content moderation teams could work from home, since a lot of that job involves looking at fairly sensitive material. So, there may be reasons this is not really doable just yet.

Still, this case demonstrates a key point that we've tried to raise about the impossibility of doing content moderation at scale. So much of it is not about biases, or incompetence, or bad policies, or not wanting to do what's right. A hell of a lot of it is just... when you're trying to keep a website used by half the world operating, mistakes are going to be made.

Filed Under: appeals, content moderation, free speech, india, minorities, mistakes, review, takedowns
Companies: facebook, oversight board

The Most Important Part Of The Facebook / Oversight Board Interaction Happened Last Week And Almost No One Cared

from the pay-attention dept

The whole dynamic between Facebook and the Oversight Board has received lots of attention -- with many people insisting that the Board's lack of official power makes it effectively useless. The specifics, again, for most of you not deep in the weeds on this: Facebook has only agreed to be bound by the Oversight Board's decisions on a very narrow set of issues: if a specific piece of content was taken down and the Oversight Board says it should have been left up. Beyond that, the Oversight Board can make recommendations on policy issues, but the companies doesn't need to follow them. I think this is a legitimate criticism and concern, but it's also a case where if Facebook itself actually does follow through on the policy recommendations, and everybody involved acts as if the Board has real power... then the norms around it might mean that it does have that power (at least until there's a conflict, and you end up in the equivalent of a Constitutional crisis).

And while there's been a tremendous amount of attention paid to the Oversight Board's first set of rulings, and to the fact that Facebook asked it to review the Trump suspension, last week something potentially much more important and interesting happened. With those initial rulings on the "up/down" question, the Oversight Board also suggested a pretty long list of policy recommendations for Facebook. Again, under the setup of the Board, Facebook only needed to consider these, but was not bound to enact them.

Last week Facebook officially responded to those recommendations, saying that it had agreed to take action on 11 of the 17 recommendations, is assessing the feasibility on another five, and was taking no action on just one. The company summarized those decisions in that link above, and put out a much more detailed pdf exploring the recommendations and Facebook's response. It's actually interesting reading (or, at least for someone like me who likes to dig deep into the nuances of content moderation).

Since I'm sure it's most people's first question: the one "no further action" was in response to a policy recommendation regarding COVID-19 misinformation. The Board had recommended that when a user posts information that disagrees with advice from health authorities, but where the "potential for physical harm is identified but is not imminent" that "Facebook should adopt a range of less intrusive measures." Basically, removing such information may not always make sense, especially if it's not clear that the information in disagreement with health authorities might not be actively harmful. As per usual, there's a lot of nuance here. As we discussed, early in the pandemic, the suggestions from "health authorities" later turned out to be inaccurate (like the WHO and CDC telling people not to wear masks in many cases). That makes relying on those health authorities as the be all, end all for content moderation for disinformation inherently difficult.

The Oversight Board's response in this issue more or less tried to walk that line, recognizing that health authorities' advice may adapt over time as more information becomes clear, and automatically silencing those who push back on the official suggestions from health officials may lead to over-blocking. But, obviously, this is a hellishly nuanced and complex topic. Part of the issue is that -- especially in a rapidly changing situation, where our knowledge base starts out with little information and is constantly correcting -- it's difficult to tell who is pushing back against official advice for good reasons or for conspiracy theory nonsense reasons (and there's a very wide spectrum between those two things). That creates (yet again) an impossible situation. The Oversight Board was suggesting that Facebook should be at least somewhat more forgiving in such situations, as long as they don't see any "imminent" harm from those disagreeing with health officials.

Facebook's response isn't so much pushing back against the Board's recommendation -- but rather to argue that it already takes a "less intrusive" approach. It also argued that Facebook and the Oversight Board basically disagree on the definition of "imminent danger" from bad medical advice (the specific issue came up in the context of someone in France recommending hydroxychloroquine as a treatment for COVID). Facebook said that, contrary to the Board's finding, it did think this represented imminent danger:

Our global expert stakeholder consultations have made it clear that, that in the context of a health emergency, the harm from certain types of health misinformation does lead to imminent physical harm. That is why we remove this content from the platform. We use a wide variety of proportionate measures to support the distribution of authoritative health misinformation. We also partner with independent third-party fact-checkers and label other kinds of health misinformation.

We know from our work with the World Health Organization (WHO) and other public health authorities that if people think there is a cure for COVID-19 they are less likely to follow safe health practices, like social distancing or mask-wearing. Exponential viral replication rates mean one person’s behavior can transmit the virus to thousands of others within a few days.

We also note that one reason the board decided to allow this content was that the person who posted the content was based in France, and in France, it is not possible to obtain hydroxychloroquine without a prescription. However, readers of French content may be anywhere in the world, and cross-border flows for medication are well established. The fact that a particular pharmaceutical item is only available via prescription in France should not be a determinative element in decision-making.

As a bit of a tangent, I'll just note the interesting dynamic here: despite "the narrative" which claims that Facebook has no incentive to moderate content due to things like Section 230, here the company is arguing for the ability to be more heavy handed in its moderation to protect the public from danger, and against the Oversight Board which is asking the company to be more permissive.

As for the items that Facebook "took action" on, a lot of them are sort of bland commitments to do "something" rather than concrete changes. For example, at the top of the list were things around confusion between the Instagram community guidelines and the Facebook community guidelines, and to be more transparent about how those are enforced. Facebook says that they're "committed to action" on this, but I'm not sure I can actually tell you what actions it's actually taken.

We’ll continue to explore how best to provide transparency to people about enforcement actions, within the limits of what is technologically feasible. We’ll start with ensuring consistent communication across Facebook and Instagram to build on our commitment above to clarify the overall relationship between Facebook’s Community Standards and Instagram’s Community Guidelines.

Um... great? But what does that actually mean? I have no idea.

Evelyn Douek, who studies this issue basically more than anyone else, notes that many of these commitments from Facebook are kind of weak:

Some of the “commitments” are likely things that Facebook had in train already; others are broad and vague. And while the dialogue between the FOB and Facebook has shed some light on previously opaque parts of Facebook’s content moderation processes, Facebook can do much better.

As Douek notes, some of the answers do reveal some pretty interesting things that weren't publicly known before -- such as how its AI deals with nudity, and how it tries to distinguish the nudity it doesn't want from things like nudity around breast cancer awareness:

Facebook explained the error choice calculation it has to make when using automated tools to detect adult nudity while trying to avoid taking down images raising awareness about breast cancer (something at issue in one of the initial FOB cases). Facebook detailed that its tools can recognize the words “breast cancer” but users have used these words to evade nudity detection systems, so Facebook can’t just rely on just leaving up every post that says “breast cancer.” Facebook has committed to providing its models with more negative samples to decrease error rates.

Douek also notes that some of Facebook's claims to be implementing the Board's recommendations are... misleading. They're actually rejecting the Board's full recommendation:

In response to the FOB’s request for a specific transparency report about Community Standard enforcement during the COVID-19 pandemic, Facebook said it was “committed to action.” Great! What “action,” you might ask? It says that it had already been sharing metrics throughout the pandemic and would continue to do so. Oh. This is actually a rejection of the FOB’s recommendation. The FOB knows about Facebook’s ongoing reporting and found it inadequate. It recommended a specific report, with a range of details, about how the pandemic had affected Facebook’s content moderation. The pandemic provided a natural experiment and a learning opportunity: Because of remote work restrictions, Facebook had to rely on automated moderation more than normal. The FOB was not the first to note that Facebook’s current transparency reporting is not sufficient to meaningfully assess the results of this experiment.

Still, what's amazing to me is that these issues, which might actually change key aspects of Facebook's moderation basically got next to zero public attention last week as compared to the simple decisions on specific takedowns (and the massive flood of attention the Trump account suspension decision will inevitably get).

Filed Under: content moderation, policies, recommendations
Companies: facebook, oversight board

Why We Filed A Comment With Facebook's Oversight Board

from the less-is-more dept

Back when Facebook's Oversight Board was just getting organized, a colleague suggested I represent people before it as part of my legal practice. As a solo lawyer, my entrepreneurial ears perked up at the possibility of future business opportunities. But the rest of me felt extremely uncomfortable with the proposition. I defend free speech, but I am a lawyer and I defend it using law. If Facebook removes you or your content that is an entirely lawful choice for it to make. It may or may not be a good decision, but there is nothing for law to defend you from. So it didn't seem a good use of my legal training to spend my time taking issue with how a private entity made the moderation decisions it was entirely within its legal rights to make.

It also worried me that people were regarding Facebook's Oversight Board as some sort of lawmaking body, and I was hesitant to use my lawyering skills to somehow validate and perpetuate that myth. No matter how successful the Board turns out to be, it is still limited in its authority and reach, and that's a good thing. What is not good is when people expect that this review system should (a) have the weight of actual law or (b) be the system that gets to evaluate all moderation decisions on the Internet.

Yet here I am, having just written a comment for the Copia Institute in one of its cases. Not because I changed my mind about any of my previous concerns, but because that particular high-profile case seemed like a good opportunity to help reset expectations about the significance of the Oversight Board's decisions.

As people who care about the online ecosystem we want those decisions to be as good as they can be because they will have impact, and we want that impact to be as good as it can be. With our comment we therefore tried to provide some guidance on what a good result would look like. But whether the Board gets its decisions right or wrong, it does no good for the public, or even the Board itself, to think its decisions mean more than they do. Nor is it necessary: the Oversight Board already has a valid and even valuable role to play. And it doesn't need to be any more than what it actually is for it to be useful.

It's useful because every platform makes moderation decisions. Many of these decisions are hard to make perfectly, and many are made at incredible scale and speed. Even with the best of intentions it is easy for platforms to make moderation decisions that would have been better decided the other way.

And that is why the basic idea of the Oversight Board is a good one. It's good for it to be able to provide independent review of Facebook's more consequential decisions and recommend how to make them better in the future. Some have alleged that the board isn't sufficiently independent, but even if this were true, it wouldn't really matter, at least insofar as Facebook goes. What is important is that there is any operational way to give Facebook's moderation decisions a second look, especially in a way that can be informed by additional considerations that may not have been included in the original decision. That the Oversight Board is designed to provide such review is an innovation worth cheering.

But all the Oversight Board can do is decide what moderation decision might have been better for Facebook and its user community. It can't articulate, and it certainly can't decree, a moderation rule that could or should apply at all times on every platform anywhere, including platforms that are much different, with different reaches, different purposes, and different user communities than Facebook has. It would be impossible to come up with a universally applicable rule. And it's also not a power this Board, or any similar board, should ever have.

As we said in our comment, and have explained countless times on these pages, platforms have the right to decide what expression to allow on their systems. We obviously hope that platforms will use this right to make these decisions in a principled way that serves the public interest, and we stand ready to criticize them as vociferously as warranted when they don't. But we will always defend their legal right to make their moderation choices however perfectly or imperfectly they may make them.

What's important to remember in thinking about the Oversight Board is that this is still Facebook making moderation decisions. Not because the Board may or may not be independent from Facebook, but because Facebook's decision to defer to the Board's judgment is itself a moderation decision. It is not Facebook waiving its legal right to make moderation choices but rather it exercising that very right to decide how to make those choices, and this is what it has decided. Deferring to the Board's judgment does not obviate real-world law protecting its choice; it's a choice that real world law pointedly allows Facebook to make (and, thanks to Section 230, even encourages Facebook to try).

The confusion about the mandate of the Oversight Board seems to stem in part from the way the Board has been empowered and operates. In many ways it bears the hallmarks of a self-contained system of private law, and in and of itself that's fine. Private law is nothing new. For instance, when you hear the term "arbitration," that's basically what arbitration is: a system of private law. Private law can exist alongside regular, public, democratically-generated law just fine, although sometimes there are tensions because for it to work all the parties need to agree to abide by it instead of public law, and sometimes that consent isn't sufficiently voluntary.

But consent is not an issue here: before the Oversight Board came along Facebook users had no legal leverage of any kind over Facebook, so this is now a system of private law that Facebook has agreed can give them some. We can and should of course care that this system of private law is a good one, well-balanced and equitable, and thus far we've seen no basis for any significant concern. We instead see a lot of thoughtful people working very hard to try to get it right and open to being nudged to do better if such nudging should happen to be needed. But even if they were getting everything all wrong, in the big picture it doesn't really matter either, because ultimately it is only Facebook's oversight board, inherently limited in its authority and reach to that platform.

The misapprehension that this Board can or should somehow rule over all moderation decisions on the Internet is also not helped by the decision to call it the "Oversight Board," rather than the "Facebook Oversight Board." Perhaps it could become a model for other platforms to use, and maybe, just maybe, if it really does become a fully spun-off independent, sustainable, self-contained private law system it might someday be able to supply review services to other platforms too—provided, of course, that the Board is equipped to address these platforms' own particularities and priorities, which may differ significantly from Facebook's.

But right now it is only a solution for Facebook and only set up to consider the unique nature of the Facebook platform and what Facebook and its user community want from it. It is far from a one-size-fits-all solution for Internet content moderation generally, and our comment said as much, noting that the relative merit of the moderation decision in question ultimately hinged on what Facebook wanted its platform to be.

Nevertheless, it is absolutely fine for it to be so limited in its mission, and far better than if it were more. Just as Facebook had the right to acquiesce to this oversight board, other platforms equally have the right, and need to have the right, to say no to it or any other such board. It won't stop being important for the First Amendment to protect this discretion, regardless of how good a job this or any other board might do. While the Oversight Board can, and likely should, try to incorporate First Amendment values into its decisions to the extent it can, actual First Amendment law operates on a different axis than this system of private law ever would or could, with different interests and concerns to be balanced.

It is a mistake to think we could simply supplant all of those considerations with the judgment of this Oversight Board. No matter how thoughtful its decisions, nor how great the impact of what it decides, the Oversight Board is still not a government body. Neither it (nor even Facebook) has the sort of power the state has, nor any of the Constitutional limitations that would check it. Facebook remains a private actor, a company with a social media platform, and Facebook's Oversight Board simply an organization built to help it make its platform better. We should be extremely wary of expecting it to be anything other than that.

Especially because that's already plenty for it to be in order for it to be able to do some good.

Filed Under: arbitration, content moderation, private law
Companies: facebook, oversight board

Zuckerberg's Grand Illusion: Understanding The Oversight Board Experiment

from the tinker-bell dept

Everyone, it seems, has an opinion about The Oversight Board -- which everyone refers to as the Facebook Oversight Board, because despite its plans to work with other social media companies, it was created by Facebook, and feels inevitably connected to Facebook by way of its umbilical cord. As we noted earlier this month, after the Oversight Board's first decisions came down, everyone who had a strong opinion about the Oversight Board seemed to use the results to confirm their existing beliefs about it.

To some, the Oversight Board is just an attempt for Facebook and Mark Zuckerberg to avoid taking responsibility for societal-level impacts of its platform. For others it's a cynical ploy/PR campaign to make it look like it's giving up some of its power. To still others, it's a weak attempt to avoid regulation. To many, it's a combination of all three. And, then, to some, it's an interesting experiment in content moderation that attempts to actually separate some final decision making ability from a website itself. And, again, it could still be some combination of all of those. As I've said since it launched, I find it to be an interesting experiment, and even if the cynical reasons are a driving force behind it, that may not matter if the Board actually turns into a sort of authority that creates change. As I recently talked about in a podcast episode, the norms may become important.

That is, even if the whole thing is a cynical marketing ploy by a finger-waving Mark Zuckerberg, that might not matter if the Board itself actually is able to create meaningful change within the company and how it handles both moderation decisions and content moderation policy. And it's reasonable to point out that this has a high chance of failure and that there are a variety of structural problems in how the Board is setup, but that doesn't mean failure is guaranteed. And there's enough of a chance that the Board could make a difference that I think it's worth paying close attention to what happens with it.

And, if you believe that it's important to understand, then you owe it to yourself to read Prof. Kate Klonick's brilliant, thorough and detailed account of the making of the Board with lots of behind-the-scenes tidbits. I can think of no one better to do this kind of reporting. Klonick, a former reporter who has a JD and a PhD and is now a law professor, wrote the seminal paper on social media content moderation, The New Governors, which is also required reading for anyone seeking to understand content moderation online.

Buried deep within the article is an important point that gets to what I say above, about how the norms around the Board might make it powerful, even if the Board is not initially imbued with actual power:

Harris likened the board to Tinkerbell: “At the end of the day you can build all the things, but you just have to have enough people that believe in order to make it real.”

Some have been arguing that Klonick's New Yorker article is, itself, Facebook PR or a puff piece, but claiming such is only evidence that either (a) you haven't read it or (b) that your reading comprehension is so bad as to call into question pretty much anything else you write. Facebook does not come out of the article looking very good. Nor does the Oversight Board. But what the article does reveal is... just how fucking messy all of this is. It's a keen distillation of my impossibility theorem as applied to the world's largest collection of speech.

Indeed, the article includes this line that I'm sure will be repeated by critics of Facebook and the Board until the end of time itself:

At one point, a director of policy joked about ways to make the board seem independent, asking, “How many decisions do we have to let the Oversight Board win to make it legit?”

The article also makes clear the competing interests (some of which are seen in the opinions people hold about the Board) regarding how much power to give the Board, and the eventual decision to... well... give it very, very little. Facebook employees were worried that giving it too much power would give the Board the ability to literally kill Facebook. Though, on the flip side, the lack of power... makes the Board seem fairly hapless.

There's also a fun section on how people wanted Board members to be selected:

One attendee suggested letting the first board members choose the rest, to preserve their independence from the company. Lauren Rivera, a professor at Northwestern’s business school, cautioned against this approach: “It’s empirically proven that when you have a group self-select, in the absence of any kind of guidance, they just pick more people that look like them.” The experts then began giving their own ideas. Journalists said that the board should be mostly journalists. International human-rights lawyers said that it should be all international human-rights lawyers. Information scientists said that it should be “anyone but lawyers.” A white man at a think tank said that it should be populated with “regular people.”

Even, it seems, picking the judges presents something of a content moderation challenge. Indeed, as you might predict, in a moment of heightened political polarization, that was felt in the process of setting up the Board:

People familiar with the process told me that some Republicans were upset about what they perceived to be the board’s liberal slant. In the months leading up to the appointments, conservative groups pushed the company to make the board more sympathetic to Trump. They suggested their own lists of candidates, which sometimes included members of the President’s family, most notably Ivanka and the President’s sons. “The idea was, either fill this board with Trump-supporting conservatives or kill it,” one person familiar with the process said. In early May, shortly after the board members were announced, Trump personally called Zuckerberg to say that he was unhappy with the makeup of the board. He was especially angry about the selection of Pamela Karlan, a Stanford Law professor who had testified against him during his first impeachment. “He used Pam as an example of how the board was this deeply offensive thing to him,” the person familiar with the process said. Zuckerberg listened, and then told Trump that the members had been chosen based on their qualifications. Despite the pressure from Trump, Facebook did not change the composition of the board. (Trump declined to comment.)

That seems quite notable, of course, now that the Board is in the middle of deciding if Trump can get his account back.

But perhaps the most notable part of the article is Klonick's interview with Mark Zuckerberg himself. As Klonick notes, Zuckerberg has more or less accidentally put himself in the position of being the arbiter of what is and what is not allowed on the biggest platform for communication ever created. And he seems to be struggling with that, because figuring out how to deal with the societal-level problems that it creates is not nearly as interesting as just building products, which appears to be much more of his passion.

He looked tired. He seemed more at ease talking about “product” or “building tools” than he did discussing ethics or politics. It struck me that he was essentially a coder who had found himself managing the world’s marketplace of ideas. “The core job of what we do is building products that help people connect and communicate,” he said. “It’s actually quite different from the work of governing a community.” He hoped to separate these jobs: there would be groups of people who built apps and products, and others—including Facebook’s policy team and now the board—who deliberated the thorny questions that came along with them. I brought up a speech he gave at Georgetown, in 2019, in which he noted that the board was personally important to him, because it helped him feel that, when he eventually left, he would be leaving the company in safe hands. “One day, I’m not going to be running the company,” he told me. “I would like to not be in the position, long term, of choosing between someone who either is more aligned with my moral view and values, or actually is more aligned with being able to build high-quality products.”

I asked what kinds of cases he hopes the board will take. “If I was them, I’d be wary of choosing something that was so charged right off the bat that it was immediately going to polarize the whole board, and people’s perception of the board, and society,” he told me. He knew that critics wished the board had more power: “This is certainly a big experiment. It’s certainly not as broad as everyone would like it to be, upfront, but I think there’s a path for getting there.” But he rejected the notion that it was a fig leaf. “I’m not setting this up to take pressure off me or the company in the near term,” he said. “The reason that I’m doing this is that I think, over the long term, if we build up a structure that people can trust, then that can help create legitimacy and create real oversight. But I think there is a real risk, if it gets too polarized too quickly, that it will never be able to blossom into that.”

These statements are, at the same time, obviously true, obviously predictable... and just not very insightful. I don't envy anyone being in a position where much of the world's ills are blamed on what was your college coding side hustle, but Zuckerberg has now had a good 15 years to consider that all of these things are connected. You can't separate the policy and the product building. You can't have "the builders" and "the thinkers" as if they're two separate camps. They kind of all need to be combined.

In some sense, this reminds me (on a much different scale) of the rush in the early aughts to have companies appoint "Chief Digital Officers" in all sorts of old school companies. As we discussed at the time, thinking digitally isn't a separate job function. For most companies, digital thinking has to be a part of everything -- and that applies to understanding the policy and ethical implications of product design as well.

It's disappointing -- though not surprising -- that Zuckerberg hasn't quite grasped this yet.

But that doesn't make the Board itself any less of an interesting experiment. Indeed, if enough people actually believe in this Tinker Bell, it might not even matter that Zuckerberg himself misunderstands how the Board itself could be useful. It might just make itself useful by accident.

Filed Under: content moderation, kate klonick, mark zuckerberg, morality, oversight, regulation
Companies: facebook, oversight board