Content Moderation Case Studies: The Challenges In Moderating Information Regarding Eating Disorders (2012)

from the not-as-simple-as-you-might-think dept

Summary: In 2012, the Huffington Post did an exposé on eating disorder blogs, mainly on the site Tumblr. It discussed the world of “thinspo” and “thinspiration” blogs, that focused on building a community around losing unhealthy amounts of weight. In response, Tumblr announced that it was banning “self harm” blogs, and classified eating disorder blogs among those no longer allowed.

Three years later, a study by Munmun De Choudhury discussed how there was still eating disorder information on Tumblr, but that it was mainly split into two different categories: those who were supportive of eating disorders such as anorexia (referred to as “proana”) as well as communities built up around recovering from eating disorders. One interesting finding of the report was that the “recovery” groups often used the same keywords and messaging, in an attempt to permeate among the “proana” groups, in order to try to encourage those with eating disorders to seek support, therapy, and help towards recovery.

That same year, Amanda Hess argued in Slate that the rush to ban content about eating disorders on social media (or, in the case of France, where such things were outlawed) was the wrong approach.

"But while we know anorexia can kill, we’re not quite sure what happens to people who read about it online. In an article published last month, Canadian criminologists Debra Langan and Nicole Schott could find 'no scholarly evidence' that pro-ana blogs pose a threat to their audiences. If they do, there’s no proof that any of our social remedies—censorship, PSAs, or prison time—do anything to help. These campaigns are most obviously effective at flattering the egos of the lawmakers and tech execs who champion them. When a girl searches Tumblr for a pro-ana–adjacent term like #thinspo or #thighgap now, Tumblr intercepts her request with bland concern ('Everything OK?'), then advises her to check out the cutesy motivational messaging on the National Eating Disorders Association’s Tumblr instead. However the girl responds, Tumblr can feel satisfied it’s performed its civic responsibility. The strategy recalls the one favored by a 19th-century doctor who believed that reading novels caused hysteria in women: He counseled men to confiscate their wives’ fiction and replace it with a book on 'some practical subject,' like 'beekeeping.'"

The following year, De Choudhury and other authors released another study detailing how pro-eating disorder groups would get around social media blocks on their content by changing words around, or slightly misspelling them, suggesting that the out-and-out blocking method was likely to continue to be ineffective.

Another article suggested that the blocks almost made it easier to find information about eating disorders, because dozens of new hashtags were created for the community, rather than just a few before social media sites began to ban such content.

A study released in the peer-reviewed New Media & Society journal in 2018 highlighted how easy it appeared to be for users to get around attempts to block content regarding certain eating disorders. The researcher, Ysabel Gerrard, looked mainly at Pinterest, Tumblr, and Instagram, finding that while all three had some policies in place regarding eating disorder information, it was not difficult to find groups or sites dedicated to such information.

As summarized by Wired:

"She immediately found that Instagram’s pro-ED hashtag ban has an easy workaround: You can search for people who have the keywords in their usernames, just not hashtagged in their posts. She identified 74 public accounts that had terms like 'proana,' 'proanorexia,' or 'thighgap' in their names or bios and who also posted pro-ED content. Then, she analyzed 1,612 of their posts—only 561 of which had hashtags—by cataloguing the content of the image and its caption.

"On Tumblr, Gerrard followed a number of terms related to pro-ED content, like 'thinspo,' 'proana,' and 'bulimic.' Tumblr allows you to follow topics without needing to follow specific users. For example, you can simply follow 'movies' without following any specific user who posts about that topic. Through this method, she found 50 pro-ED blogs and analyzed 20 posts from each, or 1,000 posts total. Only 218 of the posts were tagged."

The report also found that the recommendation algorithms often drove users towards more such content. By saving a few 'proana' blogs, Gerrard found that Tumblr began recommending more. While it did also recommend some recovery blogs, Gerrard found them easy to exclude.

"Once I had followed ED-related terms - anorexia, anorexic, bulimia, bulimic, thinspiration, thinspo, proana, purge, purging - the platform delivered this content to me through my dashboard and also via email. Tumblr showed me relevant posts and suggested a list of users whose accounts I should follow. As some of these terms are not straightforwardly pro-ED (unlike, for example, proana), I was presented with blogs identifying as ‘pro-recovery’ in their biographies. But I excluded these blogs from the dataset as they were not the focus of my analysis. Tumblr recommended blogs that were, for example, ‘big in proana’ or ‘like’ other popular blogs. I identified fifty pro-ED users through this method."

Another study from 2014 argued that even the “proana” content represented a “double-edged sword” and might help some of those either with eating disorders, or those at risk, to recognize that what they were exploring was unhealthy.

Decisions to be made by Tumblr/Instagram/Pinterest:

• How do you deal with information about eating disorders? Is it actually possible to ban it?

• How do you distinguish between “proana” and “recovery” content?

• Are there other interventions that can be done, such as putting up warning labels, or directing users towards “recovery” resources when they search on certain terms?

• How should recommendation algorithms handle information about eating disorders? Does it need to be adjusted to avoid sending people towards content that glorifies such disorders?

Questions and policy implications to consider:

• Does banning content of those promoting unhealthy eating disorders actually help prevent eating disorders?

• Does reading about eating disorders function as a how-to guide for the vulnerable, or does it help those at risk recognize that risk? If both, how do you balance these two competing forces?

• Can pointing people towards recovery content or other helpful resources lead to better outcomes?

Resolution: Various websites struggle with how to deal with eating disorder information and communities. Attempts to ban it have continued to fail, as the various communities continue to figure out ways to route around any ban. The research on the impact of this content remains mixed, however, and there have been some concerns that efforts to ban such content on certain platforms only makes it move to others that are less well organized to handle the issue.

The latest is a report that teens who were engaged in eating disorder discussions on Tumblr have now moved to TikTok. However, that same article also notes that, unlike Tumblr, TikTok seems to have a number of users who celebrate healthy eating and living, and that TikTok’s algorithm may be inserting such videos mixed in with those discussing more unhealthy eating behavior.

Originally published on the Trust & Safety Foundation website.

Filed Under: anorexia, content moderation, eating disorders, proana, recovery