Content Moderation Case Study: Can An Open Encyclopedia Handle Disinformation? (2005)

from the disinfo-on-an-open-platform dept

Summary: Wikipedia was founded in 2001 and the open encyclopedia that anyone could edit grew much faster than most people (including its founders) expected. In 2005, one of the first big controversies concerning disinformation on Wikipedia arose, when journalist and political figure John Seigenthaler wrote an article in USA Today calling out claims on Wikipedia that, among other things, he was involved in the assassinations of both John F. Kennedy and Robert F. Kennedy (for whom he worked for a time).

The entry in question read:

John Seigenthaler Sr. was the assistant to Attorney General Robert Kennedy in the early 1960s. For a short time, he was thought to have been directly involved in the Kennedy assassinations of both John, and his brother, Bobby. Nothing was ever proven.

John Seigenthaler moved to the Soviet Union in 1972, and returned to the United States in 1984.

He started one of the country's largest public relations firms shortly thereafter.

Two months before publishing the USA Today article, Seigenthaler reached out to Wikipedia cofounder Jimmy Wales and requested that his entry be deleted. It was, but other Wikipedia users soon brought it back, leading Seigenthaler to publish the story.

Seigenthaler referred to this entire incident as “character assassination” and it suddenly put the fact checking of Wikipedia into tremendous focus, leading some to question whether or not the site could even survive.

Seigenthaler’s article raised many questions about Wikipedia, how it was edited, and how reliable it was as a research tool. For years, Seigenthaler insisted he wouldn’t edit his own entry because it would lend legitimacy to the service that he wanted no part of.

The controversy received widespread media coverage, and raised many questions about how Wikipedia can and should deal with misinformation on its platform while staying true to its principles and concept as an encyclopedia built on anyone’s contributions.

Decisions to be made by Wikipedia:

• Should it step in and edit Seigenthaler’s entry?
• Should it block attempts to restore the false information?
• Should it block further edits and lock down Seigenthaler’s page?
• Should it change its overall policies regarding open editing?

Questions and policy implications to consider:

• Are there ways to limit “vandalism” on an open system?
• Were certain pages at greater risk of false information that others? If so, how should that difference be handled?
• Will any changes decrease the trust people have in Wikipedia?
• Will any changes increase the trust people have in Wikipedia?

Just weeks after the incident and controversy, Nature published a study claiming that Wikipedia was equally as reliable as the esteemed Encyclopedia Britannica.

Seigenthaler continued to question the entire Wikipedia concept for years. Separately, while in the USA Today piece, Seigenthaler detailed his failed efforts to track down who had made the edits (including explaining how he considered possible legal action), the person who created the page, Brian Chase came forward a few weeks after the story went viral, saying it was just a joke that he regretted.

Filed Under: case study, content moderation, disinformation, john segenthaler
Companies: wikipedia

Content Moderation Case Study: Dealing With Misinformation During A Pandemic (2020)

from the misinfo-is-hard dept

This series of case studies is published in partnership with the Trust & Safety Foundation to examine the difficult choices and tradeoffs involved in content moderation. Learn more »

Summary: In early 2020, with the world trying to figure out how to deal with the COVID-19 pandemic, one of the big questions faced by internet platforms was how to combat mis- or disinformation regarding the pandemic. This was especially complex, given that everyone -- including global health experts were trying to figure out what was accurate themselves, and as more information has come in, the understanding of the disease, how it spread, how to treat it, the level of risk, and much, much, has kept changing.

Given the fact that no one fully understood what was going on, plenty of people rushed in to try to fill the void with information. Most social media firms put in place policies to try to limit or take down misinformation or disinformation using a variety of policies and tactics. But determining what is misinformation as opposed to legitimate truth-seeking can be very tricky in the midst of a pandemic.

In late March, as the pandemic was hitting full swing, an article appeared on the website Medium by Aaron Ginn, a self-described Silicon Valley “growth hacker,” arguing that the response to COVID-19 was overblown and driven by “hysteria.” The piece included many citations of credible data and reports, but also included a few quotes significantly downplaying the risk of COVID-19, including saying that its “transmission rates are very similar to seasonal flu.”

The story started to spread widely, mainly after a number of Fox News hosts started tweeting it. As the story got more and more attention, Carl Bergstrom, a professor of biology at the University of Washington, decided to critique Ginn’s article in great detail via an extended Twitter thread. Bergstrom makes a fairly compelling case that Ginn’s lack of expertise in epidemiology led him to making a number of mistakes in his analysis, in particular, not understanding how viruses spread, and how that information is tracked. He also argued that Ginn cherry-picked certain data to support a thesis. Bergstrom and others started arguing that Ginn’s Medium piece was (perhaps not intentionally) misinformation.

Decisions to be made by Medium:

• Should the piece be allowed to remain on the platform or should it be taken down?
• Are there alternatives to taking the piece down or leaving it as is?
• If the piece is taken down, should the user be banned?
• What policies do a potentially well-meaning, but possibly incorrect, piece about a pandemic violate?

• How do you distinguish mis- and disinformation from potentially well-meaning, but inexperienced analysis?
• Should earnest but incorrect information be allowed?
• How does that analysis change when human health is at stake?
• Do pieces like this need to be judged against official government and healthcare sources? If so, how is that reconciled with constantly updated information from those sources?
• If articles like this are taken down, does that create more “credibility” for articles that remain on the site?
• If articles like this are left up, does that create more credibility problems for other articles on the site, that may have a much stronger and more competent analysis?
• Does taking down an article like this create political backlash?  If so, does that matter?
• Do the site and its management want to feel responsible if people take bad health advice that was posted to their site, and possibly come to harm from it?

That was not the end of things, however. The article was reposted to the site ZeroHedge, and copies were stored and reposted in other places as well. It also created a short-lived uproar among those who felt that Medium’s moderation decision was unfair. The Wall Street Journal celebrated Ginn, saying that after being “targeted for censorship,” it only made Ginn more influential. Other publications, including the NY Times, the Washington Post, and Slate, all wrote about the dangers of amateurish analysis in the midst of a pandemic.

Ginn, at one point, appeared to be fine with Medium’s decision, saying that internet platforms “are free to associate with whom they want,” (though he has since removed the tweet saying that). He has continued to use other social media to argue that the claims about COVID-19 were overblown.

Filed Under: aaron ginn, carl bergstrom, case study, content moderation, covid-19, misinformation
Companies: medium

Content Moderation Case Study: Dealing With Misinformation In Search (2004)

from the misinformation-goes-way-back-online dept

This series of case studies is published in partnership with the Trust & Safety Foundation to examine the difficult choices and tradeoffs involved in content moderation. Learn more &raquo

Summary: Google’s biggest early innovation in search was that it used inbound links as a tool for determining the popularity of a website, and thus what its relevance to a particular search might be. That feature, however, created some side effects that raised concerns about how search results might lead to misinformation, or how the search engine might be gamed.

One of the earliest examples of this was the discovery in 2004 that the first result of a search on the word “jew” pointed to a blatantly anti-semitic website, Jewwatch. It was widely theorized that the reason for this was that the singular noun “jew” was more likely to be used by those pushing anti-semitic arguments, rather than the more common adjective “jewish” or the phrase “jewish wo/man” etc. Also, the site Jewwatch had been in existence for many years, and had many inbound links from other sources.

Some also believed that the people behind Jewwatch had used an early search engine optimization technique known as “Googlebombing” to purposefully game the results — deliberately linking to Jewwatch from other sites, and using the word “jew” as the link text.

As this result got attention, Google came under tremendous pressure to change the search result, as people accused the company of anti-semitism or deliberately pointing to the Jewwatch site in search results. The Anti-Defamation League sent a letter to Google asking it to explore whether or not its ranking system needed to be changed (though the ADL also posted an article to its own site telling people that it was clear that the result was not intentional, or done for nefarious reasons). Some politicians, including Senator Chuck Schumer, also got involved to pressure Google to change its results.

Decisions to be made by Google:

• Should the top search results be manually changed when it is discovered they lead to misinformation and hate?
• Should the algorithm be changed to try to avoid these results?
• Should the company do nothing and say that the algorithm decides the results, period?
• Should any decision set a precedent for future decisions, and if so, what policies and guidelines need to be put in place to deal with future cases?
• Are there other ways to respond to this situation?
• How should Google handle attempts to game search via things like Googlebombing?

Questions and policy implications to consider:

• If any changes are made, will lots of others expect similar changes to be made as well?
• Will making changes lead to questions regarding the credibility of search results and the Google algorithm?
• What sorts of policies and processes need to be in place to deal with these kinds of requests?
• Will any changes have other, unintended consequences as well?
• Are search engine optimization techniques nefarious? Can they be? If so, how do you distinguish between good intentions and bad intentions?
• If you block certain techniques, such as Googlebombing, will that stop the practice when used for good purposes as well?

Resolution: Google responded by clearly stating that it had no direct intentions to change its algorithm. However, it did decide to provide more information, by using the advertising space above the top result to encourage people to click through for more information about how the results came about:

The company also stated that it would “explore additional ways of addressing” issues like this “in the future.”

Perhaps more interesting, however, was that Google’s users took matters into their own hands, and realized that if Jewwatch was Googlebombing, they could use the same tools to diminish the result. A campaign was quickly organized online, with many people linking the word “jew” to Wikipedia’s page on Judaism, and indeed, this worked to get that result to the top of the rankings.

Over time, Google’s algorithms were adjusted globally to try to diminish the power of Googlebombing for any reason (good or bad). In 2007, the company announced that it believed its algorithm would filter out attempts at Googlebombing. In that discussion, the employees who helped stop the effectiveness of Googlebombing explained why they did so, and how they believed it was better to take a holistic approach (which was more scalable) than responding to individual “bad” results:

People have asked about how we feel about Googlebombs, and we have talked about them in the past. Because these pranks are normally for phrases that are well off the beaten path, they haven't been a very high priority for us. But over time, we've seen more people assume that they are Google's opinion, or that Google has hand-coded the results for these Googlebombed queries. That's not true, and it seemed like it was worth trying to correct that misperception. So a few of us who work here got together and came up with an algorithm that minimizes the impact of many Googlebombs.

The next natural question to ask is "Why doesn't Google just edit these search results by hand?" To answer that, you need to know a little bit about how Google works. When we're faced with a bad search result or a relevance problem, our first instinct is to look for an automatic way to solve the problem instead of trying to fix a particular search by hand. Algorithms are great because they scale well: computers can process lots of data very fast, and robust algorithms often work well in many different languages. That's what we did in this case, and the extra effort to find a good algorithm helps detect Googlebombs in many different languages. We wouldn't claim that this change handles every prank that someone has attempted. But if you are aware of other potential Googlebombs, we are happy to hear feedback in our Google Web Search Help Group.

Filed Under: case study, content moderation, misinformation, search
Companies: google