Content Moderation Case Study: Using Fact Checkers To Create A Misogynist Meme (2019)
from the content-moderation-inception dept
Summary: Many social media sites include fact checkers in order to either block or at least highlight information that is determined to be false or misleading. However, in some ways, that alone can create a content moderation challenge.
Alan Kyle, a privacy and policy analyst, noticed this in late 2019 when he came across a meme picture on Instagram from a meme account called “memealpyro” showing what appeared to be a great white shark leaping majestically out of the ocean. When he spotted the image, it had been blurred, with a notice that it had been deemed to be “false information” after being “reviewed by independent fact checkers.” When he clicked through to unblur the image, next to the image there was a small line of text saying “women are funny.” And beneath that the fact checking flag: “See why fact checkers say this is false.”
The implication of someone coming across this image with this fact check is that the fact check is on the statement, leading to the ridiculous/misogynistic conclusion that women are not funny and that an independent fact checking organization had to flag a meme image suggesting otherwise.
As Kyle discusses, however, this seemed to be an attempt to rely on fact checkers checking one part of the content, in order to create the misogynistic meme. Others had been using the same image -- which was computer generated and not an actual photo -- and claiming that it was National Geographic’s “Picture of the Year.” This belief was so widespread that National Geographic had to debunk the claim (though it did so by releasing other, quite real, images of sharks to appease those looking for cool shark images).
The issue, then, was that fact checkers had been trained to debunk the use of the photo, on the assumption it was being posted with the false claim that it was National Geographic’s “Photo of the Year,” and Instagram’s system didn’t seem to expect that other, different claims might be appended to the same image. When Kyle clicked through to see the explanation, it was only about the “Picture of the Year” claim (which was not made on this image), and (obviously) not on the statement about women.
Kyle’s hypothesis is that Instagram’s algorithms were trained to flag the picture as false, and then possibly send the flagged image to a human reviewer -- who may have just missed that the text associated with this image was unrelated to the text for the fact check.
Decisions to be made by Instagram:
- If the caption and a picture need to be combined to be designated as false information, how should Instagram fact checkers handle cases where that information is separated?
- How should fact checkers handle mixed media content, in which text and graphics or video may be deliberately unrelated?
- Should automated tools be used to flag viral false information in a way that might be gamed?
- How much human review should be used for algorithmically flagged “false” information?
- When there is an automated fact checking flagging algorithm, how will users with malicious intent try to game the system, as in the above example?
- Is fact checking the right approach to “meme’d” information that is misleading, but not in a meaningful way?
- Would requiring fact checking across social media lead to more “gaming” of the system as in the case above?
There are many factors working against the moderator making the right decision. Facebook (Instagram’s parent company) outsources several thousand workers to sift through flagged content, much of it horrific. Workers, who moderate hundreds of posts per day, have little time to decide a post’s fate in light of frequently changing internal policies. On top of that, much of these outsourced workers are based in places like the Philippines and India, where they are less aware of the cultural context of what they are moderating.
The Instagram moderator may not have understood that it’s the image of the shark in connection to the claim that it won a NatGeo award that deserves the false information label.
The challenges of content moderation at scale are well documented, and this shark tale joins countless others in a sea of content moderation mishaps. Indeed, this case study reflects Instagram’s own challenged content moderation model: to move fast and moderate things. Even if it means moderating the wrong things.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: content moderation, fact checking, memes
Companies: instagram
Reader Comments
The First Word
“It's the label that's wrong
The image is not false, it's manipulated. The issue here is that the "false" fact checking label is appropriate for fact checks on text eg "Trump won the election" but not for images. If they had labeled it "This image is not a real photograph" then the problem around implying the the text on the mage is false would not apply.
Can't stay to chat longer, I have to go and photoshop political some comments onto shark images ...
Subscribe: RSS
View by: Time | Thread
Fact-checker baiting
This is a great example of somebody successfully baiting "fact-checkers". I'm tempted to sympathize with the troll who posted it.
The larger problem is that "fact checkers" can't really check most facts in any meaningful, objective way.
Just to use this image as an example, how is "women are funny" even a "fact" that can be true or false? Obviously some women are more "funny" than others (even for the many different meanings of the word "funny"). I don't think it's possible to rule such a statement as clearly true or false in the first place, nor should anyone try.
Are women, as a class, "funny"? Whatever answer you give, it's an opinion, not a fact. And an opinion on a awfully vague and ill-defined question.
I've little patience with "fact checking" in general, except perhaps in the original context of internal checking within a publication, before printing a story. Most statements aren't clearly even "facts" in the first place.
[ link to this | view in chronology ]
Re: Fact-checker baiting
...you...get that the point of this story is that the fact-checkers were not intentionally evaluating whether or not the statement "women are funny" was factually true or false, right?
They were evaluating whether or not the image of the shark was National Geographic's Picture of the Year. Which, notwithstanding your handwringing, is a statement which can be objectively evaluated as true or false. It's false. That's not an opinion, it's a fact. The fact-checkers checked it; they evaluated it as false. The reason they evaluated it as false is that it is false.
The only person in this story trying to evaluate "women are funny" as a factual statement is the troll who submitted the altered image. Who *checks notes* you just said you sympathzie with.
[ link to this | view in chronology ]
Re: Re: Fact-checker baiting -- NO, it's Techdirt pretense!
The actual purpose of the story HERE is to defecate all over the place so that "moderation" looks tough, while in practice, Masnick's position is a flat and unqualified RIGHT to arbitrarily censor:
"And, I think it's fairly important to state that these platforms have their own First Amendment rights, which allow them to deny service to anyone."
https://www.techdirt.com/articles/20170825/01300738081/nazis-internet-policing-content -free-speech.shtml
You're short-sighted and WRONG as usual, "Thad, the Ant-Slayer".
[ link to this | view in chronology ]
Re: Re: Re:
How's that Trump re-election campaign coming along, bro?
[ link to this | view in chronology ]
Re: Re: Re: Fact-checker baiting -- NO, it's Techdirt pretense!
[Projects facts not in evidence]
[ link to this | view in chronology ]
Re: Re: Re: Fact-checker baiting -- NO, it's Techdirt pretense!
"The actual purpose of the story HERE is to defecate all over the place..."
I don't think the Copia Institute needs your help to shit all over a forum. God knows, you've been seriously incontinent around here for many years now.
So...did you have anything actually relevant to the OP you wanted to whine about or is it just more venting your spleen because Geigner called you a nasty name nine years ago and you've spent every waking moment since to demonstrate the truth of his statement?
[ link to this | view in chronology ]
Techdirt examples abstruse edge cases to build cred, while...
... Masnick's actual bottom line position is that corporations have total arbitrary control:
"And, I think it's fairly important to state that these platforms have their own First Amendment rights, which allow them to deny service to anyone."
https://www.techdirt.com/articles/20170825/01300738081/nazis-internet-policing-content -free-speech.shtml
[ link to this | view in chronology ]
Re: Techdirt examples abstruse edge cases to build cred, while..
New readers, if any: don't be fooled by Masnick's "it's so hard to do the right thing" diversions such as this (paid for by Silicon Valley corporate "support" of his laughable "think tank").
The block quote is one of the few times that he's been honest. Masnick believes corporations should have TOTAL ARBITRARY CONTROL without regard to The Public interest.
[ link to this | view in chronology ]
Re: Re: Techdirt examples abstruse edge cases to build cred, whi
Calling utter bullshit on that assertion. Where's your proof?
[ link to this | view in chronology ]
Re: Re: Techdirt examples abstruse edge cases to build cred, whi
"Masnick believes corporations should have TOTAL ARBITRARY CONTROL without regard to The Public interest."
Obviously corporations should have control over who they allow on their own property. That you personally believe the legal concept of property should be abolished in favor of "The People" is, rather, the more deranged idea.
But hey, if you actually want to make that happen then here's how;
1) Assemble a political party.
2) Sell 51% of the voters on the communist manifesto you keep taking your ideas from.
3) Win all the elections and rewrite the constitutional amendments preventing the government from nationalizing any sufficiently popular property.
Because the only thing you accomplish here is to make people occasionally laugh at your incomprehensible hysterics. The sum of your thousands of hours of labor remains that people flag your comment after beating you over the head with the latest sack of garbage you spilled over the forum.
That's fucking sad, Baghdad Bob, and if you weren't such a malicious mentally disabled person we'd all be inclined to show you some sympathy.
[ link to this | view in chronology ]
For what reason should the government have the right to make any interactive web service host all legally protected speech, even if the owners/operators of that service don’t want to host certain kinds of speech?
[ link to this | view in chronology ]
Re:
"For what reason should the government have the right to make any interactive web service host all legally protected speech, even if the owners/operators of that service don’t want to host certain kinds of speech?"
If Baghdad Bob - or Koby, for that matter - had any honesty at all they'd just provide the answer. The argument is clearly outlined in both The Communist Manifesto and Mao's little red book.
Admitting they're quoting outright communist philosophy doesn't fit their narrative, of course.
[ link to this | view in chronology ]
Because... don't actually click through or anything.
[ link to this | view in chronology ]
Misogynistic, really??
Branding this meme as misogynistic is an exaggeration. Sure, it's fashionable to interpret any situation in a positive light for women, and a negative light for men, whenever possible (hence the only groans on last Saturday's SNL were when Dave Chappelle dared to make a joke at the expense of women).
If the person who posted the meme had misogynistic intent, it would have read, "women are smart" or "women are strong" or something of that nature. But "women are funny" can be interpreted itself either as a positive or a negative statement about women. For instance, it could be perceived as meaning "women are strange," or "women can be good comedians."
And even if it was intended to be interpreted (in light of the fact-checking) as, "It is false that women can be good comedians," that hardly rises to the level of misogyny, which is defined as 'hatred or mistrust of women.'
If you disagree with me on that, I presume, in that case, that you'd agree that the majority of gender-related articles in the media these days are misandristic.
[ link to this | view in chronology ]
Re: Misogynistic, really??
That's a whole lot of words to say "not all men".
[ link to this | view in chronology ]
Re: Re: Misogynistic, really??
That's why it's important to campaign for futanari rights.
[ link to this | view in chronology ]
I thought the point of fact checking was to, you know, actually check facts. Not blindly tag stuff based on assumptions.
[ link to this | view in chronology ]
Mods are so funny. AI mods even more so.
[ link to this | view in chronology ]
It's the label that's wrong
The image is not false, it's manipulated. The issue here is that the "false" fact checking label is appropriate for fact checks on text eg "Trump won the election" but not for images. If they had labeled it "This image is not a real photograph" then the problem around implying the the text on the mage is false would not apply.
Can't stay to chat longer, I have to go and photoshop political some comments onto shark images ...
[ link to this | view in chronology ]
Everyone here, understands
That this is going to get WORSE.
That every Pic/video/audio file is going to need a registry.
This is going to bae as bad, as comparing Music, and its LIKE' this song. and sue everyone around it.
[ link to this | view in chronology ]
Ha ha, this is awesome.
These types of trolls are doing the yeoman's work of triggering Leftist sissies. Since Techdirt-type anti-Americans have decided to go all in on doing 'fact checks' to ensure no degenerate gets their fee-fees hurt, I'm glad to see trolls like these keep making Thought Police continue, over and over again, to publicly and embarrassingly step on their dicks.
See also: 'Islam is Right about Women'; the OK hand signal; clovergender; etc. It's damn beautiful. Leftists just can't help themselves.
Techdirt-type Leftists, please keep humiliating yourselves! It brings Americans such joy.
[ link to this | view in chronology ]
Re: Ha ha, this is awesome.
Mhm. Not sure who sounds "triggered" here. But hey have fun on Fantasy Island.
[ link to this | view in chronology ]
Case Study: Dr Phil is not a real doctor
At least a social network had some chance of eventually correcting it's Fact Check label or lack thereof.
Step 1. User flags image
Step 2. AI triages image
Step 3. Professional makes a ruling on image
Step 4. User unflags image
Step 5. Go to Step 2.
Meanwhile, Dr Phil is not a real doctor. Pass it on.
[ link to this | view in chronology ]