Again, Algorithms Suck At Determining 'Bad' Content, Often To Hilarious Degrees

from the peachy dept

A few weeks back, Mike wrote a post detailing how absolutely shitty algorithms can be at determining what is "bad" or "offensive" or otherwise "undesirable" content. While his post detailed failings in algorithms judging such weighty content as war-crime investigations versus terrorist propaganda, and Nazi hate-speech versus legitimate news reporting, the central thesis in all of this is that relying on platforms to host our speech and content when those platforms employ very, very imperfect algorithms as gatekeepers is a terrible idea. And it leads to undesirable outcomes at levels far below those of Nazis and terrorism.

Take Supper Mario Broth, for instance. SMB is a site dedicated to fun and interesting information about Nintendo and its history. It's a place that fans go to learn more weird and wonderful information about the gaming company they love. The site also has a Twitter account, which was recently flagged for posting the following tweet.

For the sin of tweeting that image out, the site's entire account was flagged as "sensitive", which means anyone visiting the account was greeted with a warning about how filthy it is. What Twitter's systems thought was offensive about the image, which comes from another video from a costume company that works with Nintendo, is literally anyone's guess. Nobody seems to be able to figure it out. My working theory is that the Princess Peach's lips resemble too closely a more private part of the female anatomy and, when coupled with the flesh-colored face surrounding it sent Twitter's algorithm screaming "Aaah! Vagina!" leading to the flagging of the account. But this is just a guess, because although the "sensitive" flag was eventually removed, SMB never got any response or explanation from Twitter at all.

SMB went as far as to test through dummy accounts whether the image was the entire problem. It was. After posting the image several times from other accounts, each account was flagged within minutes of the posting. It's an algorithm doing this, in other words, and one which seems ill-suited to its task.

What we have here is two related problems. We have a company designed to let speakers speak employing an algorithm to flag offensive content, which it is doing very, very badly. We also have a company with a staff insufficiently capable to correct the errors of its incapable algorithm. This would be annoying in any context other than current reality, which sees rising calls for internet sites to automagically block "bad" content and do so with literally inhuman speed.

That means algorithms. But the algorithms can't do the job. And with sites erring on the side of over-blocking to avoid scrutiny from both the public and governments, that means open communication is the loser in all of this. It's hard to imagine an outcome more anathema to services like Twitter than that.

Hide this

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: algorithms, free speech, hate speech, moderation, propaganda


Reader Comments

Subscribe: RSS

View by: Time | Thread


  1. icon
    Ninja (profile), 10 Apr 2018 @ 11:17am

    Of course one has to question why humanity has such a problem with penises and vaginas.

    In any case I think we should stop putting content removal in the hands of specific people or algorithms. I'd like to see a decentralized effort where if people enough flagged a determined content it would issue a warning that it was severely flagged but still be visible and after a very high threshold it would be enabled for review by actual humans and possibly removed. Not a perfect system but at least better than what we have now. One can dream, no?

    link to this | view in thread ]

  2. identicon
    Anonymous Coward, 10 Apr 2018 @ 12:14pm

    What was the tweet?

    The only pictures I see on the Kotaku page are a highly-pixelated image of Princess Peach (like, 50x30 being shown at 10x that size), and a photo of the author. The Twitter photo link says "Sorry, that page doesn’t exist!". Was the original photo really that pixelated? Or was that just done when reposting, to bypass the filter?

    link to this | view in thread ]

  3. icon
    Roger Strong (profile), 10 Apr 2018 @ 12:25pm

    Re:

    Of course one has to question why humanity has such a problem with penises and vaginas.

    Sanity self-preservation. If everything about the Stormy Daniels affair and other Donald Trump sexual misconduct allegations were allowed on TV news - from detailed descriptions to infographics to computer animation - this Presidency would resemble Call of Cthulhu.

    link to this | view in thread ]

  4. icon
    Uriel-238 (profile), 10 Apr 2018 @ 12:45pm

    Penises and Vulvas

    Have you seen the new Last Week Tonight bit on CPCs? We really have genitalia issues. Big swooping raging engorged issues.

    link to this | view in thread ]

  5. identicon
    Anonymous Coward, 10 Apr 2018 @ 12:49pm

    Re:

    So, the system in use here?

    link to this | view in thread ]

  6. identicon
    Anonymous Coward, 10 Apr 2018 @ 12:52pm

    Re: What was the tweet?

    Yeah, I can't find the image either. Really hope someone saved it. I'm curious to see what it was.

    link to this | view in thread ]

  7. identicon
    Anonymous Coward, 10 Apr 2018 @ 1:17pm

    Re: Re: What was the tweet?

    link to this | view in thread ]

  8. icon
    Ninja (profile), 10 Apr 2018 @ 1:20pm

    Re: Re:

    My Pr0n Parody Sense is tingling!

    link to this | view in thread ]

  9. identicon
    Anonymous Coward, 10 Apr 2018 @ 1:24pm

    Re: Penises and Vulvas

    It's called stigmatization.

    THE goto tactic for all groups to attack another group is to take a smaller not really a big problem and then stigmatize it until its a huge throbbing.... er problem!

    This was all brought to you by the Catholic Church over the idea that naked people go to hell and clothed people are saved.

    As a Christian, I fully recognize that Christian "institutions" have caused more damage to morals and values than secular ones though they appear to be catching up so they are not outdone.

    Once a group of humans institutionalize something regardless of what that something is it becomes are desired point of infection by evil people to work towards their own ends.

    link to this | view in thread ]

  10. identicon
    Anonymous Coward, 10 Apr 2018 @ 2:08pm

    Re: Re: Re: What was the tweet?

    Nothing that loads. Just empty gray boxes.

    link to this | view in thread ]

  11. icon
    Mason Wheeler (profile), 10 Apr 2018 @ 4:04pm

    By Twits, For Twits

    So once again, the Twits at Twitter screw things up for the Twits using Twitter, while the rest of us just sit back and laugh. I cannot for the life of me understand how such an insipid service ever gained even a fraction of the popularity it's achieved...

    link to this | view in thread ]

  12. icon
    PaulT (profile), 11 Apr 2018 @ 12:03am

    Re:

    "Of course one has to question why humanity has such a problem with penises and vaginas."

    *Humanity* doesn't. Western culture seems to have some issues, North American culture more than some others, but less than some others.

    As for the rest of your comment, the problem is that there's a push to make these companies directly and even criminally liable for content that manages to appear on their site even after moderation. A system that would allow questionable material to appear even while moderated would probably fall foul of the laws those people are trying to push through. Similarly, this is why these systems are erring on the side of caution and flagging material is obviously not objectionable - their hand is being forced into making these thing over-zealous rather than let something through.

    link to this | view in thread ]

  13. identicon
    Vic, 11 Apr 2018 @ 8:30am

    Nope, no images at any links...

    link to this | view in thread ]

  14. identicon
    Vic, 11 Apr 2018 @ 8:32am

    Re:

    Oh, false alarm, sorry. Yes, there is a princess image.

    link to this | view in thread ]

  15. identicon
    Just Passin' Thru, 11 Apr 2018 @ 9:01am

    Algorithm problem???

    So, Twitter has an automatic algorithm that scans and mistakenly flags inoffensive images. When it is later reviewed by a human and deemed to be non-offensive, why isn't the image added to a database of known, vetted, inoffensive images that the algorithm checks should it ever encounter that image again?

    link to this | view in thread ]

  16. icon
    AntiFish03 (profile), 11 Apr 2018 @ 10:23am

    Honestly I have reported more than a few posts that rant about hang ing all gun owners. Or that gun owners need to be killed and have their firearms taken by force. etc... Facebook refuses to see that as hate speech but it clearly should be. Its also potentially inciting violence against a group of people whose majority is law abiding.
    I have also reported posts that are even more inflammatory and discriminating against a lot of different groups especially the NRA but others as well. FB has given me responses each time that its not hate speech and that if I don't want to see it then I should just hide it. Yet if I were to post a rant about killing anti-gun people it would be pulled nearly instantly for inciting violence etc. Their algorithm is heavily skewed. I think this more than anything is what Cruz was trying to get at. That the bias of what is allowed leans heavily to liberal view points but is exceedingly heavy handed on conservative viewpoints.

    link to this | view in thread ]

  17. icon
    That One Guy (profile), 11 Apr 2018 @ 9:23pm

    Re: Re: Re: Re: What was the tweet?

    Strange, I just checked the page again and it loaded just fine for me. Might be something on your end.

    link to this | view in thread ]

  18. icon
    PaulT (profile), 12 Apr 2018 @ 3:54am

    Re:

    Hate speech is generally defined as being against a protected class. Gun fetishist is not a protected class. It's no more hate speech than people saying that all Pokemon fans should be shot. It might be offensive, even something that authorities should be aware of, but not hate speech according to the usual guidelines.

    "That the bias of what is allowed leans heavily to liberal view points but is exceedingly heavy handed on conservative viewpoints."

    Which conservative viewpoints? Honestly held beliefs about honest conservative values, or the racist, misogynistic, homophobic, etc., viewpoints often espoused by those who claim to be conservative? There's a big difference, and if someone is blocking those viewpoints it's not anti-conservative bias if all the racist, homophobic woman haters happen to be on that side.

    Give me examples of non-hatred being blocked and I might agree. But, if the majority of hatred is on your "team", you may wish to examine that "team" rather than complaining about censorship. If 90% of objectionable material is in the hands of one "team", it's not unfair if they get 90% of the blocks.

    link to this | view in thread ]

  19. icon
    The Wanderer (profile), 12 Apr 2018 @ 5:35am

    Re: Re: Re: Re: Re: What was the tweet?

    I'm reasonably sure that the people who are reporting problems with such images not loading (properly) are using NoScript or similar, are being passive-aggressive about the fact that Websites which don't have an inherent need to be dynamic should work properly (at a basic level) without JavaScript, and are attempting to hint that giving sites which don't respect that principle a benefit by linking to them is something which should perhaps be avoided.

    I use NoScript myself, and wouldn't allow scripting from such a site just for the purpose of loading images (although I might manually edit page source in Firebug, or potentially do it less-manually with Greasemonkey, to enable myself to view a particular image) - but I haven't followed the links in question, and would have been unlikely to speak up here about the matter anyway. (Although knowing that the linked-to site does have that problem is something that's useful to me.)

    link to this | view in thread ]

  20. icon
    Uriel-238 (profile), 12 Apr 2018 @ 10:52am

    "Protected Classes"

    It's entirely off topic, though it is a problem that we have:

    Large amounts of accepted hate speech is against classes who aren't yet (or aren't quite) protected, whether countercultures, disabled, obese people, or the biggest one, poor people, even though there are clear trends of discrimination against them. (Trump's new welfare reform serving as a notable recent example.)

    It's a problem that we classify one form of bigotry over another rather than defining specific actions that are universally regarded as acts of hatred. As a result, society's level of awareness is always many steps behind.

    link to this | view in thread ]

  21. identicon
    Châu, 17 Apr 2018 @ 6:29pm

    Open Source

    If open source algorithm, easier test and find it problems.

    link to this | view in thread ]


Follow Techdirt
Essential Reading
Techdirt Deals
Report this ad  |  Hide Techdirt ads
Techdirt Insider Discord

The latest chatter on the Techdirt Insider Discord channel...

Loading...
Recent Stories

This site, like most other sites on the web, uses cookies. For more information, see our privacy policy. Got it
Close

Email This

This feature is only available to registered users. Register or sign in to use it.