Fan Uses AI Software To Lipread What Actors Really Said In TV Series Before Chinese Authorities Censored Them
from the I-saw-what-you-said-there dept
It's hardly news to Techdirt readers that China carries out censorship on a massive scale. What may be more surprising is that its censorship extends to even the most innocuous aspects of life. The ChinAI Newsletter, which provides translations by Jeff Ding of interesting texts from the world of Chinese AI, flags up one such case. It concerns a Chinese online TV series called "The Bad Kids". Here's how the site Sixth Tone describes it:
Since its first episodes were released on China's Netflix-like video platform iQiyi in mid-June, "The Bad Kids" has earned sweeping praise for its plot, cinematography, casting, dialogue, pacing, and soundtrack. It's also generated wide-ranging online discussion on human nature due to the psychology and complex motivations of its characters.
However, as the Sixth Tone article points out, the authorities required "a lot of changes" for the series to be approved. One fan of "The Bad Kids", Eury Chen, wanted to find out what exactly had been changed, and why that might be. In a blog post translated by ChinAI, Chen explained how he went about this:
Two days ago, I watched the TV series "The Bad Kids" in one go, and the plot was quite exciting. The disadvantage is that in order for the series to pass the review (of the National Radio and Television Administration), the edited sequences for episodes 11 and 12 were disrupted, even to the point that lines were modified, so that there are several places in the film where the actor's mouth movements and lines are not matched, which makes the plot confusing to people. Therefore, I tried to restore the modified lines through artificial intelligence technology, thereby restoring some of the original plot, which contained a darker truth.
The AI technology involved using Google's Facemesh package, which can track key "landmarks" on faces in images and videos. By analyzing the lip movements, it is possible to predict the sounds of a Chinese syllable. However, there is a particular problem that makes it hard to lipread Chinese using AI. There are many homophones in Chinese (similar sounds, different meanings). In order to get around this problem, Chen explored the possible sequences of Chinese characters to find the ones that best match the plot at that point. As his blog post (and the ChinAI translation) explains, this allowed him to work out why certain lines were blocked by the Chinese authorities -- turns out it was for totally petty reasons.
Perhaps more interesting than the details of this particular case, is the fact that it was possible to use AI to carry out most of the lipreading, leaving human knowledge to choose among the list of possible Chinese phrases. Most languages don't require that extra stage, since they rarely have the same number of homophones that Chinese does. Indeed, for English phrases, researchers already claimed in 2016 that their AI-based LipNet achieved "95.2% accuracy in sentence-level, overlapped speaker split task, outperforming experienced human lipreaders".
It's clear that we are fast approaching a situation where AI is able to lipread a video in any language. That is obviously a boon for the deaf or hard of hearing, but there's a serious downside. It means that soon all those millions of high-quality CCTV systems around the world will not only be able to use facial recognition software to work out who we are, but also run AI modules to lipread what we are saying.
Follow me @glynmoody on Twitter, Diaspora, or Mastodon.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: ai, censorship, china, facial recognition, lip reading
Reader Comments
Subscribe: RSS
View by: Time | Thread
AI lipreading
Masks: They're not just for Covid-19 anymore!
[ link to this | view in chronology ]
Re: AI lipreading
My thoughts exactly
[ link to this | view in chronology ]
I foresee a massive growth(in both uses of the word) in mustaches in the near future. Large, bushy, lip-covering mustaches...
[ link to this | view in chronology ]
Re:
...and a penchant for actors to speak while turned away from the camera. A renaissance of the mouth-covering veil for the ladies and the mysterious mouth-covering scarf a la "The Shadow" for the gents.
[ link to this | view in chronology ]
Re: Re:
Why bother? If they dubbed out the prohibited lines as poorly as described, the logical next step is that next time they poorly dub out the lines and apply an equally crude visual censor to the actor's mouth - the classic black bar used for nudity, or the hyper-pixelization used to obscure a face.
[ link to this | view in chronology ]
Re: Re: Re:
In some places they will probably end up dubbing everyone involved out of existence
[ link to this | view in chronology ]
Re: Re: Re:
"Why bother? If they dubbed out the prohibited lines as poorly as described..."
With the soviets you could sort of predict that the propaganda and censorship was heavyhanded to the point of absurdity. With China it's a bit of hit and miss - some of their PR stunts growing on the midden of the blunt and obvious old school copypasta of political manifestos - and some of their more subtle PR is good enough gaslighting to convince even the more sceptical for a while.
[ link to this | view in chronology ]
CCTV lip-reading.
Then it's a good thing we're all already wearing masks. Right? Right? (crickets).
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Re:
Righteeo
[ link to this | view in chronology ]
Excuse me, this article is about how lip-reading software was used to subvert a third-party's petty attempt at censorship.
Why are the comments so far about trying to prevent lip-reading from working?
[ link to this | view in chronology ]
Re:
"Why are the comments so far about trying to prevent lip-reading from working?"
More like the generally highly cynical crowd of Techdirt commenters are predicting the response of chinese censors to lip-reading.
[ link to this | view in chronology ]
Also because we're scared of surveillance
Here in the states, we've become acutely aware of how often law enforcement is trying to listen in on communication, and lip-reading AI will add to their repertoire of techniques.
[ link to this | view in chronology ]
Re: Also because we're scared of surveillance
And give them another tool for generating probable cause after they have selected a target.
[ link to this | view in chronology ]
Re: Re: Also because we're scared of surveillance
"...another tool for generating probable cause..."
Because their current "He's breathing oxygen" hasn't served them well until now?
[ link to this | view in chronology ]
Curiously petty censorship is a problem here in the states
Rare government documents that are censored for declassification twice have revealed different reviewers will black out different parts of the document. Many of our departments lack consistent standards for declassification (and it follows, for classification).
[ link to this | view in chronology ]
Re: Curiously petty censorship is a problem here in the states
Many of our departments lack consistent standards for declassification (and it follows, for classification).
Perhaps. It's also quite possible they do have consistent standards, but like pretty much all other sets of standards for moderation, different people with access to the same training and information will still interpret the same set of standards in wildly different and yet still reasonable ways.
[ link to this | view in chronology ]
Re: Re: Curiously petty censorship is a problem here in the stat
If different people with access to the same training and information are interpreting the same set of standards in wildly different and yet still reasonable ways, and is inconsistent in its application, then the standard fails to be consistent.
SEE ALSO: The holy bible.
[ link to this | view in chronology ]
Re: Re: Curiously petty censorship is a problem here in the stat
Then it isn't a standard. If it is standard, it will always have the same results.
[ link to this | view in chronology ]
BT Internet Email
Hi , Just wanted to say thanks for this fantastic article.
Website: http://btmaills.com
[ link to this | view in chronology ]
Err?
In reply, another interesting factoid, with the lipreading ai, there comes a face reading ai, and a deep fake ai. Now they do not have to catch you, they just have to modify your patterns to a blank screen, now which is real?
[ link to this | view in chronology ]
Read my lips
No new taxes!
[ link to this | view in chronology ]
Re: Read my lips
...that Nixon quote never gets old. As good an example of the old three word rhetoric trick as it gets.
[ link to this | view in chronology ]
We've had lipreading computers since 2001.
[ link to this | view in chronology ]
So the AI can do what 90% of the people posting on the internet can't?
Because it seems that absolutely nobody today knows the difference between "your" and "You're", or "its" and "it's". Hell, they don't even know the difference between "there", "their" and "they're" most of the time, or that "of" isn't a replacement for "have" when it follows a word ending in "ould".
[ link to this | view in chronology ]
Re:
Oh you would love Kingdom of Loathing's 'literacy test'...
[ link to this | view in chronology ]
Common homophones
Some advanced word-processing and copy editing software will scan for homophones and highlight the ones whose meanings seem to be inconsistent with context.
So yes, this is a thing AI can do better than humans, but still a far reach from perfectly.
Maybe as we teach kids how to talk to their personal digital assistant software (e.g. SIRI, Cortana, Google Assistant) the language will develop towards something computers can more easily understand.
[ link to this | view in chronology ]
Re: Common homophones
In the above sentence, probably 90% of today's users would have written "who's".
[ link to this | view in chronology ]
"90% of today's users would have written 'who's'"
And I would have cringed reading it. But I too sometimes get confused by the pronoun possessive rules, usually when my coffee intake has been insufficient.
[ link to this | view in chronology ]
Re:
It's even more fun when we get to common phrases. Words get substituted, showing that people (mis)using such phrases do not even understand the meaning of words, they merely parrot things.
[ link to this | view in chronology ]