There Is Simply No Scientific Backing For TSA's Behavioral Detection Program
from the hearsay-isn't-evidence dept
The Government Accountability Office has taken a run at the TSA's Behavioral Detection program in the past. Its findings were far from complimentary. Specially-trained "Behavior Detection Officers" (BDOs) were basically human coin flips. Deciding whether or not someone was a threat came down to a lot of subjective readings of human behavior, rather than proven principles.
In response to this report, the TSA started trimming back the number of BDOs it deployed, converting about 500 of them back into regular TSA officers. But the TSA still believed there was something to its pseudoscience patchwork, so it's still sending out 2,600 BDOs to covertly stare at travelers' throats and eyes (no, really) until terrorism reveals itself.
The GAO's second report focuses on the TSA's stubborn insistence that the Behavior Detection program is worth what we're paying for it. It has made claims to oversight that this program is scientifically-backed and scientifically-based. The GAO's investigation [PDF] finds almost nothing that backs these assertions.
We reviewed and categorized all 178 sources that, as of April 2017, TSA cited as providing support for specific indicators in its revised list of behavioral indicators to identify the extent to which they present valid evidence. We defined valid evidence as original research that meets generally accepted research standards and presents evidence that is applicable in supporting TSA’s specific behavioral indicators.
[...]
Of the 178 total sources TSA cited, we determined that 137 are news or opinion sources, and we took no further action because these do not meet our definition of valid evidence. We determined that 21 of the 178 sources are reviews of studies— sources that rely on the source author’s assertion of support for the indicator rather than original analysis, methods, or data that can be independently used as valid evidence.
Of the 178 sources cited by the TSA (with great confidence, one imagines), only 20 met the GAO's definition of valid evidence.
The TSA claims its BDOs have the ability to sniff out terrorism by looking for behavioral indicators. There are long lists of indicators to look for but, again, the TSA has nothing in the way of evidence to back these up.
In our review of all 178 sources TSA cited in support of its revised list, we found that 98 percent (175 of 178) of the sources do not provide valid evidence applicable to the specific indicators that TSA identified them as supporting. In total, we found that TSA does not have valid evidence to support 28 of its 36 revised behavioral indicators, has one source of valid evidence to support each of 7 indicators, and has 2 sources of valid evidence to support 1 indicator.
In fact, it appears the TSA may just be Googling for indicators, as its supporting "evidence" is extremely scattershot.
For example, among other sources, TSA cited four separate news articles that provide eyewitness accounts of a single 2005 suicide bombing incident at an Israeli bus station as support for 2 behavioral indicators. In another example, TSA cited a 2004 newsletter published by the New Hampshire Police Standards and Training Council as support for 11 behavioral indicators.
A partial snapshot of the chart created by the GAO shows the utter failure of the TSA's behavioral "science."
The individual breakdown of the scientific "support" for behavioral indicators is even more damning.
As a result of our analysis, we determined that TSA does not have valid evidence supporting 28 of its 36 revised behavioral indicators. Specifically, every source TSA cited in support of these 28 indicators is a news or opinion source, a review of studies source, or an original research source that either does not meet generally accepted research standards or is not applicable to the behavioral indicator it was provided to support. For example, TSA cited 105 sources to support the use of indicator number 11, which involves BDOs identifying individuals who seem to be attempting to conceal their normal appearance. However, we found that none of the 105 sources present original research that meets generally accepted research standards. In another example, TSA cited 63 sources to support the use of indicator number 5, which involves BDOs identifying individuals who seem to be sweating heavily. While we found that one of the 63 sources cited is original research that meets generally accepted research standards, we also found that this source does not present evidence that is applicable as support for this indicator.
To which is appended this footnote:
Some reviews of studies sources TSA cited as support for behavioral indicators present conflicting information on whether the use of behavioral indicators and behavior detection more generally is useful in identifying individuals who may pose a threat to security. For example, one article states that indicators of suicide bombers “are drawn from multiple sources and have not been formally or empirically validated.” Another states that indicators used by law enforcement to identify potential suicide bombers are “vague, contradictory, and so broad as to be useless.”
The DHS responded to the GAO's draft by claiming it still had plenty of research and science justifying the deployment of BDOs. More "evidence" was submitted and found to be just as useless as everything the GAO had already examined.
[I]n its letter, DHS cited passages from a 2013 RAND report as providing support for TSA’s use of behavior detection. However, the report DHS cited does not clearly support TSA’s use of behavior detection. As DHS noted, the RAND report states that there is value and unrealized potential for using behavioral indicators as part of a system to detect attacks. However, the indicators reviewed in the RAND study could not be used in real time in an airport environment, as we reported in 2013. Further, the RAND report refers to behavioral indicators that are defined and used significantly more broadly than those used by TSA. For example, the RAND report includes indicators such as mobile device tracking, monitoring online activity, and changes in lifestyle patterns.
The DHS also said the program was justified because trained BDOs also sometimes saw things even untrained non-professionals might be able to spot.
DHS also stated that there are certain common-sense indicators that TSA cannot reasonably ignore, with or without valid evidence. Specifically, DHS highlighted the indicator “unusual exposed wires or electrical switches on a person.”
Of course, these could (MAYBE) be detected by normal screeners, not specially-trained officers costing the taxpayers millions each year. The GAO agrees BDOs should ignore common sense. But much of what the DHS considers to be "common sense indicators" are highly-subjective takes on normal human behavior.
We recognize DHS’s position that certain common-sense indicators of mal-intent should not be ignored. However, TSA’s revised list of 36 behavioral indicators also includes indicators that are subjective, such as assessing the way an individual swallows or evaluating the degree to which an individual’s eyes are open.
Finally, the DHS protests it should be able to do it because other agencies (CBP, FBI) and other countries do it. The GAO patiently responds that behavioral detection may work for others, but the point is the TSA has yet to prove it works when it does it and can offer no solid evidence to back its claims.
The TSA's Behavioral Detection program isn't dead yet, but a few more nails have been driven into the coffin. If the regular TSA screening procedures are security theater, the BDOs are the mimes. A whole lot of moving around, but in the end, there's really nothing there.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: behavioral detection, dhs, gao, research, scientific basis, security theater, studies, tsa
Reader Comments
Subscribe: RSS
View by: Time | Thread
How about pseudo scientific backing?
Having a bunch of indicators you can check off gives it an air of legitimacy.
[ link to this | view in chronology ]
I offer my services
I only request that I be recompensed $5,000 per month in hopes my struggles will make our skies 0.01% safer.
[ link to this | view in chronology ]
Re: I offer my services
[ link to this | view in chronology ]
Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.
[ link to this | view in chronology ]
Re: Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.
[ link to this | view in chronology ]
Re: Re: Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.
What's your point? "Scientific" is the word used in title, and means a method of approaching / solving problems.
SHEESH. No matter how neutral and uncontroversial of comments, on this site full of contrarians you can be sure that some weenie will try to gainsay.
[ link to this | view in chronology ]
Re: Re: Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.
What's your point? "Scientific" is the word used in title, and means a method of approaching / solving problems.
SHEESH. No matter how neutral and uncontroversial of comments, on this site full of contrarians you can be sure that some weenie will try to gainsay.
[ link to this | view in chronology ]
Re: Re: Re: Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.
I agree with your question "What's your point?"
"science is not the same as application" has no meaningful context.
"SHEESH. No matter how neutral and uncontroversial of comments,..."
Everything is uncontroversial until someone creates a controversy out of it.
"... on this site full of contrarians you can be sure that some weenie will try to gainsay."
This is the nature of social interaction. Someone usually has something to say about something. It is not bad to offer new ideas, even if they are stupid. If the ignorant appear, they become teachable moments. If knowledge is presented, then everyone has the opportunity for enrichment.
[ link to this | view in chronology ]
Re: Re: Re: Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.
[ link to this | view in chronology ]
Re: Re: Re: Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.
Scientific does, however, mean scientific. Calling something scientific when it is, in fact, not, is either a lie or extremely sloppy thinking. Or both.
It also does not help to confuse the scientific method (which is what "scientific" means here), and any products of a scientific endeavour, which may be used for good or ill, and in which the particular endeavour itself may have been a bad idea or outright nefarious to begin with.
In that sense, scientific is not necessarily a good, but is also irrelevant when the subject at hand is not in the least scientific to begin with.
Further, there is little which is scientific in the example given. If one wanted to specifically address Nazi sciences, some of them are entirely unscientific also, merely being labelled as science. Others are entirely common to any system, such as base R&D which might be applied to weapons systems, but most of the development is actually engineering issues. (But it's funny how i have never heard similar commentary for engineering as i have heard regarding science, or what someone wishes to label as science. Which is the sign of something being super trope-y and not particularly a reflection of reality.)
Controversial? I don't care to speculate. But it is highly inaccurate. And there is at least one weenie somewhere who will always support some poorly reasoned statement or line of outright bullshit, no?
[ link to this | view in chronology ]
Re: Re: Re: Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.
site full of contrarians
Pot, kettle.
The irony burns.
[ link to this | view in chronology ]
Re: Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.
That'd come from a proper high-school science class. Where you're taught how to properly put a claim or theory to the test, with control groups and whatnot to avoid false conclusions and bias. Taught how to rule out - or acknowledge - other possible explanations. Taught to make your test repeatable by other independent testers to further rule out mistakes, deception and bias.
That includes the TSA methods described above. There's no scientific backing for it. Nazis, "legalizing state actions" and corporatocracy have nothing to do with science. Technology is only vaguely related.
You complain of "training the slaves to obey authority without question." Science class is all about teaching kids to question everything. Add a method for uncovering the truth and debunking garbage claims, and yes, it's a good thing.
[ link to this | view in chronology ]
Re: Re: Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.
Well, that's what it SHOULD be about. In most cases, it's about regurgitating currently accepted "facts" no matter how ridiculous they seem when considered logically. Critical thinking is indeed NOT taught as then you have a generation of people capable of thinking for themselves, which makes them unsuitable to be cogs in the wheel.
[ link to this | view in chronology ]
Re: Re: Re: Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.
Like the so-called "theory of evolution". Anyone with half a brain can see how ridiculous that is.
[ link to this | view in chronology ]
Re: Re: Re: Re: Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.
[ link to this | view in chronology ]
Re: Re: Re: Re: Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.
[ link to this | view in chronology ]
Re: Re: Re: Re: Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Self fulfilling
[ link to this | view in chronology ]
There is!
It just depends on which science you are using. Read up on science surrounding oppressing citizens and conditioning them to accept commands from any form of authority out of habit.
The TSA is doing a fantastic job at obedience training. The people are lining up like cattle and subjecting themselves to every demand being made with little resistance.
[ link to this | view in chronology ]
Re: There is!
[ link to this | view in chronology ]
A different approach
*A ride on public transit is a low-cost thing to steal, (most systems incur near zero marginal cost per passenger until they have to increase capacity somewhere) and
* public transit is generally subsidized, either by the fare-paying public or by taxes,
How about not being too worried about a few free tickets to the enterprising or the poor who just might need them?
[ link to this | view in chronology ]
Re: A different approach
Who the hell wants to live in a place where that crap is allowed?
[ link to this | view in chronology ]
Re: A different approach
[ link to this | view in chronology ]
I can see how this would be effective. People that are up to no good and know they are being watched tend to exhibit signs. Of course, some people may just be nervous and end up being delayed a bit, but whatever. When is the last time you heard of an El Al flight having a problem?
Same with good cops, good police work involves watching people and seeing things that are outside the norm and then questioning. That is just good police work.
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Ben Gurion Airport's security teams have been doing exactly this, successfully, for decades.
The TSA should go there and learn that instead of trying to reinvent it.
[ link to this | view in chronology ]
Re:
I feel like we've been down this road before, but:
Trump Says Profiling Has Protected Israel. It Probably Wouldn’t Work Here.
[ link to this | view in chronology ]
Just train the existing profiling staff in the same way.
They already had plenty of those and could easily staff up.
They also have thousands times as many veterans available.
[ link to this | view in chronology ]
Re:
It's extremely rare for any flight to have a security problem. We keep finding out it's really easy to get guns, bombs, etc. onto US flights, but when does it ever happen? The US and Israel spend a lot on security. Is there evidence they're more secure than countries that spend less?
In lots of countries you can get on a train without any security check. When was the last time you heard of a train having a problem? All the ones I can think of were unintentional derailments.
[ link to this | view in chronology ]
Re:
This rock keeps tigers away ...
[ link to this | view in chronology ]
Re: Re:
*Or anything else the TSA does.
[ link to this | view in chronology ]
Sorry
[ link to this | view in chronology ]
Re: Sorry
[ link to this | view in chronology ]
Re: Re: Sorry
Even the ones that blew up? Oh, wait, those were the ones where the bureaucrats overrode the engineers. Never mind.
[ link to this | view in chronology ]
Snake oil
[ link to this | view in chronology ]
Brought to you by...
[ link to this | view in chronology ]
I wonder if any of their sources refer to the show as well?
[ link to this | view in chronology ]