The Repeated Failure Of The US And UK Governments' 'Add More Hay' Approach To Surveillance
from the sipping-from-a-firehose dept
Recently we wrote about a UK Parliamentary report absolving the UK spy agencies of any responsibility for the failure to stop the killing of a British soldier last year. Significantly, one explanation given for the fact that the UK's MI5 undervalued the threat, despite investigating the men responsible several times, was that it has several thousand suspects under surveillance at any one time, and so it was beyond its capabilities to follow all leads thoroughly.
Of course, that is a consequence of the "needle in a haystack" approach that the US and UK agencies have adopted: collect as much information as possible in the hope that somehow it will be possible to sift through all the irrelevant hay to find the needle. But as an important piece by Coleen Rowley in the Guardian points out, this is not the first time that a "failure to connect the dots" from information to hand resulted in missed opportunities to stop attacks:
as an FBI whistleblower and witness for several US official inquiries into 9/11 intelligence failures, I fear that terrorists will succeed in carrying out future attacks -- not despite the massive collect-it-all, dragnet approach to intelligence implemented since 9/11, but because of it. This approach has made terrorist activity more difficult to spot and prevent.
She reminds us:
The common refrain back then was that, pre 9/11, intelligence had been flowing so fast and furiously, it was like a fire hose, "and you can’t get a sip from a fire hose". Intelligence such as the Phoenix memo -- which warned in July 2001 that terrorist suspects had been in flight schools and urgently requested further investigation -- went unread.
She details other instances of recent intelligence failures where having too much information meant that key leads were buried, and opportunities to stop attacks lost. In other words, US and UK agencies have over ten years of bad experiences with the "collect it all" and "needle in a haystack" approach to intelligence gathering, and yet it remains their principal response to successive attacks and continuing failures. Rowley concludes:
After Edward Snowden described just how massive and irrelevant the US and UK monitoring had become, people started to grasp the significance of the saying: "If you’re looking for a needle in a haystack, how does it help to add hay?"
Of course, she's not the first person to make this observation -- for example, Techdirt pointed this out over a year ago. But as someone who has worked for the FBI, and been privy to many secret details of surveillance operations, Rowley speaks with a special authority. Her article in the Guardian is well-worth reading -- especially by all those who continue to advocate the disproportionate and ineffective "add more hay" approach to keeping the public safe.
Follow me @glynmoody on Twitter or identi.ca, and +glynmoody on Google+
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: gchq, haystacks, information overload, needles, nsa, surveillance
Reader Comments
Subscribe: RSS
View by: Time | Thread
[ link to this | view in chronology ]
Another point of view
You can then spend the rest of your career looking through the haystack - as long as you are "working on it", you are doing your job.
The odds of finding are something just less than zero divided by infinity - but that's totally irrelevant. A high paying job has been saved, a massively expensive data center has been justified, and the Bureaucrats love you.
No matter what happens, you can find the data to prove that you knew it would.
uh.. wait..somethings wrong there...
[ link to this | view in chronology ]
Re: Another point of view
You have made one false assumption, that the needle is actually in a hay stack when it might be in a pile of leaves. The spy agencies are fixated on electronic communications, while any sensible terrorist will be using mail or courier, and cash.
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Needle analogy
But really, what's more difficult: finding a needle in a haystack or finding a needle in Nebraska?
[ link to this | view in chronology ]
Clandestine isn't silent and black, only as quite as the background noise.
To paraphrase Sherlock Holmes in The Sign of Four, finding a needle in a pile of identical needles.
Part of the problem is not just that there's tons and tons and tons of intelligence, but that there's bunches of background noise that is identical to that intelligence.
And given now that the people of the US are paranoid of its government and talking angrily of revolution, the background noise presents false positives even more.
But yeah, given how much has been invested into dragnet surveillance and given its poor record of actually providing useful data successfully filtered out, they're going to have to continue to expand the purpose of the program in order to justify the expense, disregarding the damage it does to the state's reputation, and disregarding the rights upon which the program infringes.
[ link to this | view in chronology ]
[ link to this | view in chronology ]
911 - failure to recognize hijacked planes
This information can be found at your local laundromat.
[ link to this | view in chronology ]
Re: 911 - failure to recognize hijacked planes
[ link to this | view in chronology ]
The 911 planes and the military.
The problem with shooting down the planes is that doing so would take direct responsibility for ending the lives of hundreds of civilians, and even a general has difficulty making a decision like that, even when doing so would allegedly save the lives of magnitudes more.
We were more used to the plane hijackings of the 70s which involved landing and exchanging hostages. To be fair, due to a certain complacency and lack of imagination, we didn't anticipate the planes being used as bombs, themselves, so we had a playbook that didn't apply to the 9/11 attacks.
On the other hand, they've essentially ruined it for future attacks with airplanes. So long as 9/11 is fresh in our minds, passengers are willing to bum rush anyone who remotely looks like a terrorist, and generals now would shoot down a plane and label as "heroes" and "patriots" all the passengers sacrificed, rather than see another airplane fuel tank be used as a guided bomb.
[ link to this | view in chronology ]
Re: The 911 planes and the military.
Meanwhile, those same passengers are having their nail trimmers confiscated to ensure they are as unprepared to do so as possible. After all, the less able the people are to protect themselves, the more they have to depend on bigger government to do it for them.
[ link to this | view in chronology ]
Security Theater.
But we do love our bear patrols and tiger-repellant rocks
[ link to this | view in chronology ]
Re: The 911 planes and the military.
Perhaps I'm just too cynical, but based on on the military operations over the past couple of decades, I thought that generals couldn't care less about civilian deaths except insofar as they create a PR problem.
"we didn't anticipate the planes being used as bombs, themselves"
If by "we" you mean the military side of the federal government, then this was totally anticipated. It was even a wargame scenario.
[ link to this | view in chronology ]
Hugeness
If by "we" you mean the military side of the federal government, then this was totally anticipated. It was even a wargame scenario.
The military side of the federal government is huge, and totally does not engage in any kind of collective groupthink. So yeah, by we I meant our air-traffic control sector and the specific military subdivisions that connect with them to identify and respond to airborne threats.
I'm sure that we had theorists that imagined large-scale kamikaze attacks, but their notions never trickled down to the agents in charge of responding to realworld events.
Now that I think of it, I suspect that the current intelligence / surveillance sector suffers from the same problem now. It's huge and it takes a ton of people to analyze a given bit of intel, determine whether it is legitimate and relevant and process its meaning.
[ link to this | view in chronology ]
Re: Hugeness
True. I should clarify: I meant in the upper echelons of the military structure.
[ link to this | view in chronology ]
Re: 911 - failure to recognize hijacked planes
http://www.historycommons.org/context.jsp?item=a930langleylaunch
The alert fighters learn of 9/11 by phone call from fiance and not from a tower alert?
Yet before 9/11 occurred there were alerts on these terrorists, including one taking flight training stopped and ticketed for no drivers license in Florida. There is so much here that was missed even though they had ample warning. One agency not sharing info with another does not improve sorting though the haystack.
[ link to this | view in chronology ]
Back in the old days...
[ link to this | view in chronology ]
Re: Back in the old days...
[ link to this | view in chronology ]
It is all about their image
They happily spend taxpayer money like a drunken sailor on shore leave.
Thanks to them the various government agencies are spying on everything and everyone while proclaiming they have special secret laws that allow them to completely ignore the Constitution.
Politicians sworn to uphold and defend the Constitution stand by and do nothing.
Meanwhile everyone become less safe as the spy agencies drown in data they cannot possibly process.
Lobbyists: Their livelihood is tied to getting politicians to make laws and award contracts that favor their particular industry. They are under no obligation whatsoever to tell the truth... so they whisper terrifying lies into the ears of the politicians who in turn recoil in horror at the prospect of not appearing proactive in the fight against terror.
Thus more bad laws are created and wasteful contracts are awarded.
Corporations: The companies aligned towards information gathering and processing get huge contracts to help in the fight against the elusive terrorist. They have the ability to gather every form of electronic emission on the planet.
Of course there is no way to process all of that data, but never mind as the corporations are raking in billions of dollars as a result of their lobbyists appealing to the politicians to do something before another disaster strikes (not to mention the hefty campaign contributions that are handed out to those same politicians to "help" them reach the proper decisions).
American citizens are the only ones not benefiting from all of these idiotic actions by corporations, lobbyists, and the politicians.
We are far less safe and our Constitutional rights are becoming vague memories.
It seems as if there is no sane or honest person left in politics. If some lobbying group decided to pay politicians to create a law making Bozo the clown the next Czar against terrorists, you can bet it would get done.
Politicians don't need any common sense, knowledge about the laws they are asked to create, or oversight of the intelligence agencies monsters they have created. They just need to appear to do something against terror (and to collect the campaign contributions they really care about).
Nothing else matters...
[ link to this | view in chronology ]
The Cover-Your-Own-Ass Society
[ link to this | view in chronology ]
Re: It is all about their image
Sure they are! You've got massive spending on security theatre, patriotic whistleblowers running for their lives, seventeen overlapping anti-terror spy agencies, and you're starting to look as bad as the Soviets and ChiComs (and Savak and Stasi) on their worst day. You've got cops harassing people walking down the street with their hands in their pockets. Your military has never been as potentially destructive, and you're almost constantly at war with somebody. The war on drugs has never been as fever pitched. Prohibition was a drop in the bucket compared to what you have now.
American (USA) citizens are the only ones who can stop this too, as well being the ones to blame for it all. Apparently, they don't care to. We get the gov't we deserve.
If that's not entertainment, I don't know what is.
[ link to this | view in chronology ]
Re: Re: It is all about their image
You're going to love one of the upcoming articles then, but to sum it up, despite overwhelming bi-partisan support, an amendment to a bill, designed to reign in the NSA was shelved by the House leadership and Congress is basically making it impossible to add it back in.
When a handful or less can override the majority like that when it comes to dealing with the NSA, the idea that it's the public's fault the government is out of control gets more than a little unbelievable.
The public told their reps, 'Do something about the NSA', the reps attempted to do something, only to have their efforts removed and blocked by a group of NSA supporters in the right/wrong(depending on which side you're on) positions.
[ link to this | view in chronology ]
Re: Re: Re: It is all about their image
Can parties in the US impeach their own caucus leaders? Wouldn't that be fun to watch?
[ link to this | view in chronology ]
The government we deserve.
A cat is faster than the mouse, deadlier than the mouse and can hear, see and smell better than the mouse. The mouse as an individual has no advantages on the cat. (Get as a species, the mouse wins because it by far outbreeds the cat. There is much more mouse-mass on earth than there is cat-mass.)
Democracy isn't the government we deserve it's the best form of government we could come up with so far, that distributes power the most widely without completely collapsing. American democracy is a democracy after it's festered to corruption beyond reconciliation or reform (reform isn't impossible, but an improvement rate that outpaces the rate of corruption isn't likely) so ultimately the US is going to end in a lot of blood.
You can't tell us this is the government we deserve anymore than you can tell the serfs that Joffrey is the king you deserve. (Or John Lackland, if you're looking for a historical example).
We've suffered enough to know that centralized power sucks. And we've bled enough that I think we've earned our utopian regime many times over. But nature doesn't care how much blood is shed. Good governance is not purchased, but rather it is invented and refined. And this takes time, and is hindered by corruption and government failure.
To say that we deserve the government we have is to condemn us as a species: occasionally we rise to the grace of rationalism, but most of the time we are victims of our own fear and greed, and for that crime, we are likely to die out having never colonized other worlds, let alone the stars so that our species could survive the long game.
[ link to this | view in chronology ]
Re: The government we deserve.
Tom & Jerry taught us that cats can't fit into mouseholes. However, I agree, but the bolded part is what's important. Mice by nature don't value the individual. They choose a different path, the same one as fish in a school. Cats aren't pack animals. They're individual predators. Different strokes, for different folks, and it's worked out fairly well for both. They value different outcomes based on their nature.
Not true. I'm sure we can do better than our present iteration of democracy. *It* doesn't work. Perhaps given the chance to refine it, we could improve it and make democracy work. Or, if we try hard enough perhaps something out there as yet undiscovered would be better than democracy.
If we can keep from killing us all, or getting ourselves all killed, there's still hope. We're not all stupid, we're not all lazy, and we're not all excessively greedy, and if enough of us keep our eyes on the prize, we may yet come up with a solution that works for everyone.
What we're doing now doesn't work for anyone, rich and poor, powerful or weak. I think we are capable of figuring out what we're doing wrong. We've plenty of indications in many areas. Discovering our errors, then successfully implementing the fixed solution won't be easy, but that's what needs to be done.
I'll start it off. Authoritarianism doesn't work. We're too much like cats. Communism and socialism don't work for the same reason. Elected representational government doesn't work for the same reason. Free market economies don't appear to work, or we as a species can't figure out how to make them work, or we don't want them to work.
Where do we go from there?
[ link to this | view in chronology ]
The capable mouse.
Some of us can consider better methods than our present iteration of democracy. I think upgrading our electorate system to something other than first-past-the-post might make for a valuable improvement.
But good luck getting that implemented on a national scale here in the US.
Imagine tweaks on the US system all you want, you're not going to affect them without being a large corporation who can buy off both sides of the party line. Telling us that we get the government we deserve is telling us it's our fault for not being that.
Let's reverse the notion: Can we not, then, blame you for failing to manage the cats? Obviously, you are a capable mouse. Fix our government.
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Collect ALL the haystacks!
Even earlier ;-)
[ link to this | view in chronology ]
Weak analogy but let's run with it
Of course, this is a highly unreliable process for fighting terrorism. Again, the analogy might be finding a particular style needle that is only made one or two times, then tossed into the haystack where there are huge numbers of other needles. Even if you can suck every needle out, the odd needles look a lot like the regular ones, and now you are back to manual labor to identify them among all the others. By the time you have a process for finding that odd needle, your process is useless. Probably. But you still have to keep checking, just in case.
However, the power to automatically sift through enormous amounts of information is a highly efficient and relatively reliable method for targeting large numbers of "enemies of the state", since now we're looking for large batches of similar needles and we can "tune" our magnet accordingly.
It is completely naive to think that any established power base, whether in Syria, China, or a "democratic" country can resist a deepening addiction to this drug of information abuse once it is available. The idea of a responsible surveillance state would be as ludicrous as a 'Crystal meth, use it responsibly' slogan in fighting drug abuse.
[ link to this | view in chronology ]
In spookspeak there's a term called "Light Cover"
Such a guy serves two functions.
One is, as a magnet for those civilians who need to contact the CIA. They would get voluntary intel all the time. The CIA doesn't put out its phone number, exactly, since it's not really an investigative service in the way the FBI is. But in those cases that someone wants to help things along, he's the guy you contact.
The other is to distract his rivals while the real operators do their work. He acts suspicious as hell, drives around, walks in the park, may even plant a false dead drop somewhere while picking up dry-cleaning and collecting mail from secondary postboxes. In the meantime, all the tails are following him impatiently looking for an activity for which they can arrest him.
"Light Cover" agents worked to throw off the other side. Both sides used them. They were very annoying to the tails, but you had to follow him with enough guys that you don't lose him.
[ link to this | view in chronology ]
[ link to this | view in chronology ]
i think the answer is straight forward enough. these 'intelligence' agencies dont want to catch terrorists. what they want to do is blanket surveillance on citizens, more than anything, so as to know what the public is going to do in retaliation for the next bad law the government tries to bring in, or the next civil right it wants to remove (so as to keep the surveillance going!) and if, whilst doing all this surveillance stuff, they happen to come across, purely by luck, information relating to terrorists or terrorist plots, then they have bonuses that can be used when spinning the next crock of bullshit to the ever willing politicians who give the go ahead, after getting some 'encouragement' on the way!! with a setup like that, who needs terrorists anyway?
[ link to this | view in chronology ]
Responses were varied, well actually two variables. One agency sent a 47 page document that describes what cows eat. The rest sent "No documents are responsive to your request" notices.
[ link to this | view in chronology ]
As someone who once tossed the car keys in the trash while cleaning out the car at the car wash. I can assure you it's a lot easier to find something, if 'Intelligence' is the thing you use to limit your search to only the places it's likely to be found.
[ link to this | view in chronology ]