Snowden Docs Show GCHQ, MI5 To Be All Haystack, No Needle

from the collections-so-bulky-they're-practically-immobile dept

"Collect it all," they said. "You can't find needles without haystacks," they proclaimed. "The more you know," they rainbowed. All well and good, except the NSA, GCHQ, et al. appear to have far more in common with the protagonists of "Hoarding: Buried Alive" than with effective, finely-tuned terrorism-fighting machines.

New documents from the Snowden stash show the UK's intelligence agencies love piling data on top of data, but seemingly have no idea how to utilize this massive haul.

MI5 “can currently collect (whether itself or through partners …) significantly more than it is able to exploit fully,” the report warned. “This creates a real risk of ‘intelligence failure’ i.e. from the Service being unable to access potentially life-saving intelligence from data that it has already collected.”

A followup report from a month later echoed these concerns:

“There is an imbalance between collection and exploitation capabilities, resulting in a failure to make effective use of some of the intelligence collected today,” the report noted. “With the exception of the highest priority investigations, a lack of staff and tools means that investigators are presented with raw and unfiltered DIGINT data. Frequently, this material is not fully assessed because of the significant time required to review it.”

This isn't just an MI5 problem. And it's not just a bulk surveillance problem. GCHQ uses the same "data broker" -- a program called PRESTON, run by the National Technical Assistance Center, which is supposed to act as a go-between for intelligence agencies in order to prevent the siloing of data. But it doesn't work. It has prevented agencies from walling each other off, but the info firehose is still too much for agencies to handle -- even with more-targeted surveillance.

Targeted collections fare little better than the bulk collections, in terms of needle location. The following chart shows how much data goes unutilized in cases where suspects are known and targeted with individualized warrants.

From the 2009 report detailing these problems:

[I]n one six-month period, the PRESTON program had intercepted more than 5 million communications. Remarkably, 97 percent of the calls, messages, and data it had collected were found to have been “not viewed” by the authorities.

Despite this failure to fully use the collections they already have, UK intelligence agencies are asking the government to give them more, supposedly to capture the "growing range of services available to internet users." But they've already shown they can't handle the data they already collect. Piling more hay on the stacks is only going to allow more "needles" to escape the attention of analysts. One solution would be to throw more people at the problem, but even this might cause its own issues, as every intelligence agency is increasingly wary of becoming the temporary home of the next Snowden. Vetting procedures have ramped up and security clearances have been scaled back.

More judicious collections are a better answer, but intelligence agencies -- just like many of the internet users they track -- apparently suffer from Fear of Missing Out. But an examination of their current collections/analysis shows they're "missing out" already, and expanded powers/collections would do nothing to better secure the nation. In fact, these reports show the more they collect, the less they know.

Hide this

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: gchq, haystacks, information overload, mass surveillance, mi5, needles, surveillance


Reader Comments

Subscribe: RSS

View by: Time | Thread


  1. identicon
    Michael, 8 Jun 2016 @ 9:40am

    They aren't collecting everything so they can do predictive analysis, they are collecting everything so they have historical ammunition against individuals they determine they do not like in the future.

    Once you know the name of the person you do not like, it is easy to get the "evidence" out of the haystack.

    link to this | view in thread ]

  2. identicon
    Anonymous Coward, 8 Jun 2016 @ 9:53am

    Re:

    This, this, this! It was never about anything else other than this!

    Throughout history the old ploy of taking your liberty to fight a foreign enemy has been used again and again to GREAT effect!

    The law can never be a mechanism of prevention. Only retribution, which a lot of people mistake as justice. Law enforcement has no reason or requirement to do a damn thing for you when you are fucked over by a criminal.

    Real justice is a criminal being shot the fuck down by a 2nd Amendment citizen the moment they try to molest someone or something!

    link to this | view in thread ]

  3. identicon
    Anonymous Coward, 8 Jun 2016 @ 10:11am

    Re:

    Wholly agree - this is about collecting damning evidence to use against your domestic enemies, not the foreign terrorists.

    link to this | view in thread ]

  4. identicon
    Anonymous Coward, 8 Jun 2016 @ 10:33am

    Re: Re:

    The FBI's infamous 'director for life' J. Edgar Hoover kept files on all his potential adversaries, mainly for the purpose of blackmail, but also prosecution if and when the threat of blackmail failed to keep them in line.

    That's why the people who run the nation's spy agencies have the tools on hand to become the most powerful in the country, whether they choose to sit as president (like Russia's Putin) or prefer to remain as head of the agency (like Hoover).

    link to this | view in thread ]

  5. identicon
    Anonymous Coward, 8 Jun 2016 @ 11:20am

    Re: Re: Re:

    ...not just blackmail. But yes, it's pretty much business as usual at the FBI.

    See COINTELPRO - from (COunter INTELligence PROgram) was a series of covert, and at times illegal, projects conducted by the FBI aimed at surveilling, infiltrating, discrediting and disrupting domestic political organizations.

    link to this | view in thread ]

  6. identicon
    Anonymous Coward, 8 Jun 2016 @ 11:33am

    These programs were never about terrorism.

    NSA Surveillance Is about Power, Not "Safety"
    - by Edward S.

    "American Senators tell us that [we] should not worry, because this is not "surveillance," it's "data collection." They say it is done to keep you safe. They’re wrong. There is a huge difference between legal programs, legitimate spying, legitimate law enforcement — where individuals are targeted based on a reasonable, individualized suspicion — and these programs of dragnet mass surveillance that put entire populations under an all-seeing eye and save copies forever.

    These programs were never about terrorism: they're about economic spying, social control, and diplomatic manipulation. They're about power."

    link to this | view in thread ]

  7. identicon
    Anonymous Coward, 8 Jun 2016 @ 11:43am

    Re: These programs were never about terrorism.

    Wow, so if because this is not "surveillance," it's "data collection." can fly...

    Why can't we tar and feather a few of these bastards and say... it's not taring and feathering, it's congressional beautification.

    link to this | view in thread ]

  8. icon
    Uriel-238 (profile), 8 Jun 2016 @ 11:52am

    Put in Harlequin Romance Novel Terms...

    Brad the Handsome High-ranking US official fancies you, but you're married to a good man, Chaz. So Brad digs up some NSA dirt on Chaz to get him arrested. Conviction comes swift.

    Chaz goes to the nice jail. There he will stay so long as you play nice with Brad. You'll be his guest. You'll accept his gifts and entertainment, you'll let him prove that he's ten times the lover Chaz is.

    Don't cooperate and Chaz goes to the secret cells in Supermax facilities where people cease to exist. (Yes, we have those. It's where we're sending Camp Delta guys.)

    You have to be strong for Chaz, but if you resist Brad too much, you may lose Chaz. Oh the sturm und drang!

    link to this | view in thread ]

  9. icon
    Uriel-238 (profile), 8 Jun 2016 @ 11:56am

    "Congressional beautification."

    We can. But we'll have to forcibly tar and feather them first.

    Probably a good option prior to lynchings and assassinations.

    link to this | view in thread ]

  10. icon
    Adrian Cochrane (profile), 8 Jun 2016 @ 12:32pm

    SHOCKER! WOW!

    I'm never seen anything else from these leaks.

    Now if only they'd turn most their focus on breaking and investigating ISIS's shoddy encryption rather than the rest of ours, I might have a shred of respect for them.

    link to this | view in thread ]

  11. identicon
    Anonymous Coward, 8 Jun 2016 @ 12:44pm

    What I want is to see a study that shows percentages of needle to haystacks. If they have so much data left un-viewed they can still claim that somewhere in that stack is a needle. If a reliable study can be shown that there was an excess of data then maybe they will stop looking so much.

    link to this | view in thread ]

  12. identicon
    Chuck, 8 Jun 2016 @ 12:57pm

    Math, bitches

    It should be noted that the fact that they haven't viewed 97% of 5 million records also indicates they HAVE viewed 3% of them, which is 150,000 records.

    The thing is, let's say they're all text messages. We're talking no more than 140 characters each. Even reading at first grade level, most people can read a single text in what, like 10 seconds?

    Now, there's 28,800 seconds in an 8 hour work day (because you know government types never work more than 8 hours. Like, ever.) That's roughly 2,500 messages per day, given several minutes for government mandated smoke breaks (because you know government types never pass up a smoke break, even if they don't, yanno, smoke.)

    At this rate ONE SINGLE FIRST GRADER could read all of the messages they collected in a 6 month period in 60 days.

    How in the hell, exactly, can multiple agencies, full of full grown adults, most if not all with a college degree, not manage to achieve better than what a lone first grade student could do?

    Math, bitches. --Chuck

    link to this | view in thread ]

  13. icon
    ECA (profile), 8 Jun 2016 @ 2:00pm

    I SAID it long ago

    The Amounts of DATA..
    The number of CHATS
    The number of Forums..
    The number of GAMES with chats..
    The ways and means to HIDE information/scrambled/encoded/what ever..

    The CIA tried this ONCE...and in 1 day, filled a 10x10 foot room 4 feet high..
    Even with an inter-relational data base and search engine..ITS ALLOT OF DATA..

    You cant search a HAYSTACK until you get a GOOD MAGNET..

    link to this | view in thread ]

  14. identicon
    Anonymous Coward, 8 Jun 2016 @ 2:53pm

    this is by design. hard to control your average citizen if you cannot blackmail them into submitting to the heel of your boot.

    link to this | view in thread ]

  15. identicon
    Anonymous Coward, 8 Jun 2016 @ 3:14pm

    Re: Math, bitches

    Actually the thing is the authorisation: They are not allowed by law to look at the rest.
    That is also why they squirm at more laws to hold them back: Laws are not stopping mass collection, it is encouraging mass selection. That some high rankers dont get it and are asking for greater piles only makes the logical issue even worse.

    The databases are also favourable for ministries. As long as the people committing heinous acts are buried there, the president can claim that the intelligence agencies had the person in their data and thus can avoid the brunt of the witch-hunt from the warhawks.

    So "collect it all" is an easy way to protect politicians and the agencies from difficult targeting and prioritizing. The front is easy for the agency as it gets an easy excuse to ask for more money and less supervision. Something even politicians can formulate in the 30 second pitch test.

    link to this | view in thread ]

  16. identicon
    FM Hilton, 8 Jun 2016 @ 5:29pm

    Lack of understanding

    This is why the NSA built a huge processing/data center out in Utah-so they can collect all the information they think they need and keep it 'in case of', which is something they haven't figured out when to apply it.

    Of course it's about power-but the data is not going to be good forever, you know..so it's useless ultimately to even dare to collect it.

    Sure, knowledge is power, but in this case it leads to total insanity of not knowing exactly why.

    Kind of like those hoarder-type people, with their stacks of newspaper all over the house-they think they're important for some reason, even though the data is old, outdated and buried under tons of the same thing.

    It just doesn't make sense anymore. So using outdated and senseless data makes them happy? Go for it.

    link to this | view in thread ]

  17. identicon
    Whatever, 8 Jun 2016 @ 5:55pm

    Had to log out again and find another IP address because Masnick keeps moderating my posts. Now the thing is, even if you did disagree with the government, which I do, you would still need to punish Snowden for endangering national interests. He chose to dump all the data for everyone to see, so all that evidence should be considered tainted. Once all of the "gotcha!" moments have been removed you can see that any "threat" this surveillance might have posed was negligible at best. The government is not going to care which Hollywood movie you're downloading, but I understand if Techdirties are a little scared that their pirate protections aren't helping them.

    Sorry, Masnick, but your savior Wyden can't help you now.

    link to this | view in thread ]

  18. icon
    Uriel-238 (profile), 8 Jun 2016 @ 7:08pm

    "punish Snowden for endangering national interests..."

    We've had this conversation?

    You mean the interests of the people, which is to stop agencies from engaging in surveillance of the masses? I think that's exactly what Snowden was trying to prevent.

    Or do you mean agency interests? Interests of elected government officials? The rare monied interests that put them there? Or do you believe that what's good for General Motors is good for America?

    I think Snowden served American interests just fine. Just not the American interests that you prefer were served.

    PS: Masnick isn't moderating you. We do. But have a flag for Techdirties.

    link to this | view in thread ]

  19. icon
    Wyrm (profile), 8 Jun 2016 @ 7:13pm

    Re: These programs were never about terrorism.

    That reminds me of France, some years back: they passed a law for the sole purpose of changing in all existing laws any instance of the word "videosurveillance" into "videoprotection".
    Not changing anything to make it fit to a "protection" purpose, just changing the word so they can pretend that any new law on videosurveillance is about "protecting the public".

    That is not anything new, just the most obvious case of "changing the words, not the actions" I've ever seen. Subtle as a 30-ton truck driving through your garden.

    link to this | view in thread ]

  20. icon
    BernardoVerda (profile), 8 Jun 2016 @ 10:14pm

    Mullah Nasruddin and the Lost Key

    Mullah Nasruddin and the Lost Keys

    Late one evening, the famous folk-tale hero and "wise fool", Sufi master Mullah Nasruddin, was on his hands and knees searching for something under a streetlamp. A passer-by saw him and asked, "What are you looking for?" "My house key," Nasruddin replied. "I lost it."

    The man joined him in looking for the key, and after some time had passed without any success, the helpful fellow asked, "Are you sure you lost it around here?" Nasruddin replied, "Ummm, no... I lost it over there, by my house."

    "Then why in God's name," the man asked, "would you ever be looking for it over here?" Nasruddin promptly explained, "because the light is so much better over here, of course."

    link to this | view in thread ]

  21. icon
    Aaron Walkhouse (profile), 9 Jun 2016 @ 1:01am

    "He chose to dump all the data for everyone to see"

    He chose to hand it all over to responsible and independent
    journalists in order to prevent your imaginary dump.

    Since then they have been dripping it out bit by bit and
    only after careful analysis to prevent potential harm.

    link to this | view in thread ]

  22. identicon
    Anonymous Coward, 9 Jun 2016 @ 5:40am

    Re:

    horse with no name just hates it when due process is enforced.

    link to this | view in thread ]

  23. icon
    Whatever (profile), 9 Jun 2016 @ 7:18am

    The assumption that nothing has improved since 2009 is laughable.

    Come on guys, nerd harder.

    link to this | view in thread ]

  24. icon
    Jeffrey Nonken (profile), 9 Jun 2016 @ 10:11am

    Loss aversion.

    We should give them all the data. All of it. Open the firehose, give then everything they ask for.

    Then next time something bad happens they didn't prevent, fire the lot of them with extreme prejudice.

    *sigh* Wish that were possible.

    link to this | view in thread ]

  25. identicon
    Anonymous Coward, 9 Jun 2016 @ 5:40pm

    Re:

    Finally decided to log back in?

    link to this | view in thread ]

  26. icon
    Whatever (profile), 10 Jun 2016 @ 11:19pm

    Re: Re:

    I log in all the time. Sadly, Techdirt is totally into censorship, holding all of my comments for moderation and making them entirely worthless.

    Techdirt uses a form of prior restraint to keep my comments out of the public eye, only adding them to the site once the article is no longer in the first couple of pages. It's funny as hell to watch them do it, it's so insanely two faced.

    link to this | view in thread ]

  27. icon
    Whatever (profile), 13 Jun 2016 @ 5:17pm

    Re: Re: Re:

    For what it's worth, my comment was submitted on the 10th,and approved late on the 13th. Quite simply, Techdirt is using a form of censorship to punish me for speaking my mind.

    Insanely two faced, I have to say!

    link to this | view in thread ]

  28. identicon
    Anonymous Coward, 14 Jun 2016 @ 6:04pm

    Re: Re: Re: Re:

    You trolls don't seem to get that if you've had a proven history of spamming and/or trolling the site, anything you say is passed through algorithms to react in kind. And now you're mad because people aren't reading your sad conspiracy theories over Wyden?

    In the same article, you switch IP addresses and spam. Just who are you kidding?

    link to this | view in thread ]


Follow Techdirt
Essential Reading
Techdirt Deals
Report this ad  |  Hide Techdirt ads
Techdirt Insider Discord

The latest chatter on the Techdirt Insider Discord channel...

Loading...
Recent Stories

This site, like most other sites on the web, uses cookies. For more information, see our privacy policy. Got it
Close

Email This

This feature is only available to registered users. Register or sign in to use it.