GCHQ's Karma Police: Tracking And Profiling Every Web User, Every Website
from the this-is-what-you'll-get,-when-you-mess-with-us dept
One of the very first revelations from the Snowden leaks was a GCHQ program modestly entitled "Mastering the Internet." It was actually quite a good name, since it involved spying on vast swathes of the world's online activity by tapping into the many fiber optic cables carrying Internet traffic that entered and left the UK. The scale of the operation was colossal: the original Guardian article spoke of a theoretical intake of 21 petabytes every day. As the Guardian put it:
For the 2 billion users of the world wide web, Tempora represents a window on to their everyday lives, sucking up every form of communication from the fibre-optic cables that ring the world.
But the big question was: what exactly did GCHQ do with that huge amount of information? Two years later, we finally know, thanks to a new article in The Intercept, which provides details of another major GCHQ program called "Karma Police" -- the name of a song by Radiohead, with the repeated line "This is what you'll get, when you mess with us". A GCHQ document obtained by Snowden indicates that Karma Police goes back some years -- at least to 2008. It provides the following summary of the project's aims:
KARMA POLICE aims to correlate every user visible to passive SIGINT [signals intelligence] with every website they visit, hence providing either (a) a web browsing profile for every visible user on the internet, or (b) a user profile for every visible website on the internet.
Profiling every (visible) user, and every (visible) website seems insanely ambitious, especially back in 2008 when computer speeds and storage capacities were far lower than today. But the information that emerges from the new documents published by The Intercept suggests GCHQ really meant it -- and probably achieved it.
As of 2012, GCHQ was storing about 50 billion metadata records about online communications and Web browsing activity every day, with plans in place to boost capacity to 100 billion daily by the end of that year. The agency, under cover of secrecy, was working to create what it said would soon be the biggest government surveillance system anywhere in the world.
That's around 36 trillion metadata records gathered in 2012 alone -- and it's probably even higher now. As Techdirt has covered previously, intelligence agencies like to say this is "just" metadata -- skating over the fact that metadata is actually much more revealing than traditional content because it is much easier to combine and analyze. An important document released by The Intercept with this story tells us exactly what GCHQ considers to be metadata, and what it says is content. It's called the "Content-Metadata Matrix," and reveals that as far as GCHQ is concerned, "authentication data to a communcations service: login ID, userid, password" are all considered to be metadata, which means GCHQ believes it can legally swipe and store them. Of course, intercepting your login credentials is a good example of why GCHQ's line that it's "only metadata" is ridiculous: doing so gives them access to everything you have and do on that service.
Login ID, userid and password all considered to be "metadata"
The trillions of metadata records are stored in a huge repository called "Black Hole." In August 2009, 41 percent of Black Hole's holdings concerned web browsing histories. The rest included a wide range of other online services: email, instant messenger records, search engine queries, social media, and data about the use of tools providing anonymity online. GCHQ has developed software to analyze these other kinds of metadata in various ways:
SOCIAL ANTHROPOID, which is used to analyze metadata on emails, instant messenger chats, social media connections and conversations, plus “telephony” metadata about phone calls, cell phone locations, text and multimedia messages; MEMORY HOLE, which logs queries entered into search engines and associates each search with an IP address; MARBLED GECKO, which sifts through details about searches people have entered into Google Maps and Google Earth; and INFINITE MONKEYS, which analyzes data about the usage of online bulletin boards and forums.
In order to connect these different kinds of Internet activity with individuals, GCHQ makes great use of information stored in cookies:
A top-secret GCHQ document from March 2009 reveals the agency has targeted a range of popular websites as part of an effort to covertly collect cookies on a massive scale. It shows a sample search in which the agency was extracting data from cookies containing information about people's visits to the adult website YouPorn, search engines Yahoo and Google, and the Reuters news website.
Clearly the above activities allow incredibly-detailed pictures of an individual's online activities to be built up, not least their porn-viewing habits. One tool designed to "provide a near real-time diarisation of any IP address" is called, rather appropriately, Samuel Pepys, after the famous 17th-century English diarist.
Other websites listed as "sources" of cookies in the 2009 document are Hotmail, YouTube, Facebook, Reddit, WordPress, Amazon, and sites operated by the broadcasters CNN, BBC, and the U.K.'s Channel 4.
The extraordinary scale of GCHQ's spying on "every visible user" raises key questions about its legality. According to The Intercept story:
In 2010, GCHQ noted that what amounted to "25 percent of all Internet traffic" was transiting the U.K. through some 1,600 different cables. The agency said that it could "survey the majority of the 1,600" and "select the most valuable to switch into our processing systems."
Much of that traffic will be from UK citizens when they access global services like Google or Facebook, which GCHQ has admitted it defines as "external platforms," and which is thus completely stripped of what few safeguards UK law offers against this kind of intrusive surveillance by GCHQ.
This means that it is certain that many -- perhaps millions -- of UK citizens have been profiled by GCHQ using these newly-revealed programs, without any kind of warrant or authorization being given or even sought. The information stored in the Black Hole respository, and analyzed with tools like Samuel Pepys, provides unprecedented insights into the minutiae of their daily lives -- which websites they visit, which search terms they enter, who they contact by email or message on social networks. Within that material, there is likely to be a host of intimate facts that could prove highly damaging to the individual's career or relationships if revealed -- perfect blackmail material, in other words. Thanks to other Snowden documents, we know that the NSA had plans to use this kind of information in precisely this way. It would be naive to think it would never be used domestically, too.
It's frustrating that it has taken over two years for these latest GCHQ documents to be published, since they reveal that the scale of British online surveillance and analysis is even worse than the first Snowden documents indicated, bad as they were. They prove that the current calls for additional spying powers in the Snooper's Charter are even more outrageous than we thought, since the UK authorities already track and store British citizens' online moves in great detail.
When Edward Snowden handed over his amazing trove of documents to journalists to release as they thought best, he also placed a huge responsibility on their shoulders to do so as expeditiously as possible. If, as seems likely, there are yet more important revelations about the scale of US and UK spying to come, it is imperative that they are published as soon as possible to help the fight against those countries' continuing attempts to bolster mass surveillance and weaken our freedoms.
Follow me @glynmoody on Twitter or identi.ca, and +glynmoody on Google+
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: collect it all, gchq, karma police, logins, mass surveillance, metadata, passwords, social media, surveillance, uk
Reader Comments
Subscribe: RSS
View by: Time | Thread
This is what I have been saying for several years now
This is exactly why this kind of surveillance needs to stop. It is only a matter of time where the party in charge of the government will have enough info on the other party to keep them from making a serious run for their office. All political parties should oppose this kind of snooping but they don't. They all think they will be the one in charge when the music stops.
[ link to this | view in chronology ]
Re: This is what I have been saying for several years now
They fuck dead pigs and think its completly normal ffs.
[ link to this | view in chronology ]
Re: Re: This is what I have been saying for several years now
[ link to this | view in chronology ]
Re: Re: Re: This is what I have been saying for several years now
The pig thing is something that hasnt been an everyday thing.
Wasnt there an investigation into some late-night parties they had which involved some very underage prostitutes?
[ link to this | view in chronology ]
Re: This is what I have been saying for several years now
This is exactly the same as wanting back doors to encryption, they think that only the "good guys" will have access, and that they will only use it for "good reasons." Neither of those suppositions is true.
[ link to this | view in chronology ]
Re: Re: This is what I have been saying for several years now
It's pretty much a given that it's already been compromised, the only question is how much and by who?
[ link to this | view in chronology ]
Re: This is what I have been saying for several years now
This is the vital fact that needs to be hammered home to those people who don't think government surveillance is such a big deal, because they've done nothing wrong, and terrorists! Even if you're never directly affected by government surveillance, if it keeps going unchecked eventually it will be used in ways that completely undermine the idea of a democratic government. It's just human nature, and there are plenty of historical examples. This is actually something worth hundreds of people of people dying at the hands of terrorists, because the end result could be hundreds of millions living under a far worse form of government than we have now.
[ link to this | view in chronology ]
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Re:
I officially request that the government be forgotten
[ link to this | view in chronology ]
Re: Re:
[ link to this | view in chronology ]
Re: Re: Re:
But, i get ya, im not saying ignore the crap their doing to us
[ link to this | view in chronology ]
Insanity
This is insanity. Intelligence agencies, howling at the moon.
[ link to this | view in chronology ]
Re: Insanity
[ link to this | view in chronology ]
Re: Insanity
A member of the public who's paying attention to what's been happening? Almost certainly not.
A member of the public who's not been paying attention, and gets all their news from the government? Absolutely, after all the entire planet would be a smoking crater by now if it weren't for the brave actions of the spy agencies, as clearly demonstrated by the claims made by the very same spy agencies.
A member of one of the spy agencies? Without a doubt. Never having to worry about your budget or any pesky 'investigations', the ability to absolutely ruin anyone who speaks out against you if you care to, or even just care to hint at it(access to their accounts allows more than just monitoring them after all)... yeah, I'm sure they consider the world much improved thanks to their actions.
[ link to this | view in chronology ]
Awareness of these issues seems to have peaked and the opportunity to reform them expediently along with it.
[ link to this | view in chronology ]
Re:
Its like they think their entitled to do whatever the fuck they want with OTHER peoples lives, and expect everyone to be ok with it, and if not, they "MAKE" you "ok" with it, instead of respecting the rights and freedoms they supposedly "champion"
No choice, no consent, we are OWNED by our self appointed "betters"
I have no sympathy for their self afflicted problems.......
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
this operation can prune our unruly forest and make it the prettiest little tree farm you ever saw.
[ link to this | view in chronology ]
Our lives belong to us, not our governments
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Re:
Radiohead now officially terrorists
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
[ link to this | view in chronology ]
1984
[ link to this | view in chronology ]
Re: 1984
[ link to this | view in chronology ]
[ link to this | view in chronology ]
everyone needs to join a gang
[ link to this | view in chronology ]
Re: everyone needs to join a gang
[ link to this | view in chronology ]
Re: everyone needs to join a gang
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Those GCHQ jokers and their operation names
How fitting.
MEMORY HOLE, which logs queries entered into search engines and associates each search with an IP address
Props for the ironic 1984 reference, GCHQ.
and INFINITE MONKEYS, which analyzes data about the usage of online bulletin boards and forums.
I suggest switching the name of this operation and the name of this agency.
[ link to this | view in chronology ]
Re: Those GCHQ jokers and their operation names
It's not ironic if they consider the book a manual instead of a warning.
[ link to this | view in chronology ]
You criminal scum just keep on commenting
[ link to this | view in chronology ]
Re: You criminal scum just keep on commenting
[ link to this | view in chronology ]
I wear my tinfoil hat with pride.
Just because someone is paranoid, doesn't mean they are wrong.
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Re:
People should also be aware these programs were assisted by the NSA, who has several years experience on capturing internet traffic.
Can't wait until the story breaks on how encryption on the internet is made moot by the ghost certs these agencies use.
[ link to this | view in chronology ]
Maybe "As Expeditiously As Possible" Isn't The Best Strategy
Continuing the steady drip-drip-drip approach may be more effective in the long run. It had the advantage of driving the cycle:
1. Disclose X
2. Three-letter apparatchik denies X
3. Disclose evidence for X
4. Three-letter apparatchik admits X, but double-pinky-swears that it's only X and not Y
5. Disclose Y
6. Lather-rinse-repeat
[ link to this | view in chronology ]
Re: Maybe "As Expeditiously As Possible" Isn't The Best Strategy
As mentioned in the article, this is many times bigger than a phone metadata program, but as of yet the response has been incredibly small. Total surveillance has become pedestrian.
[ link to this | view in chronology ]
it'll be like one big tea party where we're all so nice, and nobody will bat an eye when the country we love chooses to eliminate an entire population somewhere that will have gotten between us and something we want.
tea party, yes. where the women come and go, talking of michelangelo. and of charlemagne and cortez, of course.
of course.
it'll be like heaven without the feathers.
[ link to this | view in chronology ]
Ironically, we've never had enemies worthy of this level of intrusion.
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
'after you, alphonse.'
[ link to this | view in chronology ]
Of course, all this info is shared with the 5 Eyes,
NSA: we spy only on foreigners, but we share intel with GCHQ.
GCHQ: we spy only on foreigners, but we share intel with NSA.
Sounds like "pass the hash, on steroids".
[ link to this | view in chronology ]
Re: Of course, all this info is shared with the 5 Eyes,
[ link to this | view in chronology ]
Data fog?
Bruce Schneier looks up random people on facebook. Do I need to start doing that sort of thing? I can easily schedule "wget" to make HTTP requests of sites with DNS names composed from a list of goofy words, save the cookies I get, or just send random cookies, maybe with Verizon's extra HTTP header, just to make people wonder.
Will this be effective in clogging GCHQ's rather totalitarian database? If not, how many people doing this sort of thing would clog it?
[ link to this | view in chronology ]
Re: Data fog?
The gather it all approach mainly has use in controlling the population and heading off protest and the organization of political parties that could challenge the status quo.
[ link to this | view in chronology ]
Re: Re: Data fog?
Thankyou for putting it so eloquantly
[ link to this | view in chronology ]
Re: Data fog?
[ link to this | view in chronology ]
Re: Re: Data fog?
[ link to this | view in chronology ]
https://letsencrypt.org/
Fuck GCHQ, NSA, and all other spy agencies.
[ link to this | view in chronology ]
/sprays GCHQ agents with water like a cat
[ link to this | view in chronology ]
And The Good News is:
You ARE Being Watched.
---
[ link to this | view in chronology ]
Corruption
Met senior management and dps
[ link to this | view in chronology ]