GCHQ's Karma Police: Tracking And Profiling Every Web User, Every Website

from the this-is-what-you'll-get,-when-you-mess-with-us dept

Mon, Sep 28th 2015 3:47am — Glyn Moody

One of the very first revelations from the Snowden leaks was a GCHQ program modestly entitled "Mastering the Internet." It was actually quite a good name, since it involved spying on vast swathes of the world's online activity by tapping into the many fiber optic cables carrying Internet traffic that entered and left the UK. The scale of the operation was colossal: the original Guardian article spoke of a theoretical intake of 21 petabytes every day. As the Guardian put it:

For the 2 billion users of the world wide web, Tempora represents a window on to their everyday lives, sucking up every form of communication from the fibre-optic cables that ring the world.

But the big question was: what exactly did GCHQ do with that huge amount of information? Two years later, we finally know, thanks to a new article in The Intercept, which provides details of another major GCHQ program called "Karma Police" -- the name of a song by Radiohead, with the repeated line "This is what you'll get, when you mess with us". A GCHQ document obtained by Snowden indicates that Karma Police goes back some years -- at least to 2008. It provides the following summary of the project's aims: KARMA POLICE aims to correlate every user visible to passive SIGINT [signals intelligence] with every website they visit, hence providing either (a) a web browsing profile for every visible user on the internet, or (b) a user profile for every visible website on the internet.
Profiling every (visible) user, and every (visible) website seems insanely ambitious, especially back in 2008 when computer speeds and storage capacities were far lower than today. But the information that emerges from the new documents published by The Intercept suggests GCHQ really meant it -- and probably achieved it. As of 2012, GCHQ was storing about 50 billion metadata records about online communications and Web browsing activity every day, with plans in place to boost capacity to 100 billion daily by the end of that year. The agency, under cover of secrecy, was working to create what it said would soon be the biggest government surveillance system anywhere in the world.
That's around 36 trillion metadata records gathered in 2012 alone -- and it's probably even higher now. As Techdirt has covered previously, intelligence agencies like to say this is "just" metadata -- skating over the fact that metadata is actually much more revealing than traditional content because it is much easier to combine and analyze. An important document released by The Intercept with this story tells us exactly what GCHQ considers to be metadata, and what it says is content. It's called the "Content-Metadata Matrix," and reveals that as far as GCHQ is concerned, "authentication data to a communcations service: login ID, userid, password" are all considered to be metadata, which means GCHQ believes it can legally swipe and store them. Of course, intercepting your login credentials is a good example of why GCHQ's line that it's "only metadata" is ridiculous: doing so gives them access to everything you have and do on that service.

Login ID, userid and password all considered to be "metadata"

The trillions of metadata records are stored in a huge repository called "Black Hole." In August 2009, 41 percent of Black Hole's holdings concerned web browsing histories. The rest included a wide range of other online services: email, instant messenger records, search engine queries, social media, and data about the use of tools providing anonymity online. GCHQ has developed software to analyze these other kinds of metadata in various ways:

SOCIAL ANTHROPOID, which is used to analyze metadata on emails, instant messenger chats, social media connections and conversations, plus “telephony” metadata about phone calls, cell phone locations, text and multimedia messages; MEMORY HOLE, which logs queries entered into search engines and associates each search with an IP address; MARBLED GECKO, which sifts through details about searches people have entered into Google Maps and Google Earth; and INFINITE MONKEYS, which analyzes data about the usage of online bulletin boards and forums.

In order to connect these different kinds of Internet activity with individuals, GCHQ makes great use of information stored in cookies: A top-secret GCHQ document from March 2009 reveals the agency has targeted a range of popular websites as part of an effort to covertly collect cookies on a massive scale. It shows a sample search in which the agency was extracting data from cookies containing information about people's visits to the adult website YouPorn, search engines Yahoo and Google, and the Reuters news website.

Other websites listed as "sources" of cookies in the 2009 document are Hotmail, YouTube, Facebook, Reddit, WordPress, Amazon, and sites operated by the broadcasters CNN, BBC, and the U.K.'s Channel 4.
Clearly the above activities allow incredibly-detailed pictures of an individual's online activities to be built up, not least their porn-viewing habits. One tool designed to "provide a near real-time diarisation of any IP address" is called, rather appropriately, Samuel Pepys, after the famous 17th-century English diarist.

The extraordinary scale of GCHQ's spying on "every visible user" raises key questions about its legality. According to The Intercept story:

In 2010, GCHQ noted that what amounted to "25 percent of all Internet traffic" was transiting the U.K. through some 1,600 different cables. The agency said that it could "survey the majority of the 1,600" and "select the most valuable to switch into our processing systems."

Much of that traffic will be from UK citizens when they access global services like Google or Facebook, which GCHQ has admitted it defines as "external platforms," and which is thus completely stripped of what few safeguards UK law offers against this kind of intrusive surveillance by GCHQ.

This means that it is certain that many -- perhaps millions -- of UK citizens have been profiled by GCHQ using these newly-revealed programs, without any kind of warrant or authorization being given or even sought. The information stored in the Black Hole respository, and analyzed with tools like Samuel Pepys, provides unprecedented insights into the minutiae of their daily lives -- which websites they visit, which search terms they enter, who they contact by email or message on social networks. Within that material, there is likely to be a host of intimate facts that could prove highly damaging to the individual's career or relationships if revealed -- perfect blackmail material, in other words. Thanks to other Snowden documents, we know that the NSA had plans to use this kind of information in precisely this way. It would be naive to think it would never be used domestically, too.

It's frustrating that it has taken over two years for these latest GCHQ documents to be published, since they reveal that the scale of British online surveillance and analysis is even worse than the first Snowden documents indicated, bad as they were. They prove that the current calls for additional spying powers in the Snooper's Charter are even more outrageous than we thought, since the UK authorities already track and store British citizens' online moves in great detail.

When Edward Snowden handed over his amazing trove of documents to journalists to release as they thought best, he also placed a huge responsibility on their shoulders to do so as expeditiously as possible. If, as seems likely, there are yet more important revelations about the scale of US and UK spying to come, it is imperative that they are published as soon as possible to help the fight against those countries' continuing attempts to bolster mass surveillance and weaken our freedoms.

Follow me @glynmoody on Twitter or identi.ca, and +glynmoody on Google+

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: collect it all, gchq, karma police, logins, mass surveillance, metadata, passwords, social media, surveillance, uk

60 Comments

If you liked this post, you may also be interested in...

Reader Comments

Subscribe: RSS

View by: Time | Thread

Anonymous Coward, 28 Sep 2015 @ 4:12am

This is what I have been saying for several years now
there is likely to be a host of intimate facts that could prove highly damaging to the individual's career or relationships if revealed -- perfect blackmail material, in other words.

This is exactly why this kind of surveillance needs to stop. It is only a matter of time where the party in charge of the government will have enough info on the other party to keep them from making a serious run for their office. All political parties should oppose this kind of snooping but they don't. They all think they will be the one in charge when the music stops.
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 4:14am

the whole "it's not a crime when we do it" does not hold any weight when you start breaking other countries laws, instead of just your own.
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 4:17am

Re: This is what I have been saying for several years now
Its a bit naive to think that they havent blackmailed anyone yet...
They fuck dead pigs and think its completly normal ffs.
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 4:18am

I wonder if the right to be forgotten can be applied here.
[ link to this | view in thread ]
Klaus, 28 Sep 2015 @ 4:20am

Insanity
And has collecting all of this personal, supposedly private information done a damn thing? Is the world of 2015 a better place for it?

This is insanity. Intelligence agencies, howling at the moon.
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 4:28am

The true story is how little press this is getting outside of tech/privacy circles. This is orders of magnitude more invasive than phone metadata collection, but now the public treats it with a halfhearted "meh".

Awareness of these issues seems to have peaked and the opportunity to reform them expediently along with it.
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 4:28am

Way worse than Stasi.
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 4:28am

this is exactly what i've been expecting to read.
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 4:40am

blackmail is only half the story here: what about those goon squads we've created that make routine mistakes like reaching for the wrong weapon or assuming the dead man had a knife where he was scratching an itch?

this operation can prune our unruly forest and make it the prettiest little tree farm you ever saw.
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 4:45am

Re:
Wont know unless we try

I officially request that the government be forgotten
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 4:54am

I'd just like to say thanks techdirt for posting this intercept article i read a few days ago, and giving me the opportunity to say, without having to register, very well done article, intercept, not many articles go into this kind of depth/information when it comes to the disgusting surveillance entitled behaviour

Our lives belong to us, not our governments
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 5:03am

Radiohead should sue the GCHQ for copyright infringement.
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 5:04am

Re:
Also, after now just reading the techdirt article, i'd like to say, very well analyzed
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 5:07am

Mass surveillance code named karma POLICE, NOT karma TERRORISM
[ link to this | view in thread ]
Spaceman Spiff (profile), 28 Sep 2015 @ 5:11am

1984
That whirring sound you hear is George Orwell spinning in his grave!
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 5:14am

george orwell would tell you we're 31 years past 1984. in dog years that would be many centuries.
[ link to this | view in thread ]
Guardian, 28 Sep 2015 @ 5:16am

everyone needs to join a gang
y9ou need to join a gang or bike club ...perhaps when all of us belong to one then we might vote in someone whom might not want all this law and crap
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 5:18am

Dont worry folks, im sure theres somethings in the snoopers charter to make some of these illegal activities RETROACTIVELY legal......dont worry, their criminals today, but good guys tomorow, because, they say so
[ link to this | view in thread ]
Just Another Anonymous Troll, 28 Sep 2015 @ 5:21am

Those GCHQ jokers and their operation names
The trillions of metadata records are stored in a huge repository called "Black Hole".
How fitting.

MEMORY HOLE, which logs queries entered into search engines and associates each search with an IP address
Props for the ironic 1984 reference, GCHQ.

and INFINITE MONKEYS, which analyzes data about the usage of online bulletin boards and forums.
I suggest switching the name of this operation and the name of this agency.
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 5:21am

Re: Insanity
The boy who cried wolf, howling at the moon
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 5:28am

Re: Re: This is what I have been saying for several years now
Not quite true...but it's still remarkable, the instead of all the rampant corruption in the Conservative party exposed in the same book, everyone focusses on the Pig thing.
[ link to this | view in thread ]
Techanon, 28 Sep 2015 @ 5:30am

Re: Those GCHQ jokers and their operation names
Props for the ironic 1984 reference, GCHQ.

It's not ironic if they consider the book a manual instead of a warning.
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 5:34am

Re:
objecting about surveillance likely gets you surveyed, a vicious fucking circle, thats gonna make people more angrier, over THEIR actions of surveillance

Its like they think their entitled to do whatever the fuck they want with OTHER peoples lives, and expect everyone to be ok with it, and if not, they "MAKE" you "ok" with it, instead of respecting the rights and freedoms they supposedly "champion"

No choice, no consent, we are OWNED by our self appointed "betters"

I have no sympathy for their self afflicted problems.......
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 5:38am

Re:
i agree, they've surpassed their idols
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 5:41am

Re:
I was surprised not a bit
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 5:44am

Re:
Radiohead files copyright complaint

Radiohead now officially terrorists
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 5:46am

Re: 1984
Intelligence services, monkey see, monkey do
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 5:48am

Re: everyone needs to join a gang
Were already a part of the humanity "gang"
[ link to this | view in thread ]
Officer of Law, 28 Sep 2015 @ 5:48am

You criminal scum just keep on commenting
I'm taking notes!
[ link to this | view in thread ]
Anonymous Howard, 28 Sep 2015 @ 5:49am

I wear my tinfoil hat with pride.
I've been saying for years, admittedly driven only by my own paranoia, that this level of surveillance totally goes on.

Just because someone is paranoid, doesn't mean they are wrong.
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 5:54am

The devs who wrote these programs for them should be ashamed of themselves
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 5:58am

Re:
As a Techdirt reader, you should know why that would be nonsense.
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 5:58am

Re: Re:
They should never be forgotten
[ link to this | view in thread ]
hobo, 28 Sep 2015 @ 5:59am

Re: This is what I have been saying for several years now
It's worse than this as it's not just politicians who are at stake. It is only a matter of time before databases like this are compromised and data dumps occur. Then your history, your passwords, your life is out on public display.

This is exactly the same as wanting back doors to encryption, they think that only the "good guys" will have access, and that they will only use it for "good reasons." Neither of those suppositions is true.
[ link to this | view in thread ]
Violynne (profile), 28 Sep 2015 @ 6:06am

Re:
This would be equivalent to saying "Those who work for spying-on-the-public agencies should be ashamed of themselves", because these programmers work for the agencies pull the data.

People should also be aware these programs were assisted by the NSA, who has several years experience on capturing internet traffic.

Can't wait until the story breaks on how encryption on the internet is made moot by the ghost certs these agencies use.
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 6:16am

Re: everyone needs to join a gang
The problem with politics is that it is gang based, just that the gangs call themselves political parties. Anybody can join their gangs, but because their gangs put psychopaths in charge, very few people actually want to join.
[ link to this | view in thread ]
That One Guy (profile), 28 Sep 2015 @ 6:57am

Re: Re: This is what I have been saying for several years now
'Only a matter of time'? Yeah, that database would be priceless for countless groups, the odds that not a single one of them has cracked it by now is so low as to be non-existent.

It's pretty much a given that it's already been compromised, the only question is how much and by who?
[ link to this | view in thread ]
SteveMB (profile), 28 Sep 2015 @ 7:00am

Maybe "As Expeditiously As Possible" Isn't The Best Strategy
When Edward Snowden handed over his amazing trove of documents to journalists to release as they thought best, he also placed a huge responsibility on their shoulders to do so as expeditiously as possible.

Continuing the steady drip-drip-drip approach may be more effective in the long run. It had the advantage of driving the cycle:

1. Disclose X
2. Three-letter apparatchik denies X
3. Disclose evidence for X
4. Three-letter apparatchik admits X, but double-pinky-swears that it's only X and not Y
5. Disclose Y
6. Lather-rinse-repeat
[ link to this | view in thread ]
That One Guy (profile), 28 Sep 2015 @ 7:12am

Re: Insanity
Depends on who you ask.

A member of the public who's paying attention to what's been happening? Almost certainly not.

A member of the public who's not been paying attention, and gets all their news from the government? Absolutely, after all the entire planet would be a smoking crater by now if it weren't for the brave actions of the spy agencies, as clearly demonstrated by the claims made by the very same spy agencies.

A member of one of the spy agencies? Without a doubt. Never having to worry about your budget or any pesky 'investigations', the ability to absolutely ruin anyone who speaks out against you if you care to, or even just care to hint at it(access to their accounts allows more than just monitoring them after all)... yeah, I'm sure they consider the world much improved thanks to their actions.
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 7:13am

Re:
They broke the rules for the "Greater Good". That means they didn't really break any rules and everything they did was legal. /s
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 7:37am

really, it's all going to be so nice. all will be in agreement and no need for hard feelings or unpleasantry.

it'll be like one big tea party where we're all so nice, and nobody will bat an eye when the country we love chooses to eliminate an entire population somewhere that will have gotten between us and something we want.

tea party, yes. where the women come and go, talking of michelangelo. and of charlemagne and cortez, of course.

of course.

it'll be like heaven without the feathers.
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 7:43am

If only we had enemies worthy of this level of intrusion.

Ironically, we've never had enemies worthy of this level of intrusion.
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 8:18am

this has nothing to do with enemies and all to do with subjugation. nice subjugation, but subjugation all the same.

'after you, alphonse.'
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 9:01am

Of course, all this info is shared with the 5 Eyes,
including NSA.

NSA: we spy only on foreigners, but we share intel with GCHQ.

GCHQ: we spy only on foreigners, but we share intel with NSA.

Sounds like "pass the hash, on steroids".
[ link to this | view in thread ]
Median Wilfred, 28 Sep 2015 @ 9:46am

Data fog?
How does GCHQ work around deliberate or accidental data fog? I've spent several days with my "Google" login set to that of one of my kids, for example, which is going to mess up the nice profile for that kid.

Bruce Schneier looks up random people on facebook. Do I need to start doing that sort of thing? I can easily schedule "wget" to make HTTP requests of sites with DNS names composed from a list of goofy words, save the cookies I get, or just send random cookies, maybe with Verizon's extra HTTP header, just to make people wonder.

Will this be effective in clogging GCHQ's rather totalitarian database? If not, how many people doing this sort of thing would clog it?
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 9:56am

Re: Re: Re:
I was leaning towards, if their forgotten, then they lose their authority and just go away.......however freaking unlikely that is

But, i get ya, im not saying ignore the crap their doing to us
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 10:00am

Re: Re: Re: This is what I have been saying for several years now
Probably because people got used to politicians being corrupt.
The pig thing is something that hasnt been an everyday thing.
Wasnt there an investigation into some late-night parties they had which involved some very underage prostitutes?
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 10:03am

Re: You criminal scum just keep on commenting
Feels like a badge of honour
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 10:03am

Re: Data fog?
The database is already fogged when it comes to finding the small number of well organised terrorists and criminals. It allow them to see a popular protest forming, giving a chance to send in the trolls to put reasonable people off of taking part. It is also useful for identifying ring leaders of protests.
The gather it all approach mainly has use in controlling the population and heading off protest and the organization of political parties that could challenge the status quo.
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 10:18am

Re:
You presume that government and populace are on the same team
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 10:29am

Re: Data fog?
Data fog, i like it, but im partial to "Up Yours Five Eyes" app
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 10:31am

Re: Of course, all this info is shared with the 5 Eyes,
I WISH they were that restraint
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 10:42am

Re: Re: Data fog?
Nail meet hammer

Thankyou for putting it so eloquantly
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 10:45am

Re: Re: Data fog?
Five eyes nations
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 10:48am

Re: Maybe "As Expeditiously As Possible" Isn't The Best Strategy
>Continuing the steady drip-drip-drip approach may be more effective in the long run

As mentioned in the article, this is many times bigger than a phone metadata program, but as of yet the response has been incredibly small. Total surveillance has become pedestrian.
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 1:44pm

People who own a domain can now get a certificate for free.

https://letsencrypt.org/

Fuck GCHQ, NSA, and all other spy agencies.
[ link to this | view in thread ]
JMT (profile), 28 Sep 2015 @ 5:15pm

Re: This is what I have been saying for several years now
"It is only a matter of time where the party in charge of the government will have enough info on the other party to keep them from making a serious run for their office."

This is the vital fact that needs to be hammered home to those people who don't think government surveillance is such a big deal, because they've done nothing wrong, and terrorists! Even if you're never directly affected by government surveillance, if it keeps going unchecked eventually it will be used in ways that completely undermine the idea of a democratic government. It's just human nature, and there are plenty of historical examples. This is actually something worth hundreds of people of people dying at the hands of terrorists, because the end result could be hundreds of millions living under a far worse form of government than we have now.
[ link to this | view in thread ]
Anonymous Coward, 28 Sep 2015 @ 9:52pm

After reading this I had to double check that cookies are disabled in my web browser. No milk and cookies for GCHQ! Bad GCHQ! Stop it.

/sprays GCHQ agents with water like a cat
[ link to this | view in thread ]
GEMont, 30 Sep 2015 @ 2:30pm

And The Good News is:
Schizophrenics can stop taking their meds now.

You ARE Being Watched.

---
[ link to this | view in thread ]
Carl, 9 Jul 2016 @ 1:02am

Corruption
Know first hand that gchq gives full disclosure on anyone's computer history and text records going way back for the metropolitan police to blackmail its own staff with full collusion of
Met senior management and dps
[ link to this | view in thread ]