Did The NSA Think The Public Can't Do Math? Attempt To Downplay Data Collection Fails Miserably

from the carry-the-one... dept

Last week we wrote about the NSA's ridiculous attempt to justify its surveillance efforts, including this really wacky callout designed to show just how "little" data the NSA collects.

Scope and Scale of NSA Collection

According to figures published by a major tech provider, the Internet carries 1,826 Petabytes of information per day. In its foreign intelligence mission, NSA touches about 1.6% of that. However, of the 1.6% of the data, only 0.025% is actually selected for review. The net effect is that NSA analysts look at 0.00004% of the world's traffic in conducting their mission -- that's less than one part in a million. Put another way, if a standard basketball court represented the global communications environment, NSA's total collection would be represented by an area smaller than a dime on that basketball court.

This was bizarre on a number of levels, not the least of which is the wacky basketball court-to-dime scale. Next time, maybe we can play "is it bigger than a breadbox" with the NSA. But, as for what any of this meant, it hasn't been at all clear. Since the NSA has already redefined basic English words like "collect," "target," "datamine," and "relevant" it's not at all clear what is meant by "touch." However, some are starting to dig into the numbers, and contrary to the NSA's attempt to suggest that this is "nothing to fear," a bit of analysis certainly suggests they're collecting quite a bit of info.

First up, we have Jeff Jarvis, who highlights a bunch of important comparative datapoints including that Sandvine claims that only 2.9% of US traffic is communication traffic and 68.8% of all email is spam -- meaning that it's entirely possible that the NSA collects nearly all non-spam email and it would still be within its 1.6% number. He also points out that 62% of traffic on the internet is considered entertainment, and we can assume that the NSA doesn't need to collect every copy of Game of Thrones that people are passing around (I'm sure one or two will do the job). He similarly points out that Google itself claims to only index approximately 0.004% of traffic on the internet, suggesting that the NSA may be collecting more info than Google indexes by two orders of magnitude.

Meanwhile, Sean Gallagher, over at Ars Technica, digs a bit deeper into the numbers, suggesting that the NSA's data collection is closer to being on par with Google, but still greater than Google:

The dime on the basketball court, as NSA describes it, is still 29.21 petabytes of data a day. That means NSA is "touching" more data than Google processes every day (a mere 20 petabytes).

Gallagher also looks much more closely at the recently revealed details of the Xkeyscore program, to show how that 1.6% of "touched" internet communications can cover pretty much everything important.

As a result, if properly tuned, the packet analyzer gear at the front-end of XKeyscore (and other deep packet inspection systems) can pick out a very small fraction of the actual packets sent over the wire while still extracting a great deal of information (or metadata) about who is sending what to who. This leaves disk space for "full log data" on connections of particular interest.

In other words, while the 1.6% number was put forth by the NSA to try to make people think this is no big deal, when you look at what it means, it suggests it's a very big deal indeed. In fact, the NSA may be collecting even more information that people had believed before.

Filed Under: data, data collection, internet traffic, nsa, nsa surveillance, scale

The Scale Of Money

from the spend-a-few-hours dept

No one I know puts "scale" into perspective better than Randall Munroe at xkcd. If you haven't seen the latest, you should take the time to dive into what may be his largest image ever (and he's known for creating large images) dealing with money. I warn you, though, it may suck up a lot of time as you go through it: While he's offering a poster of it for sale (along with some of his other "giant" images), I'm not sure even a poster does something like this justice, which is why he's also offering it as a "custom-printed four-poster tile pack. It comes as four individual 36"x24" posters which can be tiled on the wall, for a six-foot-wide mural view of the chart, allowing you to clearly read even the finest details." I may have to put that on my holiday wishlist.

Filed Under: money, scale, xkcd

Losers Relish Scarcity, Winners Leverage Scale

from the yeah,-what-he-said dept

For quite some time now, we've been trying to convince many folks in a variety of industries, but especially the recording industry, to recognize why their rush to create and embrace business models that rely on artificial scarcity was not sustainable. What's been most amazing is how the folks in the industry themselves turn a blind eye to it (or falsely seem to accuse us of just promoting "theft.") However, with the recording industry continuing to struggle, it looks like insiders are finally starting to hear the message. Mathew Ingram points us to a talk given by Yahoo!'s Ian Rogers to a music industry conference last month with the title: Losers Relish Scarcity, Winners Leverage Scale. He goes on to discuss the "physics" of media, which is nothing more than basic economics. However, using the word "physics" makes sense here. For whatever reason (and economists may be at fault here), too many people still assume that economics are what people want to happen or what should happen, rather than recognizing that it describes forces that actually are happening. Talking about economics as "physics" helps get that point across. It's a good pitch, though my experience suggests that the important people won't listen (or, rather won't "hear" what's being said). While much of the industry is figuring this out, the big bosses of the record labels are still a long way from waking up to the reality they face.

Filed Under: economics, scale, scarcity
Companies: yahoo