How Much Would It Cost To Store All US Phone Calls Made In A Year?
from the cheaper-than-you-think dept
An early criticism of Snowden's leak about NSA spying activity was that the $20 million annual cost for PRISM -- whatever that turns out to be -- was simply too low to be credible. One person who knows more about storage costs than practically anyone -- well, outside the NSA, at least -- is Brewster Kahle, who set up the Internet Archive, essentially a backup for the entire Web plus a wonderfully rich store of many other materials. He's carried out a fascinating back-of-the envelope calculation of how much it would cost annually to record every phone call made in the US and store it in the cloud:
These estimates show only $27M in capital cost, and $2M in electricity and take less than 5,000 square feet of space to store and process all US phonecalls made in a year. The NSA seems to be spending $1.7 billion on a 100k square foot datacenter that could easily handle this and much much more. Therefore, money and technology would not hold back such a project -- it would be held back if someone did not have the opportunity or will.
Kahle has made the calculation available as a shared document (on Google, appropriately enough), so you can inspect his assumptions there and play around with the numbers. It's also worth reading through the comments to his short post, since they make some interesting points. However, even if the numbers are off by a factor or two, there's no doubt about the feasibility of recording all US phone calls.
And that's for sound files, which take up quite a lot of space. Text-based information pulled in from emails, Web pages and chat logs could be stored more compactly. That would make the routine recording of vast swathes of what those in the US -- and outside it -- do online not just plausible, but so cheap in comparison to the NSA's presumably large budget, that the latter might feel it would be crazy not to do so as a matter of course.
Follow me @glynmoody on Twitter or identi.ca, and on Google+
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: brewster kahle, cost, nsa surveillance, phone calls, recording, surveillance
Reader Comments
Subscribe: RSS
View by: Time | Thread
Transit costs
There are, of course, two basic approaches:
• Simultaneous transport to long-term storage
• Delayed transport transport to long-term storage
Those two different approaches will have different costs.
[ link to this | view in thread ]
Funny the problem should be paying out for food stamps in a tough economy.
This whole thing is a fisaco that should be followed right down to the last penny, the last broken law, the last false interpretation of what the law says.
It's looking more every day like you could change the symbol and color of the flag and be China or Russia. Sorry but this doesn't look like the country I grew up in at all.
[ link to this | view in thread ]
Something else to consider:
[ link to this | view in thread ]
This is government, remember
The difference between "the cost to store and manage the data" and "the cost for a dozen Oracle licenses because we're inherently terrified of open source" is many millions of dollars.
[ link to this | view in thread ]
Voice call minutes per month
Andy's deriving his numbers (by calculation) from:
CTIA. (2011). Year-end 2010 top-line survey results.
See Andy's footnote 9.
[ link to this | view in thread ]
Re: Transit costs
[ link to this | view in thread ]
Re: Re: Transit costs
How about the number of central offices in the U.S.? That number would be handy right now.
Or just go ahead and make comments with no sources and no numbers and no calculations and no thought.
[ link to this | view in thread ]
Central office storage and transit utilization ratio
Data networks are lightly utilized, and will stay that way, A. M. Odlyzko, Review of Network Economics, 2 (no. 3), September 2003, p. 218.
It occurs to me that for an intercept network, that even if it's desired to have most calls forwarded to the processing/storage facility in near real time, it may not make sense to engineer the transit network for peak load conditions.
That is, if there's some storage provisioned at the call capture point, then even if most call captures are forwarded immediately, the intercept network can even out bursts during holidays, and other peak periods.
In addition, a day or two of storage at the capture point would allow call captures to be retained even in the event of an outage in the intercept transit network.
[ link to this | view in thread ]
Re: Re: Transit costs
https://en.wikipedia.org/wiki/Open_Garden
That should do it.
[ link to this | view in thread ]
Re: Re: Re: Transit costs
[ link to this | view in thread ]
[ link to this | view in thread ]
Re: Re: Re: Re: Transit costs
But I don't necessarily like the source I've got for that $3 number.
[ link to this | view in thread ]
Re: Re: Transit costs
[ link to this | view in thread ]
Re: Re: Re: Re: Re: Transit costs
[ link to this | view in thread ]
all of this is nice but
Don't let reality get in the way of painting the current administration into a corner.
(oh, and day 6 of getting my posts "moderated" rather than posted... censorship lives at Techdirt!)
[ link to this | view in thread ]
storage is one thing...
[ link to this | view in thread ]
Re:
[ link to this | view in thread ]
[ link to this | view in thread ]
The phone companies also maintain specific servers to hold the information requested, it would also not be a very large amount of information to be stored anywhere. You can store a huge amount of data on a 2TB drive and they are cheap..
[ link to this | view in thread ]
Re: Re: Re: Re: Re: Re: Transit costs
“IP transit price declines steepen”, TeleGeography, 2 Aug 2012
[ link to this | view in thread ]
Interesting estimates but completely irrelevant
So while this discussion is interesting from an academic point of view, it's completely meaningless to the real world. Presume, for all practical purposes, that the NSA has infinite money and reason accordingly.
[ link to this | view in thread ]
Re: Interesting estimates but completely irrelevant
[ link to this | view in thread ]
Re:
[ link to this | view in thread ]
Telco CO count
According to that report, from Table 17.4 (p.17-8) (p.142 in PDF), there are 24,357 telco central offices in the United States. Note that this number may be an undercount, because the FCC WCB is relying on industry reports which may not be required from all telcos. In addition, it seems probable that this number is only for wireline central offices.
Let's call it twenty-five thousand ( 25,000 ) in round numbers.
Other tables in that report (table 17.1 and 17.2) give us an idea of the distribution of traffic across switches associated with those telco COs (CO switches and remote switches). Note though, that table 17.1 comes from numbers which are only reported to the FCC by RBOCs. And table 17.2 comes from numbers which are only reported by ILECs (including RBOCs).
Acronyms:
CO: Central Office
FCC: Federal Communications Commission
WCB: FCC Wireline Competition Bureau
RBOC: Regional Bell Operating Company
ILEC: Incumbent Local Exchange Carrier
CLEC: Competitive Local Exchange Carrier
[ link to this | view in thread ]
Re: Re: Re: Transit costs
According to the FCC (p.214 in PDF):
Consider just the 25,000 wireline central offices (neglecting the unknown number of mobile switch centers), let's say those telco COs are on average 1,000 miles away from the Utah facility. That's probably low, considering the population and infrastructure density on the East coast. If the network is built as one big physical star, then that's 25 million miles of fiber (conservatively).
If fiber is priced at the bargain-basement price of only $10,000/mile, then it comes to a quarter trillion dollars for a physical star network. That would make a dent in the federal budget.
No, I don't think, “the NSA can afford to string all kinds of really really fast optical fibers from wherever to wherever”.
[ link to this | view in thread ]