Senate Not Concerned About How Often NSA Spies On Americans, But Very Concerned That It Built Open Source Software To Do So

from the priorities,-people dept

Wed, Jul 18th 2012 12:17pm — Mike Masnick

Wired has a troubling story of how the Senate Armed Services Committee is pushing a bill that would likely kill off an open source NoSQL project that came out of the NSA called Accumulo. Like many other such NoSQL efforts, the NSA basically took some Google white papers about its BigTable distributed database setup, and built its own open source version, with a few improvements... and then open sourced the whole thing and put it under the Apache Foundation. It's kind of rare to see such a secretive agency like the NSA open source anything, but it does seem like the kind of thing that ought to be encouraged.

Unfortunately, the Senate Armed Services Committee sees things very differently. As part of a 600-page bill that's being floated, it actually calls out Accumulo by name, and suggests that it violates a policy that says the government shouldn't build its own software when there are other competing commercial offerings on the market. The reasoning is basically that the government shouldn't spend resources reinventing the wheel if it can spend fewer resources using existing code. You can see the basic reasoning behind that, but applying it here makes little sense. As the article notes, here we're talking about software that's already been developed and released -- not a new effort to rebuild existing software. In fact, those who follow this stuff closely note that Accumulo did "break new ground" with some of its features when it was being built. To then kill it afterwards seems not just counterproductive, but could also create a chilling effect for government open source efforts, which seem like something we should be encouraging, not killing.

What's really odd is the close interest that the Senate seems to be paying to this. The discussion is very specific, naming Accumulo and some of the competing offerings on the market. They're specifically calling out this one product. Of course, as Julian Sanchez notes, there's a bit of irony in the fact that the very same Senate appears to have absolutely no interest in finding out how often the NSA spies on Americans... but sure is concerned about what database it uses to store all of the information it's getting.

Of course... all of this raises a separate issue in my mind: can the NSA even open source Accumulo? I though that creations of the federal government were automatically public domain, rather than under copyright. And, thus, putting it under a specific license might, in fact, present limitations that the government can't actually impose on the software.... Thus, shouldn't the software code actually be completely open as a public domain project? The government should be able set up an Apache-like setup, but one without any restrictions on the code.

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: accumulo, bigtable, databases, nosql, nsa, open source, senate, spying

34 Comments

If you liked this post, you may also be interested in...

Reader Comments

Subscribe: RSS

View by: Time | Thread

Anonymous Coward, 18 Jul 2012 @ 12:24pm

The policy actually states that the government cannot spend money to build software when it would be less expensive to buy a commercial software product. Not that it can't build software if there is a competing commercial product.

It's a distinction that will probably end up in favor of the NSA actually, since they undoubtedly made a product with features that weren't offered in a commercial product and would have been expensive to contract a commercial entity to add.
[ link to this | view in thread ]
sehlat (profile), 18 Jul 2012 @ 12:34pm

The Real Problem With Open Source Spy Software
Heaven forfend that the proles might get the same high-quality tools that their government gets!
[ link to this | view in thread ]
el_segfaulto (profile), 18 Jul 2012 @ 12:35pm

I get this a lot
I work for a state government as a developer. There are a number of projects that I've been involved with where the mere mention of using open source software is met with derision. The cost/benefit analysis I typically give is that a few weeks of my salary is far less than a commercial purpose, but even then we still shop around and get quotes from contractors (where the real money is!).

My personal theory is that no government agency wants to save money since it means they'll have to do without the difference for the next budget cycle. I've heard stories of tens of thousands of dollars being spent on useless software just to ensure that the agency in question would be able to maintain their budget.
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2012 @ 12:39pm

Re: The Real Problem With Open Source Spy Software
Of course the question becomes, should NSA grade Spyware really be in the hands of the public?
[ link to this | view in thread ]
The Mighty Buzzard (profile), 18 Jul 2012 @ 12:52pm

Re: Re: The Real Problem With Open Source Spy Software
Right, because data storage systems are NSA grade spyware.
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2012 @ 12:56pm

Re: Re: Re: The Real Problem With Open Source Spy Software
Wait, it's just data storage? Didn't realize that. Still if you read the comments it looks like the NSA has the advantage here.
[ link to this | view in thread ]
Ninja (profile), 18 Jul 2012 @ 12:57pm

Re: I get this a lot
I'm not a developer but I see your point as a state government employee. Once you lose the budget it'll be a war that will last months, if not years to get it back. Sometimes you must "waste" the money to make sure you won't miss it in the future. You can't "save" that money for later, bigger and more important projects as it often would be possible if it wasn't for the budget issue.

I think a more sensible and reasonable solution would be to keep the unused portion of the budget as a credit to the agency/department (up to a limit and/or to an amount that finances a bigger project) for posterior use.

Wishful thinking. People sometimes are quick to point the waste of taxpaying money but they aren't aware of the bureaucracy hell inside Governmental bodies that end up forcing all sorts of inefficiency.
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2012 @ 12:59pm

Contrary to popular opinion, Americans are not the world's dumbest people...
That title is reserved for their government.
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2012 @ 1:22pm

Re: Contrary to popular opinion, Americans are not the world's dumbest people...
'Mericans are the ones who let this shit slide
[ link to this | view in thread ]
Bengie, 18 Jul 2012 @ 1:22pm

Fine then
NoSQL is a commercially available free product. I don't see the issue.

Any changes they made could be akin to doing performance tweaking/testing with any other product.
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2012 @ 1:37pm

Re: Fine then
The issue is with that special lobbyist who is pushing this from under the table getting his fat paycheck for this quarter.
[ link to this | view in thread ]
jessej, 18 Jul 2012 @ 1:57pm

I'm reminded of a book . . .
I read 'The First Circle' by A. Solzhenitsyn (sp?) back in the early 70's. He described Russian bureaucrats as always buying large 50 gal drums of black ink at the end of every fiscal year, to use up their budgets by getting a resource they could 'always use' and show the higher ups that they would need at least the same amount of cash for the next annual funding cycle. That was an old idea back then. Naturally, there were warehouses full of ink barrels . . .
[ link to this | view in thread ]
Wally (profile), 18 Jul 2012 @ 2:14pm

Spending
NSA: Let's save money using an open source language and code to save money. We can help keep the country safe without the red tape involved.

Senate (publicly): We want to know how many people you've spied on Americans.

NSA: You don't have the security clearence or the brain power to know what is actually done.

Senate (privately): How dare you not SPEND money to create jobs.

Given the current US economy, the fact that the NSA used open source programming languages to do any spying makes a lot of sense. Under the GNU, you have to distribute it for free if you release the code to the public. Since they haven't released the code to the public, I see no problem with that. They saved their department a lot of money and made us slightly less dependent on the government. Well you know how spending alone creates debt....and that's why our debt ceieling is so high.
[ link to this | view in thread ]
Jeffry Houser (profile), 18 Jul 2012 @ 2:19pm

What Restrictions does the Apache License place on code?
"The government should be able set up an Apache-like setup, but one without any restrictions on the code."

What restrictions does the Apache License place upon the code?

The license, as I understand it; says one can do anything to the code. You can modify it; extend it, sell it, create your own version without any restrictions, release it under a different license, etc...

I'm surprised at how open the Apache license is.
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2012 @ 3:11pm

Sounds like someone important wants to start selling the NSA database software. So the senate is doing its part to throw out their homemade working code and replace it with some 2 billion dollar hunk of shit, probably managed by Lockheed.
[ link to this | view in thread ]
Fritzr (profile), 18 Jul 2012 @ 3:17pm

Re: Re: I get this a lot
The reasoning behind the current system is that each budget request specifies what if being paid for. Unneeded money is then available for other agencies to use for their needs.

The effect of budget reviewers saying your are asking How Much!! for this change??? is simple. Budget authors pad the request and require the purchasing agents in the department to make sure the entire request is used, preferably with a little bit of a budget overrun that can be used to justify "We failed to ask for enough last time...give us a raise to cover our expected needs"

Actually planning and requesting one time funding for a project is a nightmare as there is no history for the budget committee to look at and see that the request is not unusual. Unusual in the eyes of the bureaucratic bean counter is almost identical to unnecessary.
[ link to this | view in thread ]
Fritzr (profile), 18 Jul 2012 @ 3:22pm

Re: What Restrictions does the Apache License place on code?
The unmodified NSA code can probably be treated as Public Domain, but the contributions from outsiders to the Apache Project are not government contributions and will be subject to the license limitations.

It makes sense to put it out as a licensed managed project, rather than simply posting the source & docs on a website and saying "This is public domain, take it or leave it".

The managed project will gather outside improvements that are then available for the classified internal version of NoSQL.
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2012 @ 3:57pm

You know, with all this new ability to spy on us, I wonder how stringently they are following the "No Disclosure Without Consent Rule."
[ link to this | view in thread ]
Wally (profile), 18 Jul 2012 @ 4:16pm

Re: Re: Re: Re: The Real Problem With Open Source Spy Software
Try cracking into the database....you'll have the FBI, CIA, Secret Service, and a SWAT Twam knocking at your door pdq.

There is so much superfluous data and information going through their systems it really doesn't matter if they have your personal information. Nobody else but the NSA can see what they have collected without an order from the CIA.

I think it's safe to say congress is just worried that they aren't "stimulating" the economy by spending away the federal budget (or lack thereof).
[ link to this | view in thread ]
Wally (profile), 18 Jul 2012 @ 4:28pm

Re: Re: Contrary to popular opinion, Americans are not the world's dumbest people...
*rolls eyes in sarcasm* Yes, because Europe's debt to the US is definitely not helping...
[ link to this | view in thread ]
Wally (profile), 18 Jul 2012 @ 4:31pm

Re: Re: What Restrictions does the Apache License place on code?
Just bevsuse the programming language is public domain doesn't mean the coding of a program using that language has to be.
[ link to this | view in thread ]
Wally (profile), 18 Jul 2012 @ 4:34pm

Re: I'm reminded of a book . . .
It also indicates one of the main reasons Soviet Russia fell economically in the late 80's.
[ link to this | view in thread ]
Rapnel (profile), 18 Jul 2012 @ 4:42pm

I will
I will take this kit that readily serves many of my needs. I will make it better by fine tuning it to suite my needs better. I will not be thwarted by agreements, closed source, non-disclosure, meetings and money. I will make it better by making it serve many more needs. I will return it from whence it came and I will return it in a better state than that which I found it in.

Now, if only we could do that with the Congress.

Open source. Open media. Open government.

If you do not subscribe to at least two of the above, in whole or in part, then you might be part of a rather sizable problem.

As for prioritizing the "issue" of software selection over one of the very tenets of this fucking country... fuck you you worthless pieces of shit.
[ link to this | view in thread ]
Pseudonym, 18 Jul 2012 @ 5:05pm

Re: Re: What Restrictions does the Apache License place on code?
Yes. Anything the NSA writes is public domain, but there's nothing that prevents them from donating that public domain code to the Apache project and them slapping a copyright notice on any modified version they maintain or release.
[ link to this | view in thread ]
Tim McCormack (profile), 18 Jul 2012 @ 5:51pm

Re: Spending
* "open source programming languages"
[ link to this | view in thread ]
Tim McCormack (profile), 18 Jul 2012 @ 5:55pm

Re: Re: Spending
Sorry, TechDirt apparently strips anything following an open angle bracket. Someone needs 3 whacks with the eager-sanitization stick.

* "open source programming languages" -- who said anything about programming languages? This is open-source software.
* "Under the GNU" -- It's the Apache License, not the GPL (if that's what you meant to refer to)
* "you have to distribute it for free if you release the code to the public" -- You have deeply misunderstood the GPL. It's the other way around. Sort of. And the GPL places no burden on the original author, only on licensees.
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2012 @ 7:38pm

Re: Time To Dig
Maybe managed by Lockheed, but the real movers behind the scenes are likely to be Microsoft, Oracle, IBM or some combination of them. They make a lot of money out of database software. The last thing they want is open source software getting used on any large scale. There will have been plenty of late night calls to senators, making all kinds of threats and promises.

Dig a little deeper.
[ link to this | view in thread ]
Sad Mac, 18 Jul 2012 @ 10:06pm

Re: Re: Re: What Restrictions does the Apache License place on code?
I'm not sure how why they would. I don't think the NSA would be inclined to release such software in the first place. Even if the program language is under public domain, they still have the right not to release the code they created. Why should they?
[ link to this | view in thread ]
Wally (profile), 18 Jul 2012 @ 10:17pm

Re: Disclosure
"You know, with all this new ability to spy on us, I wonder how stringently they are following the "No Disclosure Without Consent Rule." "

They follow it quite well. People tend to forget that human beings actually work at the NSA unlike congess, where we have a bunch of howler monkies trying to gain power for their side alone. I kind of wish congess would see we are human as the NSA does.
[ link to this | view in thread ]
NSA, 18 Jul 2012 @ 10:20pm

"Open source. Open media. Open government."

Precisely what we do for you and people around the world :-)
[ link to this | view in thread ]
DrZZ, 19 Jul 2012 @ 5:52am

federal government works

I though that creations of the federal government were automatically public domain, rather than under copyright.

Depends on how you define "creations of the federal government". Works that are created by federal employees can not be copyrighted, but works directed by federal employees, but actually done by contractors can be copyrighted. (more info) It is not who pays for it, it is who actually does the writing that counts. I couldn't tell from the article whether contractors were used or not.
[ link to this | view in thread ]
Anonymous Coward, 19 Jul 2012 @ 7:27am

If the government produces an open source, public domain product people want to use, the companies that want to charge people for a product that does the same thing will complain the loudest.
[ link to this | view in thread ]
Wayne, 21 Jul 2012 @ 9:57am

Government Copyright
It all comes down to how the government built the software. If it was contracted, then the contractor gets the copyright and the government gets "government purpose rights".

If the government actually built it with government employees then it would be public domain in terms of copyright law but it might still be restricted from release to the general public (in context think trade secret rather than patent).

Different parts can be under different restrictions. So, even if the project is open source, any changes done by the government may still be restricted or otherwise withheld from the general public. The government can, at that point, still license the software under those restrictions.

Of course, if someone stuck those changes in a public git repository (in an official capacity) then it seems moot, but that's why we have so many lawyers.
[ link to this | view in thread ]
JackOfShadows (profile), 21 Jul 2012 @ 7:40pm

Re: Re: I'm reminded of a book . . .
Actually it fell by design.
[ link to this | view in thread ]