The NSA Reveals That It Does 20 Million Database Queries Per Month

from the that's-a-lot dept

As we noted earlier today, the NSA's two key "defenses" of the thousands of abuses and violations of the law that recently came out thanks to a leaked document are that there wasn't "intent" to abuse the system (we had no idea that made illegal things legal...) and, second, that it was such a small percentage of the activity that it's really no big deal. Glenn Greenwald quickly noted that the NSA is actually saying "we collect billion of emails and calls every day, so what's a few thousand privacy violations?" hoping that everyone focuses on the second half of the sentence. But the key point is actually the first half of that sentence. In fact, as we noted in that last post, the NSA's top compliance guy actually revealed a startling fact in his attempt to push the meaningless "ratio" of violations to queries:
The official, John DeLong, the N.S.A. director of compliance, said that the number of mistakes by the agency was extremely low compared with its overall activities. The report showed about 100 errors by analysts in making queries of databases of already-collected communications data; by comparison, he said, the agency performs about 20 million such queries each month.
Again, the ratio is a meaningless number. You're not declared innocent of murder because you didn't happen to murder someone every other day of your life. But, perhaps more important in this is the revelation of the 20 million queries every single month. Or, approximately 600,000 queries every day. How about 25,000 queries every hour? Or 417 queries every minute? Seven queries every single second. Holy crap, that's a lot of queries.

Remember, too, that the NSA has insisted that it doesn't datamine its data collection, which is clearly hogwash. That many queries means they're trolling through that database all the time. Remember how the NSA was trying to play down how often it did queries by saying that only 300 phone numbers had been used to "initiate" a query? Yeah, well, once again, it would appear that the NSA was not being fully forthcoming about these sorts of things. Shocking, I know, but I'd imagine they'd claim it was the "least untruthful" answer they could come up with after having a good week or so to answer the question.
Hide this

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: database queries, datamining, nsa, nsa surveillance, privacy


Reader Comments

Subscribe: RSS

View by: Time | Thread


  • icon
    Ninja (profile), 19 Aug 2013 @ 9:47am

    Well, if you consider Snowden came from the NSA and he's letting all this in the wild then the NSA is indirectly telling the truth. O.o

    And some of the basic infra-structure of the web is in American soil. Sure this mass surveillance on Americans is worrisome for the Americans themselves but shouldn't the rest of the world be worried too?

    link to this | view in chronology ]

  • icon
    Zakida Paul (profile), 19 Aug 2013 @ 10:20am

    Well, to be honest that's not a lot considering the US has over 300 million terrorists inside it's borders.

    link to this | view in chronology ]

    • identicon
      Drew, 19 Aug 2013 @ 8:17pm

      Re:

      That's about .002% of the population?

      Er.. I need to stop posting stupid shit like this before the NSA decides to recruit me as an numbers analyst.

      link to this | view in chronology ]

  • identicon
    Anonymous Coward, 19 Aug 2013 @ 10:31am

    SELECT * FROM Public_Data_All
    --WHERE IsTerrorist = 1

    link to this | view in chronology ]

    • identicon
      Anonymous Coward, 19 Aug 2013 @ 10:36am

      Re:

      Where is the mapreduce of that one?

      link to this | view in chronology ]

    • icon
      RyanNerd (profile), 19 Aug 2013 @ 10:42am

      Re:

      SELECT * FROM Public_Data_All
      WHERE last_name like 'Snowde%'

      FTFY

      link to this | view in chronology ]

    • identicon
      Anonymous Coward, 19 Aug 2013 @ 10:51am

      Re:

      SELECT * FROM Public_Data_All
      --WHERE IsTerrorist = 1


      Commenting out the where clause explains why they think every American is a terrorist.

      link to this | view in chronology ]

    • icon
      Ninja (profile), 19 Aug 2013 @ 11:44am

      Re:

      We should replace some commonly used word for terrorist for the lulz. Ie: reader

      Techdirt terrorists often discuss about tech stuff. According to these terrorists technology can make carrying their daily terrorism (reading) tasks easier. Filter gone haywire!

      link to this | view in chronology ]

    • identicon
      Anonymous Coward, 19 Aug 2013 @ 12:51pm

      Re:

      SELECT * FROM PublicData
      WHERE Exists = 1

      link to this | view in chronology ]

  • This comment has been flagged by the community. Click here to show it
    identicon
    out_of_the_blue, 19 Aug 2013 @ 10:33am

    How many of those use "direct access" to Google's servers?

    Which is worse: NSA or the collaborating corporations?

    And which spying corporation is the worst? Google, Facebook, Microsoft, Yahoo, Apple...

    link to this | view in chronology ]

    • icon
      Zakida Paul (profile), 19 Aug 2013 @ 10:39am

      Re: How many of those use "direct access" to Google's servers?

      "Which is worse: NSA or the collaborating corporations?"

      Two sides of the same coin. The NSA uses the private corporations to harvest data on the population.

      Corporate spying alone is way down the list because people have the choice as to whether or not they want to use that company's services.

      I don't know why I bothered because you will inevitably ignore what I said and continue with your anti corporation rants.

      link to this | view in chronology ]

      • identicon
        Anonymous Coward, 19 Aug 2013 @ 11:00am

        Corporate money out of politics

        Being anti-corporate is not exactly irrational. Partially misdirected blame but even if a person is completely "anti-corporation", is it really that bad of an ideology ?

        OBLIGATION of business to make profit disregarding any societal cost.
        Without regulation and laws then growing people to be slaves would be legitimate pro-corporation ideology.
        It's the laws and regulations that need to be the focus.

        By law, google are not allowed to grow slaves.
        By law, google are required to hand over all access to NSA.
        Who is funding the American Laws ?
        Corporations.

        So being anti-corporation is actually legitimate.
        Being anti corporate funding of politics is smarter.


        Until the money is out of politics, there will be a valid reason for people to be anti-corporatist.

        link to this | view in chronology ]

    • identicon
      Anonymous Coward, 19 Aug 2013 @ 10:45am

      Re: How many of those use "direct access" to Google's servers?

      You forgot the companies of the industrial military complex.

      Corporations --------> $$$$$$$$ ---------> Government Officials

      Government Officials --------> Beneficial Laws ---------> Corporations


      Google, Facebook, Microsoft, Yahoo, Apple etc... are pretty much victims and are just complying with these new "Laws".
      It's the Corporations that are developing/supplying Tech and services for the NSA that are the ones making the laws via bribery.

      link to this | view in chronology ]

      • icon
        GoldHoarder (profile), 19 Aug 2013 @ 11:51am

        Re: Re: How many of those use "direct access" to Google's servers?

        Corporations --------> $$ ---------> Government Officials

        Government Officials --------> Beneficial Laws ---------> Corporations------> Hire Government Officials upon leaving office or positions for $$$$$$$$$$$$$$

        Fixed

        link to this | view in chronology ]

    • identicon
      RD, 19 Aug 2013 @ 11:17am

      Re: How many of those use "direct access" to Google's servers?

      "Which is worse: NSA or the collaborating corporations?"

      Fucking idiot. The fact that you even have to ask that shows your bias and insane focus on Google to the exclusion of all else.

      The NSA is FAR worse than any corporate collaborator, because the NSA has the full weight of the government and law behind it.

      A corporation can't arrest you or put a gun to your head or order you murdered via drone strike.

      link to this | view in chronology ]

  • identicon
    Anonymous Coward, 19 Aug 2013 @ 10:35am

    'only 300 phone numbers had been used to "initiate" a query'

    Well, if those 300 people each called 41 people, and those all called 41 people, and those all called 41 people, we're already over 20 million.

    link to this | view in chronology ]

  • icon
    RyanNerd (profile), 19 Aug 2013 @ 10:39am

    Holy Crap that's quite a bit of data mining

    I worked for a health insurance company. And on occasion we would have to query the raw data for a patient's prescription history. We only did this when an issue came up or we were performing an audit. Even then there were strict guidelines for how we were to perform a search. This was to protect the privacy of everyone.
    For example: If we had a first and last name but no social security number. We were forbidden from doing a search unless we also had their zip code of residence. The searches were to be exact and targeted as possible; which if you care about privacy is how things should be.
    This broad untargeted data-mining by the NSA is astounding. Makes me sick.

    link to this | view in chronology ]

    • identicon
      Michael, 19 Aug 2013 @ 10:45am

      Re: Holy Crap that's quite a bit of data mining

      It's a bit more like data-strip-mining.

      link to this | view in chronology ]

      • identicon
        Anonymous Coward, 20 Aug 2013 @ 2:51pm

        Re: Re: Holy Crap that's quite a bit of data mining

        Is data-strip-mining a form of strip search?

        link to this | view in chronology ]

  • identicon
    Michael, 19 Aug 2013 @ 10:42am

    Not to nit pick, but the number of queries does not necessarily tell you much. Many people are thinking "select * from phonecalls where country <> 'US'", but that's not how all databases work.

    Some data analytics systems will run thousands of queries to satisfy a single request.

    They certainly should not be using the total volume of queries to try to minimize the number of abuses, but at this point, they are simply throwing out numbers that are meaningless without context.


    We should all be focused on the simple number of times they have abused their power or broken the law. Trust me, the number of times we were not speeding does not matter one bit when it comes to defending the time the radar gun got us.

    link to this | view in chronology ]

    • icon
      RyanNerd (profile), 19 Aug 2013 @ 11:05am

      Re:

      By no stretch of the imagination does 20 million queries per month == targeted information gathering. Plain and simple this is indiscriminate data-mining. There is no legitimate argument to the contrary.

      link to this | view in chronology ]

    • icon
      Ninja (profile), 19 Aug 2013 @ 11:47am

      Re:

      This is a matter of semantics. Are they using queries in the technical sense (and then you would be right while not making their abuse any better) or in the common sense of the word where they do search 20 million times (thus the number of technical queries is way higher) which makes their abuse sound even worse... Now, which definition is the NSA playing by?

      link to this | view in chronology ]

      • identicon
        Anonymous Coward, 19 Aug 2013 @ 12:22pm

        Re: Re:

        Some other definition that neither common folk nor database administrators would ever dream of.

        link to this | view in chronology ]

  • identicon
    NSA, 19 Aug 2013 @ 10:44am

    don't crossreference our statements!

    thats our job, we do millions every month and are damn good at it.

    link to this | view in chronology ]

  • identicon
    Anonymous Coward, 19 Aug 2013 @ 10:50am

    The worrying problem is that they have the data available to allow them to make mistakes. This can only increase the pressure to allow other law and security services to access the data increasing the opportunities for mistakes and abuse.

    link to this | view in chronology ]

    • identicon
      Michael, 19 Aug 2013 @ 10:57am

      Re:

      The NSA does not make mistakes. They simply expand their searches to other relevant data.

      link to this | view in chronology ]

    • identicon
      Anonymous Coward, 19 Aug 2013 @ 12:26pm

      Re:

      And how long before ICE is using to assist their funders, sorry clients, sorry (what are they?) in the music and movie industries in sending out shakedown, sorry settlement letters?

      link to this | view in chronology ]

  • icon
    Uriel-238 (profile), 19 Aug 2013 @ 10:57am

    If the Bacon numbers are a good indicator.

    That vastly improves the chances that they're reading my personal email.

    And yours too.

    I suspect everyone is few enough degrees from a terrorist suspect to be worthy of inclusion in search queries.

    link to this | view in chronology ]

    • identicon
      Michael, 19 Aug 2013 @ 11:02am

      Re: If the Bacon numbers are a good indicator.

      Are you calling Kevin Bacon a terrorist?

      Well, I suppose all of that anti-establishment dancing could raise some red flags these days.

      link to this | view in chronology ]

      • icon
        That Anonymous Coward (profile), 19 Aug 2013 @ 11:15am

        Re: Re: If the Bacon numbers are a good indicator.

        HE IS!
        There was a law against dancing, and he violated it and lead others to violate it as well!

        link to this | view in chronology ]

  • identicon
    Anonymous Coward, 19 Aug 2013 @ 11:17am

    So I'm getting between 7 and 8 queries per second, there. Assuming around-the-clock querying.

    Between 34 and 35 per second if the querying is only during business hours, where 20 business days defines a "month".

    link to this | view in chronology ]

  • icon
    ArkieGuy (profile), 19 Aug 2013 @ 11:25am

    One query for 3 out of 4 Americans every year

    Another way to think of this is the NSA is doing 3 queries for every 4 people in the US.

    link to this | view in chronology ]

  • identicon
    Anonymous Coward, 19 Aug 2013 @ 1:01pm

    How is this not illegal searching through illegally seized information.

    That's not a question, it's a statement. If anyone does have any questions, I refer you to the Constitution, Amendment 4.

    link to this | view in chronology ]

  • identicon
    Anonymous Coward, 19 Aug 2013 @ 1:06pm

    Why break the local laws? Should just share.

    What I don't understand is why all the government allies don't just work together and save themselves some huge embarrassment and political ire. If I had put this together I would have our spies force US companies to hand over our info to the UK spies. The UK spies have their companies giving us their info. We could then both LEGALLY spy on the others' citizens and share all their findings back with the respective governments legally. Seems so simple.

    Everything I needed to know I learned in kindergarten.

    link to this | view in chronology ]

  • identicon
    Anonymous, 19 Aug 2013 @ 3:53pm

    In the words of Bonnie Raitt...

    Let's give 'em something to talk about.

    link to this | view in chronology ]

  • identicon
    Anonymous Coward, 19 Aug 2013 @ 4:41pm

    The abuses come from LOGGED data, the biggest abuses are in the UNLOGGED queries.


    Do you really think there is no skunkworks version of that app that doesn't log queries? Because I think that's naive.

    link to this | view in chronology ]

    • icon
      nasch (profile), 20 Aug 2013 @ 12:51pm

      Re:

      Do you really think there is no skunkworks version of that app that doesn't log queries? Because I think that's naive.

      I don't know, the NSA seems to have a compulsive need to keep track of everything. I'm not sure they would be willing to just not log something.

      link to this | view in chronology ]

      • icon
        Uriel-238 (profile), 20 Aug 2013 @ 3:49pm

        The NSA compulsion to keep track

        I don't know, the NSA seems to have a compulsive need to keep track of everything. I'm not sure they would be willing to just not log something.

        I hope you're right. Thorough records will be convenient when rounding up those responsible for the program.

        link to this | view in chronology ]

  • identicon
    Anonymous Coward, 20 Aug 2013 @ 7:59am

    So Much Open Resistance

    It seems incredible at the amount of open resistance to tactics of Security agencies here at Techdirt, given the fact that we are all still speaking English and not some ancient Chinese or Mesopotamian dialect..

    link to this | view in chronology ]

  • identicon
    bb, 20 Aug 2013 @ 10:09pm

    Of course they're obviously not lying

    LOL.

    Of course they're obviously not lying, and it's like 10 times that amount...or more...they never lie.

    fool me three times...derrr

    link to this | view in chronology ]

  • identicon
    gmathol, 21 Aug 2013 @ 12:32am

    I remember the data ware house we build 10 years ago had to endure more queries than 20 million a month and those were build for business reporting.

    link to this | view in chronology ]


Follow Techdirt
Essential Reading
Techdirt Deals
Report this ad  |  Hide Techdirt ads
Techdirt Insider Discord

The latest chatter on the Techdirt Insider Discord channel...

Loading...
Recent Stories

This site, like most other sites on the web, uses cookies. For more information, see our privacy policy. Got it
Close

Email This

This feature is only available to registered users. Register or sign in to use it.