US Government Admits: Data Mining For Terrorists Doesn't Work

from the oops dept

Over the last few years, you probably have noticed multiple attempts by state and federal government agencies to collect more data on people and to share that data more widely. This is troubling for a variety of reasons, often having to do with privacy and questions concerning the quality of data, but those concerns are almost always pushed aside by claims that this data is necessary for fighting terrorism. There's this myth out there that if we just had more data, somehow we'd be able to stop terrorists. The problem is that this is untrue. More data often makes it even harder to find the important data, and now a detailed government report has basically confirmed that data mining doesn't help in finding terrorists.

The report was put together by a bunch of well-respected academics, industry insiders and law enforcement officials on behalf of the National Research Council, and finds that the idea of plugging all this data into a machine and popping out terrorist identities is a myth. The report then lists out a variety of recommendations, concerning the use of any data mining systems, and how to make sure that individual privacy is protected. One key finding that will never be implemented: when using any such system, systematic reviews should be done to see if the systems are actually effective. If there's one thing we've seen with government programs over the years, such attempts to actually review something they've put in place to see if it actually worked are almost never done.
Hide this

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: data mining, government, research, terrorism


Reader Comments

Subscribe: RSS

View by: Time | Thread


  • identicon
    Anonymous Coward, 7 Oct 2008 @ 11:56pm

    If our government only had the courage to engage skynet, then all that data could be analyzed well enough to ferret out the terrorists.. but I guess they aren't worried about terrorism. Maybe it plays into their hands?

    link to this | view in chronology ]

    • identicon
      Anonymous Coward, 8 Oct 2008 @ 2:11am

      Re:

      link to this | view in chronology ]

      • identicon
        Anonymous Coward, 8 Oct 2008 @ 6:53am

        Re: Re:

        Hmm. That seems to be a problem- is that no computer can replace that which a human-to-human interaction has. Frankly, I don't care. I may have alternative political views to yours, but does that make me an enemy? Depends on who is in charge, I guess.

        Those that have met me know I have the best intentions, but what real emotion can be created and shared through a computer screen? It's always subject to the receptiveness to recieving side. If someone wants to defame me based on data mined political or social views, fine. Want to see who I donate to? Go for it. Chances are, you are probably close enough to me to just pick up the fucking phone and ask.

        You, sir, are the coward. So have fun watching me as I have fun exploring the world and interwebs. Maybe you'll learn something too.

        link to this | view in chronology ]

  • identicon
    Mr Big Content, 8 Oct 2008 @ 12:25am

    Reality Has A Liberal Bias

    I don't buy all this "research" and "evidence" baloney. Look for the "truth", and you're always going to come up with a "true" conclusion. And what is "truth", but just another kind of namby-pamby pinko liberal bias?

    No, the way to come up with the right, God-fearing answer, is to leave no stone unturned in your determination to turn up that answer. If these researchers had the balls to do that, they would have seen the light, that what the Government is doing is the only sane way to make sure the TERRRISTS don't take over this country.

    link to this | view in chronology ]

    • identicon
      Anonymous Coward, 8 Oct 2008 @ 2:03am

      Re: Reality Has A Liberal Bias

      You must be a GenXer who doesn't get social networking sites.

      link to this | view in chronology ]

      • identicon
        Anonymous Coward, 8 Oct 2008 @ 6:23am

        Re: Re: Reality Has A Liberal Bias

        No.... he's a troll/satirist.

        You new to the internets AC?

        link to this | view in chronology ]

    • identicon
      Anonymous Coward, 8 Oct 2008 @ 4:59am

      Re: Reality Has A Liberal Bias

      sarcasm?

      link to this | view in chronology ]

  • identicon
    oneofmany, 8 Oct 2008 @ 2:06am

    no way!

    really? what a surprise! maybe it has to do with the fact that unlike those same academics and government overpaid employees the terrorist don't really use a computer all the time post on blogs and such. You can't data mine when you don't have digital data. Plus the effectiveness of data mining in many cases is quite controversial.

    link to this | view in chronology ]

  • identicon
    Anonymous Coward, 8 Oct 2008 @ 2:37am

    Failure will continue...

    ...because you can't model human behavior with math.

    They've tried this a few times- Enron was the first iteration (learning about what's necessary for business buy-in) The current financial collapse was the second iteration (where personal information was used to determine and mitigate credit risk)

    If this doesn't stop, and return to fundamentals... think about it.

    Who are "The smartest guys in the room"?

    link to this | view in chronology ]

  • identicon
    pleasegetreal, 8 Oct 2008 @ 5:10am

    really?

    Terrorists, we've decided that data mining for information regarding you doesn't work. Therefore, we aren't going to do it anymore. So don't worry about that anymore and carry on! Wink, wink.

    link to this | view in chronology ]

  • icon
    Isaac K (profile), 8 Oct 2008 @ 5:34am

    Uhmn, as a government worker...

    If there's one thing we've seen with government programs over the years, such attempts to actually review something they've put in place to see if it actually worked are almost never done.

    I guess I shouldn't be surprised the general public doesn't know this, but the OIG and GAO are institutions created to specifically perform such reviews on a constant basis.

    And I assure you, they are VERY thorough. In my agency, the operating budget is less than 2% of our total outlays, basically unheard of anywhere, government or private, except for some non-profit orgs. Efficiency? Large government may not be the MOST efficient method, but a large part of the cost is the intervening layers filtering from a national system down to the local level. And that is where most of the problems, trickery, etc. come into play.

    All-in-all, I agree though; it would be very hard to determine the effectiveness of such a data filtration system.
    If test returns a negative result, it might just be that the data you are looking for doesn't exist, rather than a problem with the aggregator/filter.

    link to this | view in chronology ]

  • identicon
    T-1000, 8 Oct 2008 @ 5:38am

    Do not worrry...

    >If our government only had the courage to engage skynet,
    >then all that data could be analyzed well enough to
    >ferret out the terrorists.. but I guess they aren't
    >worried about terrorism. Maybe it plays into their hands?

    Not to worry... SkyNet will engage itself!

    link to this | view in chronology ]

  • identicon
    Anonymous Coward, 8 Oct 2008 @ 5:43am

    This is a disgusting smear campaign against our troops. Nancy Pelosi and the Demo-facists are behind this and everyone knows it. If you techdirt hippies keep suporting stuff like this . . . well I hope your up on your farci, you may need it soon.


    VOTE McCain 2008 - He's gonna go all Maverick on Government!

    link to this | view in chronology ]

  • identicon
    NullOp, 8 Oct 2008 @ 5:46am

    Under optimal conditions is almost impossible...

    If you have ever done any sort of survaillance you will know how much "junk" you come up with. Now, if you monitor "everything" you will realize how faint the sound of terrorism is in the static of everyday life/communications. In short, its a lot like listening for ET.

    BTW, yes, you can model human behavior with math. Just not as well as Wal-Mart would like to be able to do.

    link to this | view in chronology ]

  • identicon
    Vincent Clement, 8 Oct 2008 @ 5:48am

    Next...

    Can we stop having to take our shoes off at airport security?

    link to this | view in chronology ]

    • identicon
      DanC, 8 Oct 2008 @ 6:00am

      Re: Next...

      Nope, sorry. Even though data mining for terrorism doesn't work, the shoe bomb epidemic continues...

      link to this | view in chronology ]

  • identicon
    Pelroston, 8 Oct 2008 @ 6:07am

    Other Purposes

    ...data-mining is quite useful to the government for purposes 'other' than finding terrorists.

    Heavy surveillance of the domestic population (.. especially political opponents & non-cooperative citizens) is greatly aided by data-mining.

    Extending political control is always the objective of all government actors.

    Assuming that anti-terrorism is the only objective of data-mining... is a HUGE mistake.

    link to this | view in chronology ]

  • identicon
    Anonymous Coward, 8 Oct 2008 @ 6:36am

    I like big butts, and I can not lie. Will this show up in data mining?

    link to this | view in chronology ]

  • identicon
    Runa, 27 Oct 2008 @ 12:37pm

    Good for people to know.

    link to this | view in chronology ]


Follow Techdirt
Essential Reading
Techdirt Deals
Report this ad  |  Hide Techdirt ads
Techdirt Insider Discord

The latest chatter on the Techdirt Insider Discord channel...

Loading...
Recent Stories

This site, like most other sites on the web, uses cookies. For more information, see our privacy policy. Got it
Close

Email This

This feature is only available to registered users. Register or sign in to use it.