Database CIA Claimed Too Difficult To Compile For FOIA Requesters Released In Full On CIA Website

from the three-years-of-fuckery dept

The CIA has millions of declassified records stashed away in Maryland -- something it claimed was accessible to the public. Actual access, however, resembles something out of Terry Gilliam's "Brazil," rather than what any reasonable person would call "accessible."

This so-called "publicly accessible" database -- known as CREST -- has been the target of MuckRock contributor Mike Best, who kickstarted an effort to liberate records from the vault through the use of manual labor. The records can be accessed by computer, but only certain computers, and only if you know exactly where to find them.

This is the CIA's "publicly available" records system front-end.

And here's how you locate it.

Accessing the information isn't easy. Researchers have to go to the back of the 3rd floor library at the National Archives building in Maryland, which is unfortunately unstaffed for half the day. Tucked away in the library are the only computers that can access the millions of pages of declassified records. If researchers ask the the main "Information" desk, they're answered with confused stares and incorrect directions. Researchers trying to look up on the National Archive's website where to access the computers, won't find it on the page about doing research at that location or on the page for electronic records at that location. That information is tucked away on the page for online databases - despite not being online.

Best's crowdfunding effort sought to free these documents from their four-computer cage. But the only way to do so was to visit in person, print out pages, and rescan them. All of this would be done under multiple forms of surveillance at the National Archive.

To expedite the process, MuckRock turned to litigation. Three years after commencing its FOIA lawsuit against the CIA, MuckRock has emerged victorious. The CIA has released the contents of the CREST vault online at its site, something it repeatedly claimed would take dozens of years and hundreds of thousands of dollars to complete.

The story behind the CIA's forced transparency is amazing. The claims made by the CIA during its opacity efforts are simply astounding. MuckRock has published a long, detailed recounting of its FOIA battle against the agency at its site and I wholeheartedly encourage you to click through and read it. But here are some of the highlights.

First off, everything in the database is in an unsearchable format by choice. The CIA only uses TIFF files, claiming that these are more resistant to alteration. But when faced with litigation, the CIA reversed course on the supposed hardiness of TIFF files.

The declaration... says that CIA cannot release these TIFF files in electronic form because they can be so easily altered by the mere act of a CIA FOIA analyst looking at them, and that the security measures they must take to remove this accidental metadata for an electronic release (involving editing each file separately by hand) would take 28 years and 1,200 CDs.

And there's where the hilarity begins. These are the DOJ's claims as to the difficulty of releasing the database it just released well ahead of its 28-year estimate. Not only is the process supposedly far too onerous to even begin to consider undertaking, but the DOJ claims the documents it just made available to the public at the CIA's website are not of public interest -- this despite being (technically) available to for public viewing at the National Archive for several years now.

In another filing, the CIA admits the 1,200 CDs that would take 28 years to compile have actually already been compiled -- and there are actually 1,450 CDs of records, but it will only consider releasing the quoted 1,200 if forced to. Unless the CIA is still porting these documents over to the CREST system at the National Archive, its claim of "28 years" looks even more ridiculous. From one of MuckRock's filings:

"Moreover, when [CIA] estimates that it would take 28 years just to create copies of the CREST database, it begs the question of how CIA loaded CREST in the first place. The system has only been operational since 2000.”

Then it claimed it would need $108,000 and six months to make copies of the CDs it already had in hand. And it doubled down on its contradictory claims about TIFF files, stating they were so easy to alter because they were so difficult to alter.

This lead to one of the best rebuttals (and lead-offs) in FOIA history:

I file a sur-reply drawing on my extensive experience as a person with a working brain, closing with the following thought:

“Last, CIA claims that ‘the act of a CIA employee opening a document on his or her terminal may cause metadata to embed itself on the image header.’ This is a frivolous statement for two reasons. First, files on PCs are not altered unless they are saved after the alteration. Simply opening a file and then closing it without changing it does not embed metadata on the file. Second, if this were a valid concern, it would apply to every file processed by the CIA FOIA office, not just CREST files. Since it clearly does not (which would paralyze the FOIA office), then the Court should view this claim very skeptically.”

Kel McClanahan -- the author of the post and MuckRock's legal rep for this case -- digs into the DOJ/CIA's repeated assertions about the "burdensomeness" of complying with this FOIA request. By the time he's done, everyone and everything is covered with government bullshit.

In 2000, using cutting edge 2000 technology, CIA populates CREST over a matter of months. For the next 15 years they insist that the only way to protect this system is to videotape people who want to access it. In 2015 they say that, using cutting edge 2015 technology, it will take them 28 years to make CD copies of CREST. Then they say that they already MADE CD copies of CREST in a matter of months if not weeks, meaning that it will only take 6 years to copy THOSE copies.

But they can copy those records to the web in 4 years. Less than a year later, they say that because of the huge public interest in CREST over the last several years (that they expressly disavowed less than 6 months before, did you remember?) they will have all of CREST online within a year because nobody needs to be videotaped any more. Then, less than 2 months later, they put all of CREST online.

The CIA and DOJ insist this stonewalling had nothing to do with the CIA's online release of CREST files. Apparently, it did all of this out of the goodness of its heart, forfeiting a shot at $100,000+ in FOIA fees and chance to do next to nothing for the next 30 years. It's obvious the effort to move these records online increased once it became apparent the government was going to come out on the losing end of these FOIA lawsuits (Jason Smathers was suing the CIA over the same database). Rather than be on the hook for legal fees, the CIA preempted any judicial judgment by releasing them to the public (the same public it said had "no interest" in these files) in bulk before final rulings could be made.

Hide this

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: cia, crest, foia
Companies: muckrock


Reader Comments

Subscribe: RSS

View by: Time | Thread


  1. icon
    Ninja (profile), 24 Jan 2017 @ 10:44am

    I'm actually surprised they didn't decide to simply ignore the court orders or say that a catastrophic leakage of rain water destroyed the entire database.

    link to this | view in thread ]

  2. icon
    DannyB (profile), 24 Jan 2017 @ 10:58am

    You're not understanding the CIA's problem

    It's one thing to consider the cost of building the database.

    But when a FOIA requester wants it, then suddenly there are all of the additional costs incurred to delay, obfuscate, redact or deny the request.

    link to this | view in thread ]

  3. icon
    Anon E. Mous (profile), 24 Jan 2017 @ 11:07am

    Ah the Prenda strategy of Deny, Delay, Deflect...Oh how I missed seeing that. Nice of the CIA to bring back the strategy held to the hearts of those who dont want to turn over anything of relevance when asked for it.

    link to this | view in thread ]

  4. identicon
    Dr evil, 24 Jan 2017 @ 11:08am

    Schrödinger's Metadata
    Lol
    You know, it's believable that the CIA would have such an ill advised and non functional database with spastic metadata and that their IT guys aren't bright enough to use or fix it. This alone should make them wonder about their reputation.....

    link to this | view in thread ]

  5. icon
    TechDescartes (profile), 24 Jan 2017 @ 11:10am

    Database Relativity

    Apparently the Heisenberg uncertainty principle applies to FOIA requests: the more you know about a document's location, the less you are allowed to know about its contents.

    link to this | view in thread ]

  6. identicon
    Anonymous Hero, 24 Jan 2017 @ 11:18am

    A little confused...

    "In 2015 they say that, using cutting edge 2015 technology, it will take them 28 years to make CD copies of CREST."

    Using cutting edge 2017 technology, how long would it take for them to make copies of CREST on Betamax?

    link to this | view in thread ]

  7. identicon
    Anonymous Coward, 24 Jan 2017 @ 12:37pm

    Re:

    They must have been using Microsoft Access.

    link to this | view in thread ]

  8. icon
    Bamboo Harvester (profile), 24 Jan 2017 @ 12:38pm

    CDs?

    "In 2015 they say that, using cutting edge 2015 technology, it will take them 28 years to make CD copies of CREST."

    CD copies? CD's are 2015 cutting edge tech?

    I'm surprised they didn't claim it would take a thousand years to write to all those 8" floppies...

    link to this | view in thread ]

  9. identicon
    I.T. Guy, 24 Jan 2017 @ 12:44pm

    Why does our Government hate us?

    link to this | view in thread ]

  10. identicon
    Anonymous Coward, 24 Jan 2017 @ 1:02pm

    Re:

    The problem is that a "publically available" database is relatively easy to cover for media which eventually would force some politicians into claiming terrible things about their handling of data.

    Releasing data after it is clear how bad of a court case they have is, stalling to extort the victim and buy time to prepare a way to make the case disappear and then claiming it was all part of a plan is pretty much standard government practice.
    Losing in court looks extremely bad and would create even worse precedence, forcing them to comply with the letter of the law instead of having a goto excuse for opacity. Preemptively complying to make the court case go away will be possible to spin into something harmless in the media since very few journalists bring anything from court documents (the journalist would be crucified by their overlords because of how boring the story will be.).

    link to this | view in thread ]

  11. identicon
    Shilling, 24 Jan 2017 @ 1:11pm

    Re:

    Because you are a threat to their existence.

    link to this | view in thread ]

  12. identicon
    Anonymous Coward, 24 Jan 2017 @ 1:52pm

    Re: Re:

    No, that's the FEC.

    link to this | view in thread ]

  13. identicon
    Christenson, 24 Jan 2017 @ 2:15pm

    Public release should not get them off the hook for legal fees...

    As there's been a change in the legal relationship, and justice requires it.

    Muckrock, please file for your legal fees!

    link to this | view in thread ]

  14. icon
    JoeCool (profile), 24 Jan 2017 @ 2:35pm

    Re: CDs?

    CD? What's that? Seriously, has anyone used a CD for computer storage in the last decade?

    link to this | view in thread ]

  15. icon
    Eldakka (profile), 24 Jan 2017 @ 5:04pm

    Re: CDs?

    Wonder how long they'd claim for machine-readable punch cards?

    One 80-column punch card can hold about 72-bytes plus an 8-byte sequence number for keeping the cards in order (depending on the actual encoding format used).

    so at ~700MB per CD, that's 10,194,489 cards per CD, multiplied by 1200 CD's is 1,223,386,800 punch cards which, at 56 cards/cm, would be a stack 2184km (1365 miles) high of machine-readable punch cards.

    link to this | view in thread ]

  16. icon
    JoeCool (profile), 24 Jan 2017 @ 6:26pm

    Re: Re: CDs?

    Your punch cards had a sequence number? How ADVANCED they were! Ours had no such thing - if you got them out of order, you were sh_t out of luck. Funniest and saddest thing I ever saw was a grad student wheeling in a load of punch cards for his project (had to have been at least 10,000 cards) only for the cart to overturn spilling cards EVERYWHERE. The poor guy cried for almost an hour while picking up cards. I kept mine in bundles of 100 held together with rubber bands and a note telling me which group was which.

    In fact, our card punch machines very often didn't have an inked ribbon, so you had to read the cards from the punches. I got really good at that. Of course, I was in heaven when I finally got an English computer account. Yes, English students had a better account than the engineering students. You got to use the VAX instead of the AS9000, which meant actual terminals for programming instead of punch cards, and you got six hours of time as opposed to one. Weird, but I guess there were fewer English students using accounts, so they had more resources to split fewer ways.

    link to this | view in thread ]

  17. icon
    afn29129 (profile), 24 Jan 2017 @ 7:03pm

    Quite small actually

    So the total collection is about 945 GB. Quite small actually.

    One cheap 1TB drive, or 19 BD-50 discs. This could of been solved in a few hrs by docking a USB drive. What asses!

    link to this | view in thread ]

  18. identicon
    Anonymous Coward, 24 Jan 2017 @ 8:10pm

    Re: Quite small actually

    Consider: If the files are TIFF, then converting them to text via OCR should reduce the size to about that many MB.

    link to this | view in thread ]

  19. identicon
    Anonymous Coward, 24 Jan 2017 @ 8:21pm

    Re: Re: Quite small actually

    I stand corrected. At least the file I viewed was a relatively low-res PDF image. Perhaps not quite designed to foil OCR, but definitely designed to require manual review and dissection.

    Other features of note:
    "collections", but no information on whether the entire database has been filed in one (or more) collections, or whether there are items unindexed (save perhaps by ID number).

    Nothing like a downloadable index into the archive. One might have success iterating through all possible ID numbers in order to see everything there is to see.

    link to this | view in thread ]

  20. icon
    afn29129 (profile), 24 Jan 2017 @ 9:12pm

    Re: Re: Re: Quite small actually

    " iterating through all possible ID numbers..." URL hacking? oh my! :}

    link to this | view in thread ]

  21. identicon
    Celeste Guarini, 25 Jan 2017 @ 1:48am

    CIA is in the news a lot lately. Maryland's databases too.

    Must be part of a pre-emptive and self protective PR push- the CIA is anticipating other 'leaks' about the special material that can be found in it's databases- and that material stretching from Thailand to Maryland, to Virginia like a big, baby eating HYDRA/octopus.

    Try an FOIA on RAF Croughton and sex trafficking (and what did the British minister find there?)- that would bring interesting results.

    Awww never mind- you might just flip out like "Michael Hastings."

    link to this | view in thread ]

  22. identicon
    Anonymous Coward, 25 Jan 2017 @ 5:44pm

    Re: You're not understanding the CIA's problem

    One problem is wmds (chemical/biological). Another problem is incapacitation due to mind control(particle beam weapons). Then there is cia torture.

    Stop torturing me. Stop torturing the cia. Stop bragging that you torture me and the cia enough to have been al qaeda designated for so many decades.

    link to this | view in thread ]

  23. identicon
    Anonymous Coward, 25 Jan 2017 @ 6:35pm

    'plus ça change, plus c'est la même chose' dept

    [ ffwd 3:30 ]
    https://www.youtube.com/watch?v=L4q6qBuSjHk
    32:52
    The Hitchhikers Guide to the Galaxy Episode 1
    Pedro Torres

    “But the plans were on display…”
    “On display? I eventually had to go down to the cellar to find them.”
    “That’s the display department.”

    “With a flashlight.”
    “Ah, well, the lights had probably gone.”
    “So had the stairs.”

    “But look, you found the notice, didn’t you?”

    “Yes,” said Arthur, “yes I did.

    It was on display
    in the bottom of a locked filing cabinet
    stuck
    in a disused lavatory
    with a sign on the door
    saying
    ‘Beware of the Leopard.”

    ― Douglas Adams, The Hitchhiker's Guide to the Galaxy

    link to this | view in thread ]

  24. identicon
    Anonymous Coward, 25 Jan 2017 @ 6:49pm

    Re: The Hitchhikers Guide to the Galaxy Episode 1

    So what your saying, is that the CIA's SOP is a book that was written while tripping on acid... Yep, sounds about right.

    link to this | view in thread ]


Follow Techdirt
Essential Reading
Techdirt Deals
Report this ad  |  Hide Techdirt ads
Techdirt Insider Discord

The latest chatter on the Techdirt Insider Discord channel...

Loading...
Recent Stories

This site, like most other sites on the web, uses cookies. For more information, see our privacy policy. Got it
Close

Email This

This feature is only available to registered users. Register or sign in to use it.