Database CIA Claimed Too Difficult To Compile For FOIA Requesters Released In Full On CIA Website
from the three-years-of-fuckery dept
The CIA has millions of declassified records stashed away in Maryland -- something it claimed was accessible to the public. Actual access, however, resembles something out of Terry Gilliam's "Brazil," rather than what any reasonable person would call "accessible."
This so-called "publicly accessible" database -- known as CREST -- has been the target of MuckRock contributor Mike Best, who kickstarted an effort to liberate records from the vault through the use of manual labor. The records can be accessed by computer, but only certain computers, and only if you know exactly where to find them.
This is the CIA's "publicly available" records system front-end.
And here's how you locate it.
Accessing the information isn't easy. Researchers have to go to the back of the 3rd floor library at the National Archives building in Maryland, which is unfortunately unstaffed for half the day. Tucked away in the library are the only computers that can access the millions of pages of declassified records. If researchers ask the the main "Information" desk, they're answered with confused stares and incorrect directions. Researchers trying to look up on the National Archive's website where to access the computers, won't find it on the page about doing research at that location or on the page for electronic records at that location. That information is tucked away on the page for online databases - despite not being online.
Best's crowdfunding effort sought to free these documents from their four-computer cage. But the only way to do so was to visit in person, print out pages, and rescan them. All of this would be done under multiple forms of surveillance at the National Archive.
To expedite the process, MuckRock turned to litigation. Three years after commencing its FOIA lawsuit against the CIA, MuckRock has emerged victorious. The CIA has released the contents of the CREST vault online at its site, something it repeatedly claimed would take dozens of years and hundreds of thousands of dollars to complete.
The story behind the CIA's forced transparency is amazing. The claims made by the CIA during its opacity efforts are simply astounding. MuckRock has published a long, detailed recounting of its FOIA battle against the agency at its site and I wholeheartedly encourage you to click through and read it. But here are some of the highlights.
First off, everything in the database is in an unsearchable format by choice. The CIA only uses TIFF files, claiming that these are more resistant to alteration. But when faced with litigation, the CIA reversed course on the supposed hardiness of TIFF files.
The declaration... says that CIA cannot release these TIFF files in electronic form because they can be so easily altered by the mere act of a CIA FOIA analyst looking at them, and that the security measures they must take to remove this accidental metadata for an electronic release (involving editing each file separately by hand) would take 28 years and 1,200 CDs.
And there's where the hilarity begins. These are the DOJ's claims as to the difficulty of releasing the database it just released well ahead of its 28-year estimate. Not only is the process supposedly far too onerous to even begin to consider undertaking, but the DOJ claims the documents it just made available to the public at the CIA's website are not of public interest -- this despite being (technically) available to for public viewing at the National Archive for several years now.
In another filing, the CIA admits the 1,200 CDs that would take 28 years to compile have actually already been compiled -- and there are actually 1,450 CDs of records, but it will only consider releasing the quoted 1,200 if forced to. Unless the CIA is still porting these documents over to the CREST system at the National Archive, its claim of "28 years" looks even more ridiculous. From one of MuckRock's filings:
"Moreover, when [CIA] estimates that it would take 28 years just to create copies of the CREST database, it begs the question of how CIA loaded CREST in the first place. The system has only been operational since 2000.”
Then it claimed it would need $108,000 and six months to make copies of the CDs it already had in hand. And it doubled down on its contradictory claims about TIFF files, stating they were so easy to alter because they were so difficult to alter.
This lead to one of the best rebuttals (and lead-offs) in FOIA history:
I file a sur-reply drawing on my extensive experience as a person with a working brain, closing with the following thought:
“Last, CIA claims that ‘the act of a CIA employee opening a document on his or her terminal may cause metadata to embed itself on the image header.’ This is a frivolous statement for two reasons. First, files on PCs are not altered unless they are saved after the alteration. Simply opening a file and then closing it without changing it does not embed metadata on the file. Second, if this were a valid concern, it would apply to every file processed by the CIA FOIA office, not just CREST files. Since it clearly does not (which would paralyze the FOIA office), then the Court should view this claim very skeptically.”
Kel McClanahan -- the author of the post and MuckRock's legal rep for this case -- digs into the DOJ/CIA's repeated assertions about the "burdensomeness" of complying with this FOIA request. By the time he's done, everyone and everything is covered with government bullshit.
In 2000, using cutting edge 2000 technology, CIA populates CREST over a matter of months. For the next 15 years they insist that the only way to protect this system is to videotape people who want to access it. In 2015 they say that, using cutting edge 2015 technology, it will take them 28 years to make CD copies of CREST. Then they say that they already MADE CD copies of CREST in a matter of months if not weeks, meaning that it will only take 6 years to copy THOSE copies.
But they can copy those records to the web in 4 years. Less than a year later, they say that because of the huge public interest in CREST over the last several years (that they expressly disavowed less than 6 months before, did you remember?) they will have all of CREST online within a year because nobody needs to be videotaped any more. Then, less than 2 months later, they put all of CREST online.
The CIA and DOJ insist this stonewalling had nothing to do with the CIA's online release of CREST files. Apparently, it did all of this out of the goodness of its heart, forfeiting a shot at $100,000+ in FOIA fees and chance to do next to nothing for the next 30 years. It's obvious the effort to move these records online increased once it became apparent the government was going to come out on the losing end of these FOIA lawsuits (Jason Smathers was suing the CIA over the same database). Rather than be on the hook for legal fees, the CIA preempted any judicial judgment by releasing them to the public (the same public it said had "no interest" in these files) in bulk before final rulings could be made.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Reader Comments
The First Word
“Database Relativity
Apparently the Heisenberg uncertainty principle applies to FOIA requests: the more you know about a document's location, the less you are allowed to know about its contents.Subscribe: RSS
View by: Time | Thread
[ link to this | view in thread ]
You're not understanding the CIA's problem
But when a FOIA requester wants it, then suddenly there are all of the additional costs incurred to delay, obfuscate, redact or deny the request.
[ link to this | view in thread ]
[ link to this | view in thread ]
Lol
You know, it's believable that the CIA would have such an ill advised and non functional database with spastic metadata and that their IT guys aren't bright enough to use or fix it. This alone should make them wonder about their reputation.....
[ link to this | view in thread ]
Database Relativity
[ link to this | view in thread ]
A little confused...
Using cutting edge 2017 technology, how long would it take for them to make copies of CREST on Betamax?
[ link to this | view in thread ]
Re:
[ link to this | view in thread ]
CDs?
CD copies? CD's are 2015 cutting edge tech?
I'm surprised they didn't claim it would take a thousand years to write to all those 8" floppies...
[ link to this | view in thread ]
[ link to this | view in thread ]
Re:
Releasing data after it is clear how bad of a court case they have is, stalling to extort the victim and buy time to prepare a way to make the case disappear and then claiming it was all part of a plan is pretty much standard government practice.
Losing in court looks extremely bad and would create even worse precedence, forcing them to comply with the letter of the law instead of having a goto excuse for opacity. Preemptively complying to make the court case go away will be possible to spin into something harmless in the media since very few journalists bring anything from court documents (the journalist would be crucified by their overlords because of how boring the story will be.).
[ link to this | view in thread ]
Re:
[ link to this | view in thread ]
Re: Re:
[ link to this | view in thread ]
Public release should not get them off the hook for legal fees...
Muckrock, please file for your legal fees!
[ link to this | view in thread ]
Re: CDs?
[ link to this | view in thread ]
Re: CDs?
One 80-column punch card can hold about 72-bytes plus an 8-byte sequence number for keeping the cards in order (depending on the actual encoding format used).
so at ~700MB per CD, that's 10,194,489 cards per CD, multiplied by 1200 CD's is 1,223,386,800 punch cards which, at 56 cards/cm, would be a stack 2184km (1365 miles) high of machine-readable punch cards.
[ link to this | view in thread ]
Re: Re: CDs?
In fact, our card punch machines very often didn't have an inked ribbon, so you had to read the cards from the punches. I got really good at that. Of course, I was in heaven when I finally got an English computer account. Yes, English students had a better account than the engineering students. You got to use the VAX instead of the AS9000, which meant actual terminals for programming instead of punch cards, and you got six hours of time as opposed to one. Weird, but I guess there were fewer English students using accounts, so they had more resources to split fewer ways.
[ link to this | view in thread ]
Quite small actually
One cheap 1TB drive, or 19 BD-50 discs. This could of been solved in a few hrs by docking a USB drive. What asses!
[ link to this | view in thread ]
Re: Quite small actually
[ link to this | view in thread ]
Re: Re: Quite small actually
Other features of note:
"collections", but no information on whether the entire database has been filed in one (or more) collections, or whether there are items unindexed (save perhaps by ID number).
Nothing like a downloadable index into the archive. One might have success iterating through all possible ID numbers in order to see everything there is to see.
[ link to this | view in thread ]
Re: Re: Re: Quite small actually
[ link to this | view in thread ]
CIA is in the news a lot lately. Maryland's databases too.
Try an FOIA on RAF Croughton and sex trafficking (and what did the British minister find there?)- that would bring interesting results.
Awww never mind- you might just flip out like "Michael Hastings."
[ link to this | view in thread ]
Re: You're not understanding the CIA's problem
Stop torturing me. Stop torturing the cia. Stop bragging that you torture me and the cia enough to have been al qaeda designated for so many decades.
[ link to this | view in thread ]
[ ffwd 3:30 ]
https://www.youtube.com/watch?v=L4q6qBuSjHk
32:52
The Hitchhikers Guide to the Galaxy Episode 1
Pedro Torres
“But the plans were on display…”
“On display? I eventually had to go down to the cellar to find them.”
“That’s the display department.”
“With a flashlight.”
“Ah, well, the lights had probably gone.”
“So had the stairs.”
“But look, you found the notice, didn’t you?”
“Yes,” said Arthur, “yes I did.
It was on display
in the bottom of a locked filing cabinet
stuck
in a disused lavatory
with a sign on the door
saying
‘Beware of the Leopard.”
― Douglas Adams, The Hitchhiker's Guide to the Galaxy
[ link to this | view in thread ]
Re: The Hitchhikers Guide to the Galaxy Episode 1
[ link to this | view in thread ]