The Stasi's Tiny Torn-Up Analog Files Defeat Modern Digital Technology's Attempts To Re-Assemble East Germany's Surveillance Records
from the too-hard-for-today's-hardware dept
It is nearly 30 years since the wall separating East and West Berlin came down, and yet work is still going on to deal with the toxic political legacy of East Germany. As Techdirt readers are well aware, one of the defining characteristics of the regime in East Germany was the unprecedented -- for the time, at least -- level of surveillance inflicted on citizens by the Stasi (short for Staatssicherheitsdienst, or State Security Service). This led to the creation of huge archives holding dossiers about millions of people.
As it became clear that East Germany's government would fall, and that its long-suffering citizens would demand to know who had been spying on them over the years, Stasi officers began to destroy the most incriminating documents. But there were so many files -- a 2008 Wired article about them says they occupied 100 miles of shelving -- that the shredding machines they used started to burn out. Eventually, Stasi agents were reduced to tearing pages by hand -- some 45 million of them, ripping them into around 600 million scraps of paper.
After thousands of bags holding the torn sheets were recovered, a team working for the Stasi records agency, the body responsible for handling the mountain of paper left behind by the secret police, began assembling the pages manually. It was hoped that the re-assembled documents would shed further light on the Stasi and its deeper secrets. But it was calculated that it would take 700 years to deal with all the scraps of paper by hand. A computerized approach was devised by the Fraunhofer Institute, best-known for devising the MP3 format, and implemented following a pilot project. After some initial successes, the program has run into problems, as the Guardian reports:
A so-called ePuzzler, working with an algorithm developed by the Fraunhofer Institute and costing about €8m of [German] federal funds, has managed to digitally reassemble about 91,000 pages since 2013. However, it has recently run into trouble.
For the last two years, the Stasi records agency has been waiting for engineers to develop more advanced hardware that can scan in smaller snippets, some of which are only the size of a fingernail.
The ePuzzler works by matching up types of paper stock, typewriter fonts, or the outline of the torn-up page. It has struggled with hand-written files that were folded before being torn, leaving several snippets with near-identical outlines.
While the hardware engineers try to come up with a suitable scanner that can handle these tiny fragments, a small team continues to match up the more crudely ripped pages manually. Inevitably, some people will be thinking: "If only the Stasi had used blockchain, all these problems could have been avoided..."
Follow me @glynmoody on Twitter or identi.ca, and +glynmoody on Google+
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: digital technology, records, stasi, surveillance
Reader Comments
Subscribe: RSS
View by: Time | Thread
"unprecedented -- for the time, at least --" but nothing compared to Google and Facebook.
Both Google and Facebook have BILLIONS OF PERSONS INDEXED. And you think it's good, merely "commercial", and that gov'ts are able to resist detailed data. -- You are the nebbishes in this sci-fi dystopia.
"Inevitably, some people will be thinking: "If only the Stasi had used blockchain, all these problems could have been avoided..." -- ??? One of your wackiest attempts at "color". Or whatever.
[ link to this | view in chronology ]
Re: "unprecedented -- for the time, at least --" but nothing compared to Google and Facebook.
So tell me oh ignorant one, does Google make people disappear? Does Google slip a note in your pocket saying if you don't cooperate all your transgressions will be revealed to Command? Does Google threaten your family with harm if you don't comply?
To compare Google to the Stasi is the stupidest thing I've ever read.
[ link to this | view in chronology ]
Re: Re: "unprecedented -- for the time, at least --" but nothing compared to Google and Facebook.
To compare Google to the Stasi is the stupidest thing I've ever read.
It is not true that comparing Google to the Stasi is stupid.
(even though his comparison was)
Both collect(ed) large amounts of personal information.
Google uses it for purposes that are not necessarily evil, and may sometimes be beneficial to the subject.
The risk with Google is a 3rd (stasi -like) party getting access to the information and using it in an evil way.
[ link to this | view in chronology ]
Re: Re: Re: "unprecedented -- for the time, at least --" but nothing compared to Google and Facebook.
I dont think so.
[ link to this | view in chronology ]
Re: Re: "unprecedented -- for the time, at least --" but nothing compared to Google and Facebook.
I would categorize ad-tracking as being within the same wheel house. Perhaps, not to "command", but to any advertiser willing to pay.
Targeted advertising supplied by Google has probably outed more homosexuals than all the angry ex-girlfriends on the planet. Not to mention outing peoples medical conditions, and just about anything else a person might be embarrassed about.
The Stassi at least charged a serious bribe. Google outs people for fractions of a cent. Notably TD contributes to this since they use Googles analytics services.
Or do you not consider the causality of the TV advertising you see in other peoples homes?
[ link to this | view in chronology ]
Re: "unprecedented -- for the time, at least --" but nothing compared to Google and Facebook.
[ link to this | view in chronology ]
Re: Re: "unprecedented -- for the time, at least --" but nothing compared to Google and Facebook.
[ link to this | view in chronology ]
Compare this...
to the Documents from the Espionage Den, from 1979.
[ link to this | view in chronology ]
And at no point did any of them think 'this would be a lot easier with a couple gallons of petrol and a pack of matches'?
[ link to this | view in chronology ]
Re:
I was thinking that same thing. Did they not realize that 'fire' was a thing? A bonfire or 'accidental' building fire would have cleared the whole thing up real quick, yet they spent I don't even want to think how much time manually ripping up pages of highly flammable paper.
The Stasi: Great at collecting evidence, idiots when it comes to destroying it.
Still, I suppose people should be glad they were stupid in that way, as there's at least a chance to reconstruct ripped up pages, a possibility that doesn't work so well when it comes to ashes.
[ link to this | view in chronology ]
Re: Re:
[ link to this | view in chronology ]
Re: Re:
Likewise, a bonfire destroying the files would have sent a very clear signal, while Stasi was all about secrecy. Maybe Stasi officers weren't motivated anymore, which leads to worse performance at work. Or maybe they just couldn't afford the gas and/or the matches.
[ link to this | view in chronology ]
Re: Re:
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re:
To quote the Wired article linked by Glyn, "Burning all those files would tip off angry Berliners that something was up."
They were outnumbered, and worried about what would happen if the public found out what they had been doing. They wanted the most incrimining stuff destroyed before anyone found out they were destroying it.
[ link to this | view in chronology ]
Book scanning writ large
In Vernor Vinge's Rainbow's End there was a (somewhat) horrifying mechanism for scanning books: toss the books in a shredder and while the shredded pieces are floating down an air-filled channel they are photographed and then digitally reassembled. It certainly gets away from having to carefully scan each piece; I just wonder if the technique would require the re-shredding (further shredding?) of the Stasi papers to deal with the less than uniform existing pieces.
Once a workable solution is found, however, this will have very big implications for the SEC and FBI trying to recover documents after corporate shredding parties.
[ link to this | view in chronology ]
Re: Book scanning writ large
For example, as part of my work, I sometimes decommission servers and thus the disks inside them. My modus operandi is to hook those disks up, one at a time, via SATA-to-USB hardware, to a system that runs quite a random write passes on the entire drive. Then I open them up and expose the platters to a degausser. Then I remove the platters and feed them to a shredder. Then I separate the the shreds and dispose of them in multiple different places.
Good luck recovering anything from that, even with the resources of an intelligence agency. (Of course, if they were interested, they likely snagged all the data while the systems in question were still live.) The point is, though, that anyone who's trying to destroy data thoroughly has the means to do so, and it's much easier than dealing with miles of (paper) files.
[ link to this | view in chronology ]
Re: Re: Book scanning writ large
[ link to this | view in chronology ]
Re: Re: Re: Book scanning writ large
[ link to this | view in chronology ]
Re: Re: Book scanning writ large
In the modern era, yes, all the data/memos would be on servers, so getting rid of it is much easier than in the past (and easier to explain away -- oh! that project ended so we decommissioned the server with a sledgehammer...)
The issue is the so-called "paperless office." The reality is people still print things out -- it is just that they can re-print it at need. That is what they have shredding parties for.
[ link to this | view in chronology ]
Re: Re: Book scanning writ large
Even that isn't theoretically 100% guaranteed to render the data unrecoverable. It's enough for the limits of modern technology and the resources of pretty much any organization that exists, but in theory, a sufficiently-powerful organization could collect the scraps and shuffle through them and reassemble them (or scan them and reassemble the scanned data objects) and read the result.
If you really want to guarantee that no one will ever be able to read what used to be on a given hard drive, you need to pull out the platters and melt them, then stir the result together till it's unrecognizable.
Any entity capable of recovering the data from that would be so far into "indistinguishable from magic" that it could just as readily recover the data without access to the hard drive in the first place.
[ link to this | view in chronology ]
Re: Re: Book scanning writ large
Half a dozen rounds of .357 Magnum would make it difficult to recover any data...
[ link to this | view in chronology ]
Why?
[ link to this | view in chronology ]
Re: Why?
[ link to this | view in chronology ]
You gotta admit that those Stasi guys were bunch of morons
[ link to this | view in chronology ]