The Stasi's Tiny Torn-Up Analog Files Defeat Modern Digital Technology's Attempts To Re-Assemble East Germany's Surveillance Records

from the too-hard-for-today's-hardware dept

It is nearly 30 years since the wall separating East and West Berlin came down, and yet work is still going on to deal with the toxic political legacy of East Germany. As Techdirt readers are well aware, one of the defining characteristics of the regime in East Germany was the unprecedented -- for the time, at least -- level of surveillance inflicted on citizens by the Stasi (short for Staatssicherheitsdienst, or State Security Service). This led to the creation of huge archives holding dossiers about millions of people.

As it became clear that East Germany's government would fall, and that its long-suffering citizens would demand to know who had been spying on them over the years, Stasi officers began to destroy the most incriminating documents. But there were so many files -- a 2008 Wired article about them says they occupied 100 miles of shelving -- that the shredding machines they used started to burn out. Eventually, Stasi agents were reduced to tearing pages by hand -- some 45 million of them, ripping them into around 600 million scraps of paper.

After thousands of bags holding the torn sheets were recovered, a team working for the Stasi records agency, the body responsible for handling the mountain of paper left behind by the secret police, began assembling the pages manually. It was hoped that the re-assembled documents would shed further light on the Stasi and its deeper secrets. But it was calculated that it would take 700 years to deal with all the scraps of paper by hand. A computerized approach was devised by the Fraunhofer Institute, best-known for devising the MP3 format, and implemented following a pilot project. After some initial successes, the program has run into problems, as the Guardian reports:

A so-called ePuzzler, working with an algorithm developed by the Fraunhofer Institute and costing about €8m of [German] federal funds, has managed to digitally reassemble about 91,000 pages since 2013. However, it has recently run into trouble.

For the last two years, the Stasi records agency has been waiting for engineers to develop more advanced hardware that can scan in smaller snippets, some of which are only the size of a fingernail.

The ePuzzler works by matching up types of paper stock, typewriter fonts, or the outline of the torn-up page. It has struggled with hand-written files that were folded before being torn, leaving several snippets with near-identical outlines.

While the hardware engineers try to come up with a suitable scanner that can handle these tiny fragments, a small team continues to match up the more crudely ripped pages manually. Inevitably, some people will be thinking: "If only the Stasi had used blockchain, all these problems could have been avoided..."

Follow me @glynmoody on Twitter or identi.ca, and +glynmoody on Google+

Hide this

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: digital technology, records, stasi, surveillance


Reader Comments

Subscribe: RSS

View by: Time | Thread


  1. This comment has been flagged by the community. Click here to show it
    identicon
    Anonymous Coward, 8 Jan 2018 @ 7:51pm

    "unprecedented -- for the time, at least --" but nothing compared to Google and Facebook.

    Both Google and Facebook have BILLIONS OF PERSONS INDEXED. And you think it's good, merely "commercial", and that gov'ts are able to resist detailed data. -- You are the nebbishes in this sci-fi dystopia.


    "Inevitably, some people will be thinking: "If only the Stasi had used blockchain, all these problems could have been avoided..." -- ??? One of your wackiest attempts at "color". Or whatever.

    link to this | view in thread ]

  2. icon
    steell (profile), 8 Jan 2018 @ 8:46pm

    Re: "unprecedented -- for the time, at least --" but nothing compared to Google and Facebook.

    Dude, generally I refrain from replying to your posts, I just flag them and move on. But this one is posted from the depths of ignorance so great the bottom may never be found.

    So tell me oh ignorant one, does Google make people disappear? Does Google slip a note in your pocket saying if you don't cooperate all your transgressions will be revealed to Command? Does Google threaten your family with harm if you don't comply?

    To compare Google to the Stasi is the stupidest thing I've ever read.

    link to this | view in thread ]

  3. This comment has been flagged by the community. Click here to show it
    identicon
    Anonymous Coward, 8 Jan 2018 @ 9:32pm

    Re: "unprecedented -- for the time, at least --" but nothing compared to Google and Facebook.

    And yet you suck the cock of the NSA at every given opportunity because they help the RIAA with suing pirates.

    link to this | view in thread ]

  4. identicon
    Anonymous Coward, 8 Jan 2018 @ 9:32pm

    Compare this...

    link to this | view in thread ]

  5. identicon
    Agammamon, 8 Jan 2018 @ 10:31pm

    "Eventually, Stasi agents were reduced to tearing pages by hand"

    And at no point did any of them think 'this would be a lot easier with a couple gallons of petrol and a pack of matches'?

    link to this | view in thread ]

  6. icon
    That One Guy (profile), 8 Jan 2018 @ 11:14pm

    Re:

    I was thinking that same thing. Did they not realize that 'fire' was a thing? A bonfire or 'accidental' building fire would have cleared the whole thing up real quick, yet they spent I don't even want to think how much time manually ripping up pages of highly flammable paper.

    The Stasi: Great at collecting evidence, idiots when it comes to destroying it.

    Still, I suppose people should be glad they were stupid in that way, as there's at least a chance to reconstruct ripped up pages, a possibility that doesn't work so well when it comes to ashes.

    link to this | view in thread ]

  7. icon
    Ben (profile), 8 Jan 2018 @ 11:53pm

    Book scanning writ large

    In Vernor Vinge's Rainbow's End there was a (somewhat) horrifying mechanism for scanning books: toss the books in a shredder and while the shredded pieces are floating down an air-filled channel they are photographed and then digitally reassembled. It certainly gets away from having to carefully scan each piece; I just wonder if the technique would require the re-shredding (further shredding?) of the Stasi papers to deal with the less than uniform existing pieces.

    Once a workable solution is found, however, this will have very big implications for the SEC and FBI trying to recover documents after corporate shredding parties.

    link to this | view in thread ]

  8. identicon
    Daydream, 9 Jan 2018 @ 2:50am

    Re:

    As I was reading the article, I actually considered that a bucket of water might have worked. Swish the paper around in the water until it's disintegrated. Or did they use water resistant paper?

    link to this | view in thread ]

  9. identicon
    Anonymous Coward, 9 Jan 2018 @ 3:01am

    Re: Re: "unprecedented -- for the time, at least --" but nothing compared to Google and Facebook.

    Well, fairness, he also sucks Trump's rancid pus-oozing cock as well. Equal time, you know. ;)

    link to this | view in thread ]

  10. identicon
    Anonymous Coward, 9 Jan 2018 @ 3:07am

    Re: Book scanning writ large

    To address that last point: maybe. Files are increasingly kept in digital form, which makes them much easier to disappear, provided those doing so have a baseline level of competence.

    For example, as part of my work, I sometimes decommission servers and thus the disks inside them. My modus operandi is to hook those disks up, one at a time, via SATA-to-USB hardware, to a system that runs quite a random write passes on the entire drive. Then I open them up and expose the platters to a degausser. Then I remove the platters and feed them to a shredder. Then I separate the the shreds and dispose of them in multiple different places.

    Good luck recovering anything from that, even with the resources of an intelligence agency. (Of course, if they were interested, they likely snagged all the data while the systems in question were still live.) The point is, though, that anyone who's trying to destroy data thoroughly has the means to do so, and it's much easier than dealing with miles of (paper) files.

    link to this | view in thread ]

  11. icon
    Richard (profile), 9 Jan 2018 @ 3:18am

    Re: Re:

    Not burning the files is all part of their evil plan. When the German economy is flattened by all the effort deciphering them they can foment a new revolution and usher back a glorious communist future...

    link to this | view in thread ]

  12. icon
    Richard (profile), 9 Jan 2018 @ 3:20am

    Re: Re: Book scanning writ large

    Standard procedure in secure military circles is to etch the magnetic medium off the platters with a suitable solvent.

    link to this | view in thread ]

  13. icon
    Richard (profile), 9 Jan 2018 @ 3:59am

    Re: Re: "unprecedented -- for the time, at least --" but nothing compared to Google and Facebook.

    To compare Google to the Stasi is the stupidest thing I've ever read.

    It is not true that comparing Google to the Stasi is stupid.

    (even though his comparison was)

    Both collect(ed) large amounts of personal information.

    Google uses it for purposes that are not necessarily evil, and may sometimes be beneficial to the subject.

    The risk with Google is a 3rd (stasi -like) party getting access to the information and using it in an evil way.

    link to this | view in thread ]

  14. identicon
    Anonymous Coward, 9 Jan 2018 @ 4:06am

    Re: Re: Re: "unprecedented -- for the time, at least --" but nothing compared to Google and Facebook.

    I suppose you could claim the goog is a stooge but the staz?
    I dont think so.

    link to this | view in thread ]

  15. identicon
    Anonymous Coward, 9 Jan 2018 @ 4:21am

    Re: Re:

    Take a look at Nazi practices: they murdered millions in remote and isolate places. But not all places were remote, some even really close to villages, for example the psychiatry in Hadamar. The fetidness from the included crematory was very well discernible by villagers (they all played along anyway).

    Likewise, a bonfire destroying the files would have sent a very clear signal, while Stasi was all about secrecy. Maybe Stasi officers weren't motivated anymore, which leads to worse performance at work. Or maybe they just couldn't afford the gas and/or the matches.

    link to this | view in thread ]

  16. icon
    XcOM987 (profile), 9 Jan 2018 @ 4:38am

    Re: Re: Re: Book scanning writ large

    I just used to put a 6" nail through all the platters to shatter them before they went off to WEEE centers to be shredded. wouldln't stop someone seriously wanting the data, but would stop the opertune thief.

    link to this | view in thread ]

  17. icon
    scotts13 (profile), 9 Jan 2018 @ 4:59am

    Why?

    Call me naive, but wouldn't everyone be much better off if the files were finally and utterly destroyed? I understand a morbid curiosity, but as long as they exist, someone may yet use the files in an unfavorable manner. Start fresh, with a level playing field.

    link to this | view in thread ]

  18. icon
    JoeCool (profile), 9 Jan 2018 @ 6:23am

    Re: Why?

    The papers are evidence of crimes. Until they are all reassembled and examined by police, you can't be sure you caught all the criminals. After their worth as evidence is expended, then you can dispose of them properly.

    link to this | view in thread ]

  19. icon
    Toom1275 (profile), 9 Jan 2018 @ 6:24am

    Re: Re:

    Depends on the ashes. If they aren't burned properly and stirred afterward, you can get largish scraps of merely charred paper on which the ink is still fairly legible.

    link to this | view in thread ]

  20. icon
    Ben (profile), 9 Jan 2018 @ 7:08am

    Re: Re: Book scanning writ large

    In the modern era, yes, all the data/memos would be on servers, so getting rid of it is much easier than in the past (and easier to explain away -- oh! that project ended so we decommissioned the server with a sledgehammer...)

    The issue is the so-called "paperless office." The reality is people still print things out -- it is just that they can re-print it at need. That is what they have shredding parties for.

    link to this | view in thread ]

  21. icon
    The Wanderer (profile), 9 Jan 2018 @ 8:30am

    Re: Re: Book scanning writ large

    Even that isn't theoretically 100% guaranteed to render the data unrecoverable. It's enough for the limits of modern technology and the resources of pretty much any organization that exists, but in theory, a sufficiently-powerful organization could collect the scraps and shuffle through them and reassemble them (or scan them and reassemble the scanned data objects) and read the result.

    If you really want to guarantee that no one will ever be able to read what used to be on a given hard drive, you need to pull out the platters and melt them, then stir the result together till it's unrecognizable.

    Any entity capable of recovering the data from that would be so far into "indistinguishable from magic" that it could just as readily recover the data without access to the hard drive in the first place.

    link to this | view in thread ]

  22. identicon
    TRX, 9 Jan 2018 @ 11:51am

    Re: Re: Book scanning writ large

    In the late 1980s engineers from IBM's Atlanta facility used to decommission hard drives by taking them to an indoor shooting range. The management didn't care as long as they swept up the mess afterward.

    Half a dozen rounds of .357 Magnum would make it difficult to recover any data...

    link to this | view in thread ]

  23. identicon
    Anonymous Coward, 9 Jan 2018 @ 1:22pm

    Re: Re: "unprecedented -- for the time, at least --" but nothing compared to Google and Facebook.

    "Does Google slip a note in your pocket saying if you don't cooperate all your transgressions will be revealed to Command?"

    I would categorize ad-tracking as being within the same wheel house. Perhaps, not to "command", but to any advertiser willing to pay.

    Targeted advertising supplied by Google has probably outed more homosexuals than all the angry ex-girlfriends on the planet. Not to mention outing peoples medical conditions, and just about anything else a person might be embarrassed about.

    The Stassi at least charged a serious bribe. Google outs people for fractions of a cent. Notably TD contributes to this since they use Googles analytics services.

    Or do you not consider the causality of the TV advertising you see in other peoples homes?

    link to this | view in thread ]

  24. identicon
    Anonymous Coward, 9 Jan 2018 @ 6:21pm

    Re:

    And at no point did any of them think 'this would be a lot easier with a couple gallons of petrol and a pack of matches'?

    To quote the Wired article linked by Glyn, "Burning all those files would tip off angry Berliners that something was up."

    They were outnumbered, and worried about what would happen if the public found out what they had been doing. They wanted the most incrimining stuff destroyed before anyone found out they were destroying it.

    link to this | view in thread ]

  25. identicon
    Darkhog, 12 Jan 2018 @ 7:16pm

    You gotta admit that those Stasi guys were bunch of morons

    All they really had to do was to make a huge bonfire - good look recovering documents from ashes.

    link to this | view in thread ]


Follow Techdirt
Essential Reading
Techdirt Deals
Report this ad  |  Hide Techdirt ads
Techdirt Insider Discord

The latest chatter on the Techdirt Insider Discord channel...

Loading...
Recent Stories

This site, like most other sites on the web, uses cookies. For more information, see our privacy policy. Got it
Close

Email This

This feature is only available to registered users. Register or sign in to use it.