Justice Dept. Official: We Could Get Lois Lerner's Emails From Backups, But It's Too Hard So Naaaaaah

from the facepalm dept

Wed, Aug 27th 2014 4:04am — Timothy Geigner

I try not to go for conspiracy theories generally, but this ongoing IRS nonsense involving conveniently disappearing emails potentially pertaining to the scandal involving targeting certain groups is making my skeptics beacon go off. The official story essentially involves a computer (server?) crash that obliterated the email data of several email accounts that would otherwise be of great interest to those trying to figure out who in the Obama administration knew what about how the IRS was operating. That crash somehow also involves the destruction of any local backups these IRS folks are required to keep as part of their job.

But what I imagine pretty much everyone that has even a modicum of interest in office technology is asking is what about the kind of backups that are typically run for disaster recovery and offsite purposes. You know, like tape drives (blech), D2D backups, or remote storage. It's difficult to believe that all copies of the data for those email accounts had somehow been disappeared by accident. Or, really, disappeared entirely at all.

As it turns out, there is backup data for those email accounts, but if you're thinking that we're finally going to get to the bottom of this nonsense, you're a silly naive person that hasn't witnessed our glorious government at work.

“A Department of Justice attorney told a Judicial Watch attorney on Friday that it turns out the federal government backs up all computer records in case something terrible happens in Washington and there’s a catastrophe, so the government can continue operating,” Judicial Watch president Tom Fitton told Fox News’s Shannon Bream. “But it would be too hard to go get Lois Lerner’s e-mails from that backup system."

Now, I know, Fox News and blah blah blah, but this is a claim worth paying attention to mostly because it only makes sense to begin with. Well, the part about there being backups, I mean. The suggestion that such backups are so difficult to parse and recover that the administration isn't going to get them for review makes zero sense because that's what damned backups are for.

Think about the logic here for a moment. The reason the federal government has backups of data in place is because there's a great deal of horrible crap that could happen to their facilities. Natural disasters, terrorist attacks, hallmark-level incompetence, Area 51 aliens going Rambo on the computers. These are the things they have to prepare for and they need the ability to restore data from a remote location should one of their facilities fall to our alien overlords. To say that restoring that data is too hard to do when requested is to say that they're ill-prepared for a disaster. The suggestion that the government can't retrieve emails for court or congressional review is to say that they wouldn't even meet FINRA compliance, something that itsy-bitsy little trading firms are required to meet.

Fitton said his group plans to ask a federal judge to order the IRS to hand over the e-mails, which conservative opponents of Lerner want to see in order to determine if there is a link between President Obama’s team and the IRS’s targeting of tea-party groups.

“If this backup system is working, Lois Lerner’s e-mails are there,” Fitton said.

And if the backup system isn't working, then fire everyone everywhere because this kind of thing is no joke. I'd say it'd be a bigger scandal to have any portion of the federal government not have basic backup and DR systems in place than to find out that any portion of the Executive administration was encouraging the IRS to target certain special interest groups.

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: backups, doj, email, irs, justice department, lois lerner

85 Comments

If you liked this post, you may also be interested in...

Reader Comments

Subscribe: RSS

View by: Time | Thread

Ninja (profile), 27 Aug 2014 @ 3:48am

Mike, you underestimate the Government. From my experience there are places where these e-mails WILL vanish if the account holder doesn't have a personal, local backup. Cross your fingers and hope there's no fire in the place.
[ link to this | view in thread ]
Anonymous Coward, 27 Aug 2014 @ 4:11am

Oh, the stories I could tell you about backups
Fortune 100 corporation: no offsite backups.

Major university: backups readable by everyone on the network, i.e., Alice could retrieve Bob's files.

Another Fortune 100 corporation: backups weren't readable. Guess how they found out.

And on and on and on. Most organizations are quite happy to buy overpriced overblown overhyped horribly insecure backup "solutions" from greedy vendors, deploy them, and mark the problem as "solved". Few -- precious few -- are actually willing to engineer solid solutions based on well-tested well-documented code (e.g., dump, tar, rsync and other basic tools) and then actually perform ongoing monitoring (as in "read the logs") and testing (as in "see if restoration is possible").

The overall state of backups across the world is best classified as "miserable".
[ link to this | view in thread ]
Anonymous Coward, 27 Aug 2014 @ 4:31am

If too hard to restore then they are useless
Your backups must be tested
So you know they work as expected
Offline is best
So you can rest
When lightening strikes unexpected
[ link to this | view in thread ]
Anonymous Coward, 27 Aug 2014 @ 4:33am

And Steve Stockman is subpoenaing the NSA to see if they have access to the emails or related metadata:
http://stockman.house.gov/media-center/press-releases/stockman-requests-subpoena-of-nsa-s-w hite-house-irs-phone-logs

TL,DR; Stockman and his ilk only oppose NSA surveillance when it benefits them personally to do so. When it incriminates someone they don't like (Fruit of the Poison Tree, Anyone?) they're all for it!
[ link to this | view in thread ]
That Anonymous Coward (profile), 27 Aug 2014 @ 4:43am

Can we just execute whoever slipped in the pork that was kicked back to the provider?

If it is to hard to get the data back, the ENTIRE thing is a waste and we should file suit. Anyone involved in continuing the contract should be assumed to be at best incompetent at worst bought off.

Is it time that they stop digging into our data & lives and maybe consider they should look in a mirror and fix the problems there.

First talking head that tries to lay this at the presidents feet gets shot.
This is decades of corruption and stupidity coming to light, and they need to find the will to fix it without finding a way to line their own pockets on the deal.
[ link to this | view in thread ]
Anonymous Coward, 27 Aug 2014 @ 5:09am

sounds very similar to the piss poor excuses used by the City of London Police when asked via FOIA for info on contacts between them and the entertainment industries.
[ link to this | view in thread ]
Anonymous Coward, 27 Aug 2014 @ 5:17am

Re:
Mike?
[ link to this | view in thread ]
Anonymous Coward, 27 Aug 2014 @ 5:17am

Well If they can't manage to do their jobs maybe they need replaced and jailed for obstruction of justice, When you I can be fined by the MPAA/RIAA into and beyond poverty for sharing files , Am I being unreasonable to ask for these guys to get the same treatment or worse for this offense .
[ link to this | view in thread ]
MadAsASnake (profile), 27 Aug 2014 @ 5:18am

Come on, really. A server crashed that took out the mail only of those of interest? Really? Lois Lerner's e-mails have been lost as a matter of policy, not fact.
If it ever becomes more useful to have those, they will magically be found never to have been lost at all.
[ link to this | view in thread ]
Anonymous Coward, 27 Aug 2014 @ 5:23am

A moot point?

Of what use are these emails when it is a well known fact that the IRS investigated applicants from all political persuasions with somewhat equal vigor? In addition, it has been reported that the only organization denied their coveted status was of a liberal slant. Add to this, that the status is not even needed in order to initially claim the exemption.

It is a side show, like Benghazi, trumped up pile on bullshit mountain.
[ link to this | view in thread ]
John Cressman, 27 Aug 2014 @ 5:26am

Purposeful
This is totally purposeful. As an IT professional for most of my life, there is no way that data wasn't backed up and no way that the backups are saved SOMEWHERE - normally - ESPECIALLY for the IRS.

You can be DARN SURE that if those were emails that had to do with auditing some poor sap and ruining his life that they would be found INSTANTLY.

The IRS is the largest, most well-funded and most brutal terrorist organization in the world. Never forget that.
[ link to this | view in thread ]
That One Guy (profile), 27 Aug 2014 @ 5:27am

Well, it's easy to see how eager and willing they are to get to the bottom of this whole mess, that's for sure. /s
[ link to this | view in thread ]
Anonymous Coward, 27 Aug 2014 @ 5:31am

I think we need a complete breakdown of how taxpayer money is being spent , How about an audit of the IRS.
[ link to this | view in thread ]
chad holbrook, 27 Aug 2014 @ 5:51am

Mail server blues
The only explanation that has ever made any sense to me (I’ve done tech support for almost 20 years) is this.

I’ve read in the news that the IRS uses a Microsoft Exchange system for email. Each user gets X amount of data storage on the server that is backed up. The user goes over that limit of X. There are two solutions to this in general. The first is increase the size of the storage on the server and the back system. The second is to set up a personal storage file (.pst) on the local computer in that person’s office. The user (with the help of a tech, usually) chooses what emails are moved from the server to the local file. The office systems are not usually backed up. If the second option was deployed by the user or tech staff and later on that office system’s hard drive crashed… well, then the data could be gone for good.

Now, they say that there is a backup? If so, I have no idea how they’ve set up this system.
[ link to this | view in thread ]
GeeC, 27 Aug 2014 @ 6:08am

Re: Mail server blues
Even if the email were removed from the server and stored locally via a .pst archive, the emails would still exist on backups that were created in the time-frame that those emails were on the server. They may have to look through a lot of old tapes, but that is their fault for not storing the .pst in another location on the network that also gets backed up.

This should be a lesson to anyone who isn't tech-savvy in today's world to learn the basics of how these things work so they can filter out the bullshit these people are feeding us.
[ link to this | view in thread ]
nasch (profile), 27 Aug 2014 @ 6:16am

Re:
It is a side show, like Benghazi, trumped up pile on bullshit mountain.

This particular story is not about the merits of the investigation, but the IRS response to it. The target of a Congressional investigation cannot be the party that decides whether it's legitimate.
[ link to this | view in thread ]
STJ, 27 Aug 2014 @ 6:19am

"And if the backup system isn't working, then fire everyone.." When was the last time there was a firing for incompetence within the government? If we did that, all we would have is very clean offices with no one in them.

Because the janitors would be the only ones working..
[ link to this | view in thread ]
Anonymous Coward, 27 Aug 2014 @ 6:24am

Typical and Far Too Common
I am an engineer employed by the federal government. Permit me to explain to you the convoluted nature of how I have to back up my data on my issued machine.

In order to make a backup, I have to plug an external drive into my machine and manually run my backup software, rather than having an automated backup to the network. I start a obscure piece of software that I have been directed to use to make backups. I then start the process, which basically locks me out of actually using my machine until the backup is complete. There is a strong encryption requirement on any data put onto an external drive, which means that making a backup is nearly an all-day affair to allow my (somewhat underpowered) machine to crank through all the encryption. I can't leave the software to run unattended, though, because it is not quite smart enough to handle that. I can finally use my machine again once the whole process is completed.

I usually line up some work to do in the lab or on my other computers when I intend to run a backup on my primary machine. Otherwise, the day tends to drag.

I had to jump through hoops just to be able to do this. You see, I was not issued the external drive. Most of the machines are set up to *not* allow backups to external drives. If I did not have an external drive, I would have to use optical media. We also only recently (within the last two years) received DVD burners as standard in our machines. I suppose they think we are just supposed to back up to an assload of CDs, like anyone will do that with any regularity.

The trouble is, the backup and restore process should be easy. That's how you get people to actually make the backups they're supposed to! That's how you engineer things so that the records aren't lost when a hard drive dies. However, making the process easy would make sense and be efficient, so there is little chance of it happening.

On another note, I am not convinced that the emergency backup alluded to will be of any use. If the IRS system is set up similar to mine, then they probably only have 1995 levels of email storage (25MB or so) on the server that would be backed up at any one time. Either that or the files would mysteriously disappear before they were restored.
[ link to this | view in thread ]
That One Guy (profile), 27 Aug 2014 @ 6:27am

Re:
That could be funny, we'd get to watch lobbyists try and bribe janitors to clean the offices of the ex-politicians they were trying to influence 'extra thoroughly'.
[ link to this | view in thread ]
Pooad, 27 Aug 2014 @ 6:27am

Everyone who thinks there are backups are wrong
This subject has been covered in depth, so I don't understand how misinformation like this goes around.

Commenter Chad Holbrook hit the nail on the head. It has already been well established that Lerner saved her email to a PST file on her local computer because their Exchange mailbox size limit is so low. The hard drive on her workstation crashed, was disposed of, and was replaced. Her workstation was NOT being backed up.

Yes, those emails were on the Exchange server for a period of time, but backup tapes get recycled and overwritten.

Those emails are irretrievably lost.
[ link to this | view in thread ]
Danny (profile), 27 Aug 2014 @ 6:28am

Backups. We don't need no stinkin' backups.
I used to work for a company you all have heard of. (Check my LinkedIn profile if you really care which one.) Back in the day (October 1989) our Bay Area office was destroyed by an earthquake you may also have heard about. Live and backup data within the office was destroyed. The policy at the time (now policy is very different) was not to rely on any local offsite backup facility as that could just as easily be destroyed by a quake. Rather, the office had a deal with a taxi driver to come every evening after daily backups and carry around a copy in the trunk of his cab. Next night he'd be back to swap disk packs and would carry the new copy.

The office recovered from the quake using data from the disk packs in his cab.

As to Tim's post: I'm a Democrat who has supported Obama from early on (though break with him on several issues of TechDirt focus). Despite my general support, I agree with everything Tim wrote--something does not smell right here. They do need to recover those emails no matter what is on them.
[ link to this | view in thread ]
Poodad, 27 Aug 2014 @ 6:34am

Re: Everyone who thinks there are backups are wrong
Oh, and to the folks who say the PST files should be on a network share so they would be backed up: good idea, but no dice. Microsoft specifically says PST files over a network link is a no-no.
[ link to this | view in thread ]
chad holbrook, 27 Aug 2014 @ 6:42am

Re: Typical and Far Too Common
Very good points. However, you state that you had to “jump through hoops” to get this system set up. I cannot imagine a high level executive of any company allowing their computer to be locked out of their systems for a full day or sit and hit “enter” at regular intervals. You’re right. Encryption and backup should be easy for the end user but you state that it’s not in your experience in the federal government. In my mind, you’re strengthening the argument that there is likely no backup of the local office system.

As for going into older tapes to get the data. Yes, that is entirely possible. However, if those tapes were being recycled and reused (common practice in a LOT of businesses) then the data could have been destroyed at any point.
[ link to this | view in thread ]
Dark Helmet (profile), 27 Aug 2014 @ 6:42am

Re: Mail server blues
"The user (with the help of a tech, usually) chooses what emails are moved from the server to the local file. The office systems are not usually backed up. If the second option was deployed by the user or tech staff and later on that office system’s hard drive crashed… well, then the data could be gone for good."

That's not typically how email backups work, in my experience. Usually, any competent organization is going to be doing some flavor of email archiving, in which a journaled Exchange account funnels copies of all email traffic to the archiving system, which is separate from the Exchange box/cluster. The user CANNOT control what is backed up by design. No system which allowed that would meet the many compliance standards the government and SGOs have set up for various industries.

As a device on the network, that message archiver is also backed up, along with the Exchange box/cluster. Backups of any legitimacy involve a local and at least one offsite location. This is harder with tape drives, which is why tape drives should go away and die forever, but there should always be at least 3-4 geographic points from where emails can reside. For all of those to disappear isn't believable.
[ link to this | view in thread ]
jimbo, 27 Aug 2014 @ 6:58am

Backups & DR
I used to work in IT for banks. We maintained a 3-day rotation on our DR backups, and tested the business recovery process every year. Understand, however, that the DR system is designed to recover the business to YESTERDAY's work. If it was deleted last week, it won't be on yesterday's backup, and likely won't be needed to recover the business.
[ link to this | view in thread ]
MadAsASnake (profile), 27 Aug 2014 @ 7:00am

Re: Backups. We don't need no stinkin' backups.
You know - a really obvious location for backup data is a reliable cloud hoster. Pity Hollywood is putting so much effort into destroying all of those because ** piracy **
[ link to this | view in thread ]
Dark Helmet (profile), 27 Aug 2014 @ 7:10am

Re: Re: Backups. We don't need no stinkin' backups.
There are a TON of good local/offsite D2D backup solutions. At my 8-5 job, we have used a couple of them to power a managed BDR service. We've switched them around a couple of times, so this isn't to pimp the Digerati Group service, but they've included the Barracuda Backup solution, Datto, and Veeam for virtualized environments. They all include onsite/offsite backups and can work with a message archiver for journaled Exchange accounts.

This stuff is waaaaaay more simple than people are led to believe. The idea of data residing primarily on a local client instead of a server and having the responsibility for backups foisted upon common users is laughable to the extreme....
[ link to this | view in thread ]
ltlw0lf (profile), 27 Aug 2014 @ 7:10am

Re: Oh, the stories I could tell you about backups
The overall state of backups across the world is best classified as "miserable".

As an administrator of a network professionally, and as a volunteer tech support person on the side (who in this business doesn't hold that title,) I wholly agree.

However, that being said, it is amazing the power of TAR/DAR (or even 7z, for those Windoze aficionados,) and external USB hard drives. They've saved me from some really serious stuff, just as long as you write to multiple hard drives, and periodically test your backups on a non-production system.

Still miserable ghetto-tech, but it would have helped here, if it wasn't in the best interest of those involved (allegedly) that it not work.
[ link to this | view in thread ]
wayout, 27 Aug 2014 @ 7:15am

Re:
Well, if it's such a moot point, then why all the lying & stalling about the emails, simply release them and shut up the critics...case solved..let them prove it is nothing but a witch hunt...make them look bad...if they can...course..if the emails back up those claims...then what..it's a plant by the extreme right I guess..
[ link to this | view in thread ]
nasch (profile), 27 Aug 2014 @ 7:18am

Re: Everyone who thinks there are backups are wrong
Those emails are irretrievably lost.

Except now the IRS says they're backed up, but too hard to get to.
[ link to this | view in thread ]
nasch (profile), 27 Aug 2014 @ 7:19am

Re: Re: Mail server blues
Usually, any competent organization

I think I see your error. ;-)
[ link to this | view in thread ]
Ben (profile), 27 Aug 2014 @ 7:20am

Re: Oh, the stories I could tell you about backups
Which is why:
And if the backup system isn't working, then fire everyone everywhere because this kind of thing is no joke.
Having "backup" just to check a box that says "yes we backup our systems" means nothing if the process isn't tested. Disaster Recovery is a level above basic backup, but for critical systems (and can you reasonably claim that the IRS isn't critical?) it has to be taken seriously. Fire the IT department; if they can't do it in-house, outsource it; there are companies that do this for a living.
[ link to this | view in thread ]
tomczerniawski, 27 Aug 2014 @ 7:21am

Too hard?

Heck, I'll do it for them. Dear IRS, I volunteer. I'll do it pro bono.
[ link to this | view in thread ]
Anonymous Coward, 27 Aug 2014 @ 7:21am

Re: If too hard to restore then they are useless
If you don't test your backups... you don't have backups!
[ link to this | view in thread ]
Anonymous Coward, 27 Aug 2014 @ 7:23am

Re:
Until the critters in congress start saying the words understand that they are collusive in the problem.

There is a big reason that they do not call each other out for actual corruption... they are all corrupt! That is off limits!
[ link to this | view in thread ]
wayout, 27 Aug 2014 @ 7:27am

pst's over a network link tend to corrupt very easily, I know from experience.
[ link to this | view in thread ]
Anonymous Coward, 27 Aug 2014 @ 7:31am

Here's the problem, they are too busy archiving *the people* of America's emails so when they go looking for "Lois Lerner" it returned multiple accounts and they don't want to sift though them all.
[ link to this | view in thread ]
Anonymous Coward, 27 Aug 2014 @ 7:47am

Re:
Sounds like a plan.
[ link to this | view in thread ]
Anonymous Coward, 27 Aug 2014 @ 7:49am

Re: Re: Oh, the stories I could tell you about backups
I don't know if I'd classify it as "ghetto tech". I work with highly sensitive data (PHI) and while we do daily backups of all our servers to a centralized backup repository, we also do periodic full backups of everything and stash those on encrypted compressed external hard drives which are kept off-site.

And yes, we test them. ;)

That's why in all the years I've been here our success rate at restoring files is 100.00%. And why, if the building burned to the ground tomorrow, I'm confident that we could restore our operation to the state that it was in as of the beginning of the month. (Which, for what we do, is acceptable.)

But I'm painfully well aware that we're one of the few exceptions in a sea of mediocrity and incompetence.
[ link to this | view in thread ]
John Fenderson (profile), 27 Aug 2014 @ 7:54am

Re: Purposeful
This does seem most likely to intentional. However...

"there is no way that data wasn't backed up and no way that the backups are saved SOMEWHERE - normally - ESPECIALLY for the IRS."

I think that's a step too far. Given that tons of companies -- including major corporations and tech companies -- totally fail in terms of backing up their stuff, and we're talking about the IRS here (who has blown through literally billions of dollars in failed attempts to modernize their computer systems), the idea that their backup systems don't work adequately is not beyond the realm of possibility.
[ link to this | view in thread ]
MadAsASnake (profile), 27 Aug 2014 @ 8:03am

Re: Re: Re: Backups. We don't need no stinkin' backups.
Don't tell Hollywood...
[ link to this | view in thread ]
Anonymous Coward, 27 Aug 2014 @ 8:10am

Re: Wet Socks
Ninja do me a favor and look at the author of the article and after you realize it's Timothy who wrote this one not Mike please go hit your self in the face with a pair of wet socks to knock the derp out of yourself.
[ link to this | view in thread ]
That One Guy (profile), 27 Aug 2014 @ 8:32am

Re: Everyone who thinks there are backups are wrong
If they were lost, they'd just say so, but they haven't, instead they're just 'difficult to get to', which suggests they could recover the emails, they just don't want to bother with trying.
[ link to this | view in thread ]
Keroberos (profile), 27 Aug 2014 @ 8:35am

Re: Re: Oh, the stories I could tell you about backups
Fire the IT department
Or, you know, maybe give them the budget and man-hours that they need to do the job right (which is the most likely culprit when you have IT issues like this).
[ link to this | view in thread ]
Vladimir Putin (profile), 27 Aug 2014 @ 8:51am

Re: Re: If too hard to restore then they are useless
In Russian Ukraine...

Backups Test YOU!
[ link to this | view in thread ]
art guerrilla (profile), 27 Aug 2014 @ 8:53am

Re: Oh, the stories I could tell you about backups
just as an aside, we have a semi-strict backup regimen where i work, but i'll be fucked if i understand what good it does us...
on a couple of occasions, i have gone to them about 'oh, that subdirectory/file is missing, can we restore from backup?', or 'i need a previous version of this file, can we restore from backup?', and the answer is 'no, we can not'... ? ? ?
apparently, the ONLY thing our 'backup' is good for, is if the whole building burns down, they can recreate the server's files...
great:
1. realistically WHAT are the odds of that ?
2. 99% of the time, it is NOT a TOTAL disaster we are recovering from, but small disasters here and there...
3. IF you can't get a previous file from your 'backup' system, it is not a backup system, but insurance...
4. the daily backups are then MOSTLY, MOST of the time, a complete waste of time & resources; we could back up once a month or once a year for the sake of the server (data files are another matter, but we aren't able to get to those anyway, so WTF...)
*shrugs* beyond my pay grade...
[ link to this | view in thread ]
Anonymous Coward, 27 Aug 2014 @ 9:03am

Re: Re: Wet Socks
Upon information and belief, Ninja is channeling a composite of the trolls, tin-hat fanciers and others on this site who seem to lack the sort of mental granularity necessary to distinguish the difference among the various authors. It's come up before.
[ link to this | view in thread ]
zip, 27 Aug 2014 @ 9:16am

Re: Re: Wet Socks
I hope I'm not the only one here to notice that Masnick and Geigner have very different writing styles.
[ link to this | view in thread ]
ltlw0lf (profile), 27 Aug 2014 @ 9:31am

Re:
Heck, I'll do it for them. Dear IRS, I volunteer. I'll do it pro bono.

As far as I am aware, unless things have changed, it is very unlikely for the government to accept your volunteer/pro-bono work. There are some places that might take your Volunteer work, but they aren't many. The government assumes a liability when dealing with volunteers, and being risk adverse, they tend not to like it.

Of course, you could go to work for a contractor, which means someone else gets paid for your volunteer work. But that would require the standard government contracting process, which would likely make it not worth your time or effort.
[ link to this | view in thread ]
Anonymous Coward, 27 Aug 2014 @ 9:33am

Re: Re: Re: Backups. We don't need no stinkin' backups.
Veeam is awesome and far easier then you would expect. Took me 30 minutes to install and setup and 15 minutes to test after the first backup cycle.
[ link to this | view in thread ]
Anonymous Coward, 27 Aug 2014 @ 9:53am

Love the disparaging remarks about Fox news
Love how you diparage Fox news. Are you implying the Communist News Network is somehow better? Anyone who doesn't watch Fox is basically getting the same story from 100 places. Fox is the only one showing the "other side". But hey, don't take your blinders off, feel free to vote for Hillary. Of course she couldn't even run the State Department or keep Obama's Benghazi lie going for more than 2 days. But if it weren't for Fox news, you would still beleive the Benghazi lie too.
[ link to this | view in thread ]
Dark Helmet (profile), 27 Aug 2014 @ 10:06am

Re: Love the disparaging remarks about Fox news
Listen, idiot. I specifically put that line in to COMBAT people dismissing this simply because it was on Fox News. I did the opposite of what you're claiming.

Don't forget breathe, now....
[ link to this | view in thread ]
Anonymous Coward, 27 Aug 2014 @ 10:07am

Re:
Said every politician ever whist the sheeple look on it fake disgust.

I am beginning to think the Americans only hate the Elected because they are shit faced jealous of them. They don't seem to get getting them out.
[ link to this | view in thread ]
Anonymous Coward, 27 Aug 2014 @ 10:07am

Re:
im good with that.
[ link to this | view in thread ]
Anonymous Coward, 27 Aug 2014 @ 10:11am

Re: Re: Purposeful
Care less about all of the failing to backup their shit fucksticks.

I work in IT, the politics and not the technology is the largest barrier.

There should only be 1 rule in the political world. If you cannot produce the records you say you keep, then you must fall upon your sword! There is simply no excuse for any of this!
[ link to this | view in thread ]
nasch (profile), 27 Aug 2014 @ 10:27am

Re: Re: Re: Wet Socks
I hope I'm not the only one here to notice that Masnick and Geigner have very different writing styles.

The difference is usually clear before the end of the first sentence.
[ link to this | view in thread ]
John Fenderson (profile), 27 Aug 2014 @ 10:36am

Re: Love the disparaging remarks about Fox news
"Love how you diparage Fox news. Are you implying the Communist News Network is somehow better?"

That's a false dichotomy. Just because FNC is abominable doesn't imply that CNN is not. Television news generally tends to be truly awful.

"Anyone who doesn't watch Fox is basically getting the same story from 100 places."

This is demonstrably untrue. I don't watch FNC, but I'm certainly not getting the same story from all of the news sources I do pay attention to.

"Fox is the only one showing the "other side".

You think there's only two sides to any given news story? If so, then I think that you need to check your own blinders before pointing out the ones on other people.
[ link to this | view in thread ]
Anonymous Coward, 27 Aug 2014 @ 10:52am

Disaster recovery backups do not equal daily, weekly, monthly, rotating backups.

Sometimes the software used is even different - imagine that.

One scenario would be to do machine backups periodically. These machine images have to be restored in toto (as opposed to file by file). If that machine is part of a complex infrastructure, the corresponding infrastructure also has to be restored in order for that machine to function.

For example, I do yearly DR tests on a small environment. That environment requires 12 virtual systems and two networks to set up. During a DR test, those new machines and networks are provisioned, then the DR information is loaded in a documented order. Finally, both system and user tests are run to complete the drill. Even with a small environment such as this, it takes 6 hours from start to finish.

DR backups recover what's required for business continuity. This may NOT be everything. If everything goes as planned, you recover from a DR snapshot, then use off-site backups to bring the DR snapshot up to date. If everything works as planned, you lose 1 day's worth of information. Yes, you can do incremental backups more frequently (mirrored via a MAN) if your business requirements dictate it.

Now, imagine this scenario. Backups have failed, so all you have are DR snapshots. If you have the DR snapshot that contains the mail in question, you're in luck. You still have to set up enough of the infrastructure to bring up the environment from the DR snapshot. If you don't have a DR snapshot that contains the mail in question (depending on full or incremental backups to completely restore the environment), then you're out of luck.

Still, it can be a major undertaking to recover single pieces of information from a DR system.
[ link to this | view in thread ]
Coyne Tibbets (profile), 27 Aug 2014 @ 11:03am

We can solve this...
Requestor: Oh. Too hard to get it off the backup. That makes sense I guess. But that's no problem. Just give us the whole backup and we'll get it off.

Gov: Wahh wah wahh wah wah!

Requestor: But the NSA grabs everything and promises it will only look at relevant stuff. We're just doing the same. We promise we'll only look at Lerner's emails. Trust us.

Gov: Wah wahh wah wahh wahh wah!

Requestor: But our promise is as good as the NSA's. You accept promises from the NSA. Look, we didn't even have our fingers crossed. You can trust us.
[ link to this | view in thread ]
KeillRandor, 27 Aug 2014 @ 11:18am

Re: Re: Mail server blues
According to what I've read (over at Ars Technica and linked from there), the problems appear to be:

The tape backup drive that the emails were probably stored on, (aside from those that were lost on the computer itself), was recycled as part of a rolling back-up system - (probably every 6 months).

If you don't like that, blame Congress, not the IRS, for not providing funding and support for a proper archival system as opposed to a short-term backup system, which they currently have - though since, of course, government funding for anything other than the military is currently being cut, YMMV.

They MIGHT be able to restore some data from the recycled back-up tape, but don't put any money on it...

This is NOT the fault of the IRS at all - which a lot of people won't want or like, so don't expect any of this to become part of the main narrative...
[ link to this | view in thread ]
Anonymous Coward, 27 Aug 2014 @ 11:32am

The justice department, where those that enforce the laws declare themselves exempt from those same laws. Go go American banana Republic
[ link to this | view in thread ]
Anonymous Coward, 27 Aug 2014 @ 11:52am

Re: Re: Typical and Far Too Common
That was my point exactly. Unless you are technically savvy enough to both understand the need and get past all the roadblocks, backups of the local machine are likely done rarely, if ever. It's a problem when you need to be accountable to the american taxpayers.
[ link to this | view in thread ]
Michael, 27 Aug 2014 @ 12:16pm

Re: Re: Re: Mail server blues
They are REQUIRED BY LAW to maintain these records longer than this plan supports, so I actually find the idea that they simply re-use media because they do not have proper funding dubious.

What really gets me is that this is the IRS - the same organization that requires me to maintain tax information for SEVEN YEARS. If they can claim "lost by stupidity" and be just fine, why can't the rest of us?
[ link to this | view in thread ]
Anon, 27 Aug 2014 @ 12:45pm

Missing the Point...
I've seen the similar nightmare with many customers, even those with relatively smaller email databases. I know this nightmare because I've contemplated it with many customers in mind.

MS-Exchange, if you use Arcserve backup or Symantec - you have two methods of backup - whole database or item by item. If you're really, really cheap, you use Windows backup to a flat file, and then backup that flat file backup of the entire database.

If your boss is really cheap and the procurement process is a nightmare, you probably also run Exchange 2003 with some limitations on maximum database size.

"Prior to SP2, each database was limited to 16 GB of total physical size. SP2 increases the licensed database size limit for Exchange Server 2003 Standard Edition from 16 GB to 75 GB; the default configured database size limit will be 18 GB"

So... now the boss won't spring for fancy backup software or bigger servers, or upgrade the Exchange software or hardware - which by the way is a project, and has to be planned and budgeted like most projects including man-hours.

So you have a backup that's permanently archived every year-end (not too cheap) but to recover that year's emails (assuming 1.5GB is a year or more of email, so January each year has not slipped off the abck end).

You will have to:
Create a domain controller first - create one of those on the network, then take it offline and put it on this private network.
Create a new Exchange server as part of this private network, join the domain.
Load the backup tape onto this private domain - oh wait, now the backup physical server with tape drive has to be transferred to the private network temporarily.

You've restores the email database, now a client on the private network, running Outlook, has to connect to the Exchange server and copy the emails down to a PST.

Repeat for each year relevant to the investigation.
What do you mean you don't have Exchange 2003 install media any more, or SP2? it was available for download when Joe set up the servers in 2004.
What do you mean you need 3 free machines to do this, plus a SCSI card for the tape drive, or you have to take our backup process offline?

This assumes to the backup software has not changed since the tape was made or the current product can read the old stuff. If they switched to VMware and VeeAm, did they think to keep the old tape hardware? (There's plies of reel tapes and punch cards out there in archiveland which are equally useful).

Whenever I dealt with clients with this sort of issue, the general rule was - your backup of Exchange is ONLY useful if the building burns down and you need to pick up where you left off. Anything else will be incredibly painful, expensive, and slow.
[ link to this | view in thread ]
John Fenderson (profile), 27 Aug 2014 @ 12:49pm

Re: Re: Re: Mail server blues
"If you don't like that, blame Congress, not the IRS, for not providing funding and support for a proper archival system as opposed to a short-term backup system"

No, this problem is squarely on the shoulders of the IRS, not Congress. Congress has given them PLENTY of money to fix these issues -- on the order of billions of dollars.

The IRS blew through all that money in a couple of projects that ended in failure.
[ link to this | view in thread ]
John Fenderson (profile), 27 Aug 2014 @ 12:52pm

Re: Typical and Far Too Common
That's pretty insane.

However, these are emails we're talking about. The backups for them would not be off o your machine, but the exchange server. Surely, they aren't backing up that server using USB drives.
[ link to this | view in thread ]
John Fenderson (profile), 27 Aug 2014 @ 12:54pm

Re: Everyone who thinks there are backups are wrong
I think we aren't misunderstanding. We're just giving the IRS a well-deserved hazing because their backup system is clearly inadequate. No, that's not harsh enough. It's incompetent.
[ link to this | view in thread ]
Anonymous Coward, 27 Aug 2014 @ 2:14pm

Sounds like aiding and abetting to me. Don't lie down with these dogs and you won't get up with fleas.
[ link to this | view in thread ]
billy, 27 Aug 2014 @ 2:42pm

I totally don't believe in conspiracies, but here's proof of a conspiracy...
How many articles is Timmy going to start by disavowing his generalized misconception of what "conspiracy theories" are before the functioning literates at Techdirt realize how absurd it makes them come across? Can I filter out articles by him? Please?
[ link to this | view in thread ]
nasch (profile), 27 Aug 2014 @ 2:51pm

Re: I totally don't believe in conspiracies, but here's proof of a conspiracy...
Can I filter out articles by him? Please?

Sure. There's this text that start with "by". If you see "Timothy Geigner" next to that, then don't read the story.
[ link to this | view in thread ]
billy, 27 Aug 2014 @ 3:03pm

Re: Re: I totally don't believe in conspiracies, but here's proof of a conspiracy...
I was thinking more of a kiddies table area, where people can spend all day talking about how totally wrong Alex Jones was about black helicopters. Scrolling past Tim's articles do Techdirt a huge disservice imo.
[ link to this | view in thread ]
Anonymous Coward, 27 Aug 2014 @ 3:13pm

I know at least one fruit based tech company that keeps its backups ON THE SAME HARDWARE as the live data.

And that is secured by an unbelievably feeble XML front-door that anyone could access..POOF! all your iSongs are gone forever.....
[ link to this | view in thread ]
Miko, 27 Aug 2014 @ 3:18pm

The backup systems are only designed for use in situations where they didn't lose the original data on purpose.
[ link to this | view in thread ]
Anonymous Coward, 27 Aug 2014 @ 3:35pm

Re: Missing the Point...
This is why you don't use proprietary technology: I have Unix backups from thirty years ago that I can read just as easily today as I could then. (They've been transported from 9-track tape to 8mm tape to USB drive, but the bytes are as-written all those years ago.)

I can also USE everything in them because everything in them is in open formats.

Once you marry yourself to proprietary formats, protocol, software, hardware, you're guaranteeing yourself future headaches. So don't do that. Ever.
[ link to this | view in thread ]
btr1701 (profile), 27 Aug 2014 @ 4:06pm

Re:
It's beyond me that whichever judge is overseeing this mess hasn't pulled the government's lawyers into court and told them in absolutely plain and simple terms that if they don't manage to find and deliver the relevant documents to the court in a reasonable amount of time, someone important is going to find his ass in a jail cell until they are so delivered.
[ link to this | view in thread ]
Sunhawk (profile), 27 Aug 2014 @ 5:01pm

I'd been wondering if this wasn't the case at times.

"It's too hard" isn't an excuse; this is (however silly the directions it goes sometimes) a Congressional investigation - *make* it easier.

Or do it anyway, chalk it up as a learning experience, and use a better backup system.
[ link to this | view in thread ]
Sunhawk (profile), 27 Aug 2014 @ 5:04pm

Re: Re: Re: Oh, the stories I could tell you about backups
... point. There's almost certainly been skimping on safeties and backups to stretch inadequate budget(it's rare that an IT department gets an adequate budget, it seems to me - particularly for things that don't demonstrate a benefit unless something bad happens).
[ link to this | view in thread ]
Eldakka (profile), 27 Aug 2014 @ 7:27pm

Re: Oh, the stories I could tell you about backups
I can agree with and expand upon this.

I work for a (non-US) government department.

Part of our IT, unfortunately including backup responsibility, was (not any longer) outsourced to a 3rd party outsourcer, one of the big ones.

I once had occasion to view the contract, specifically that dealt with backup requirements, with said outsourcer. Nowhere, not once, did it include the word "restore" in it. Or any phrases or concepts that could correspond to the term/concept, restore.

Therefore, in theory, they could backup to /dev/null (wow, backups run so fast now, use so little CPU or network bandwidth...), as nowhere was there ever a requirement to be able to restore the backups. Sometimes they could restore data, often they couldn't. When they couldn't, they'd just shrug their shoulders and say it can't be restored, not their problem.
[ link to this | view in thread ]
Anonymous Coward, 27 Aug 2014 @ 10:19pm

Didn't the DOJ also say it would be too hard to backup metadata evidence in the Jewel vs NSA unconstitutional spying case? All this evidence destruction isl rather convenient for the Executive Branch.

I'm pretty sure destruction of evidence is a crime. Possibly a felony charge. Our government is staffed by felons.
[ link to this | view in thread ]
toyotabedzrock (profile), 28 Aug 2014 @ 1:18am

There was no server crash. They just use backup tapes that get reused far too often and really small exchange email boxes which means they had to store email on their local systems.
[ link to this | view in thread ]
toyotabedzrock (profile), 28 Aug 2014 @ 1:18am

There was no server crash. They just use backup tapes that get reused far too often and really small exchange email boxes which means they had to store email on their local systems.
[ link to this | view in thread ]
Niall (profile), 28 Aug 2014 @ 2:20am

Re: Re: Everyone who thinks there are backups are wrong
So how come my .pst files work fine onto a network drive?

Oh noes, my .pst files just vanished in a puff of logic!
[ link to this | view in thread ]
Dark Helmet (profile), 28 Aug 2014 @ 4:06am

Re: Re: Re: I totally don't believe in conspiracies, but here's proof of a conspiracy...
Now I'm confused. Are you an Alex Jones loving crazy person, or someone who thinks ANY claim of conspiracy is crazy?

Either way, you're wrong, so feel free to go away....
[ link to this | view in thread ]
Niall (profile), 28 Aug 2014 @ 4:54am

Re: Re: Love the disparaging remarks about Fox news
And if scientists or Democrats said the moon is made of rock and gravity makes you fall, Fox would find 'experts' that disputed these facts... as its 'opposite opinion'.
[ link to this | view in thread ]
billy, 28 Aug 2014 @ 3:54pm

Re: Re: Re: Re: I totally don't believe in conspiracies, but here's proof of a conspiracy...
My problem is, as stated before, Tim's generalized misconception of what "conspiracy theories" are and his constant denial of being a "conspiracy theorist." Which has nothing to do with the 2 viewpoints you ascribed to me, and your conclusion that I'm "wrong." Do you often tell people who confuse you to "go away?" How does someone such as yourself learn anything?
[ link to this | view in thread ]