Justice Dept. Official: We Could Get Lois Lerner's Emails From Backups, But It's Too Hard So Naaaaaah
from the facepalm dept
I try not to go for conspiracy theories generally, but this ongoing IRS nonsense involving conveniently disappearing emails potentially pertaining to the scandal involving targeting certain groups is making my skeptics beacon go off. The official story essentially involves a computer (server?) crash that obliterated the email data of several email accounts that would otherwise be of great interest to those trying to figure out who in the Obama administration knew what about how the IRS was operating. That crash somehow also involves the destruction of any local backups these IRS folks are required to keep as part of their job.
But what I imagine pretty much everyone that has even a modicum of interest in office technology is asking is what about the kind of backups that are typically run for disaster recovery and offsite purposes. You know, like tape drives (blech), D2D backups, or remote storage. It's difficult to believe that all copies of the data for those email accounts had somehow been disappeared by accident. Or, really, disappeared entirely at all.
As it turns out, there is backup data for those email accounts, but if you're thinking that we're finally going to get to the bottom of this nonsense, you're a silly naive person that hasn't witnessed our glorious government at work.
“A Department of Justice attorney told a Judicial Watch attorney on Friday that it turns out the federal government backs up all computer records in case something terrible happens in Washington and there’s a catastrophe, so the government can continue operating,” Judicial Watch president Tom Fitton told Fox News’s Shannon Bream. “But it would be too hard to go get Lois Lerner’s e-mails from that backup system."Now, I know, Fox News and blah blah blah, but this is a claim worth paying attention to mostly because it only makes sense to begin with. Well, the part about there being backups, I mean. The suggestion that such backups are so difficult to parse and recover that the administration isn't going to get them for review makes zero sense because that's what damned backups are for.
Think about the logic here for a moment. The reason the federal government has backups of data in place is because there's a great deal of horrible crap that could happen to their facilities. Natural disasters, terrorist attacks, hallmark-level incompetence, Area 51 aliens going Rambo on the computers. These are the things they have to prepare for and they need the ability to restore data from a remote location should one of their facilities fall to our alien overlords. To say that restoring that data is too hard to do when requested is to say that they're ill-prepared for a disaster. The suggestion that the government can't retrieve emails for court or congressional review is to say that they wouldn't even meet FINRA compliance, something that itsy-bitsy little trading firms are required to meet.
Fitton said his group plans to ask a federal judge to order the IRS to hand over the e-mails, which conservative opponents of Lerner want to see in order to determine if there is a link between President Obama’s team and the IRS’s targeting of tea-party groups.And if the backup system isn't working, then fire everyone everywhere because this kind of thing is no joke. I'd say it'd be a bigger scandal to have any portion of the federal government not have basic backup and DR systems in place than to find out that any portion of the Executive administration was encouraging the IRS to target certain special interest groups.
“If this backup system is working, Lois Lerner’s e-mails are there,” Fitton said.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: backups, doj, email, irs, justice department, lois lerner
Reader Comments
Subscribe: RSS
View by: Time | Thread
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re: Wet Socks
[ link to this | view in chronology ]
Re: Re: Wet Socks
[ link to this | view in chronology ]
Re: Re: Wet Socks
[ link to this | view in chronology ]
Re: Re: Re: Wet Socks
The difference is usually clear before the end of the first sentence.
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Oh, the stories I could tell you about backups
Major university: backups readable by everyone on the network, i.e., Alice could retrieve Bob's files.
Another Fortune 100 corporation: backups weren't readable. Guess how they found out.
And on and on and on. Most organizations are quite happy to buy overpriced overblown overhyped horribly insecure backup "solutions" from greedy vendors, deploy them, and mark the problem as "solved". Few -- precious few -- are actually willing to engineer solid solutions based on well-tested well-documented code (e.g., dump, tar, rsync and other basic tools) and then actually perform ongoing monitoring (as in "read the logs") and testing (as in "see if restoration is possible").
The overall state of backups across the world is best classified as "miserable".
[ link to this | view in chronology ]
Re: Oh, the stories I could tell you about backups
As an administrator of a network professionally, and as a volunteer tech support person on the side (who in this business doesn't hold that title,) I wholly agree.
However, that being said, it is amazing the power of TAR/DAR (or even 7z, for those Windoze aficionados,) and external USB hard drives. They've saved me from some really serious stuff, just as long as you write to multiple hard drives, and periodically test your backups on a non-production system.
Still miserable ghetto-tech, but it would have helped here, if it wasn't in the best interest of those involved (allegedly) that it not work.
[ link to this | view in chronology ]
Re: Re: Oh, the stories I could tell you about backups
And yes, we test them. ;)
That's why in all the years I've been here our success rate at restoring files is 100.00%. And why, if the building burned to the ground tomorrow, I'm confident that we could restore our operation to the state that it was in as of the beginning of the month. (Which, for what we do, is acceptable.)
But I'm painfully well aware that we're one of the few exceptions in a sea of mediocrity and incompetence.
[ link to this | view in chronology ]
Re: Oh, the stories I could tell you about backups
[ link to this | view in chronology ]
Re: Re: Oh, the stories I could tell you about backups
[ link to this | view in chronology ]
Re: Re: Re: Oh, the stories I could tell you about backups
[ link to this | view in chronology ]
Re: Oh, the stories I could tell you about backups
on a couple of occasions, i have gone to them about 'oh, that subdirectory/file is missing, can we restore from backup?', or 'i need a previous version of this file, can we restore from backup?', and the answer is 'no, we can not'... ? ? ?
apparently, the ONLY thing our 'backup' is good for, is if the whole building burns down, they can recreate the server's files...
great:
1. realistically WHAT are the odds of that ?
2. 99% of the time, it is NOT a TOTAL disaster we are recovering from, but small disasters here and there...
3. IF you can't get a previous file from your 'backup' system, it is not a backup system, but insurance...
4. the daily backups are then MOSTLY, MOST of the time, a complete waste of time & resources; we could back up once a month or once a year for the sake of the server (data files are another matter, but we aren't able to get to those anyway, so WTF...)
*shrugs* beyond my pay grade...
[ link to this | view in chronology ]
Re: Oh, the stories I could tell you about backups
I work for a (non-US) government department.
Part of our IT, unfortunately including backup responsibility, was (not any longer) outsourced to a 3rd party outsourcer, one of the big ones.
I once had occasion to view the contract, specifically that dealt with backup requirements, with said outsourcer. Nowhere, not once, did it include the word "restore" in it. Or any phrases or concepts that could correspond to the term/concept, restore.
Therefore, in theory, they could backup to /dev/null (wow, backups run so fast now, use so little CPU or network bandwidth...), as nowhere was there ever a requirement to be able to restore the backups. Sometimes they could restore data, often they couldn't. When they couldn't, they'd just shrug their shoulders and say it can't be restored, not their problem.
[ link to this | view in chronology ]
If too hard to restore then they are useless
So you know they work as expected
Offline is best
So you can rest
When lightening strikes unexpected
[ link to this | view in chronology ]
Re: If too hard to restore then they are useless
[ link to this | view in chronology ]
Re: Re: If too hard to restore then they are useless
Backups Test YOU!
[ link to this | view in chronology ]
http://stockman.house.gov/media-center/press-releases/stockman-requests-subpoena-of-nsa-s-w hite-house-irs-phone-logs
TL,DR; Stockman and his ilk only oppose NSA surveillance when it benefits them personally to do so. When it incriminates someone they don't like (Fruit of the Poison Tree, Anyone?) they're all for it!
[ link to this | view in chronology ]
Re:
I am beginning to think the Americans only hate the Elected because they are shit faced jealous of them. They don't seem to get getting them out.
[ link to this | view in chronology ]
If it is to hard to get the data back, the ENTIRE thing is a waste and we should file suit. Anyone involved in continuing the contract should be assumed to be at best incompetent at worst bought off.
Is it time that they stop digging into our data & lives and maybe consider they should look in a mirror and fix the problems there.
First talking head that tries to lay this at the presidents feet gets shot.
This is decades of corruption and stupidity coming to light, and they need to find the will to fix it without finding a way to line their own pockets on the deal.
[ link to this | view in chronology ]
Re:
There is a big reason that they do not call each other out for actual corruption... they are all corrupt! That is off limits!
[ link to this | view in chronology ]
[ link to this | view in chronology ]
[ link to this | view in chronology ]
If it ever becomes more useful to have those, they will magically be found never to have been lost at all.
[ link to this | view in chronology ]
Of what use are these emails when it is a well known fact that the IRS investigated applicants from all political persuasions with somewhat equal vigor? In addition, it has been reported that the only organization denied their coveted status was of a liberal slant. Add to this, that the status is not even needed in order to initially claim the exemption.
It is a side show, like Benghazi, trumped up pile on bullshit mountain.
[ link to this | view in chronology ]
Re:
This particular story is not about the merits of the investigation, but the IRS response to it. The target of a Congressional investigation cannot be the party that decides whether it's legitimate.
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Purposeful
You can be DARN SURE that if those were emails that had to do with auditing some poor sap and ruining his life that they would be found INSTANTLY.
The IRS is the largest, most well-funded and most brutal terrorist organization in the world. Never forget that.
[ link to this | view in chronology ]
Re: Purposeful
"there is no way that data wasn't backed up and no way that the backups are saved SOMEWHERE - normally - ESPECIALLY for the IRS."
I think that's a step too far. Given that tons of companies -- including major corporations and tech companies -- totally fail in terms of backing up their stuff, and we're talking about the IRS here (who has blown through literally billions of dollars in failed attempts to modernize their computer systems), the idea that their backup systems don't work adequately is not beyond the realm of possibility.
[ link to this | view in chronology ]
Re: Re: Purposeful
I work in IT, the politics and not the technology is the largest barrier.
There should only be 1 rule in the political world. If you cannot produce the records you say you keep, then you must fall upon your sword! There is simply no excuse for any of this!
[ link to this | view in chronology ]
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Mail server blues
I’ve read in the news that the IRS uses a Microsoft Exchange system for email. Each user gets X amount of data storage on the server that is backed up. The user goes over that limit of X. There are two solutions to this in general. The first is increase the size of the storage on the server and the back system. The second is to set up a personal storage file (.pst) on the local computer in that person’s office. The user (with the help of a tech, usually) chooses what emails are moved from the server to the local file. The office systems are not usually backed up. If the second option was deployed by the user or tech staff and later on that office system’s hard drive crashed… well, then the data could be gone for good.
Now, they say that there is a backup? If so, I have no idea how they’ve set up this system.
[ link to this | view in chronology ]
Re: Mail server blues
This should be a lesson to anyone who isn't tech-savvy in today's world to learn the basics of how these things work so they can filter out the bullshit these people are feeding us.
[ link to this | view in chronology ]
Re: Mail server blues
That's not typically how email backups work, in my experience. Usually, any competent organization is going to be doing some flavor of email archiving, in which a journaled Exchange account funnels copies of all email traffic to the archiving system, which is separate from the Exchange box/cluster. The user CANNOT control what is backed up by design. No system which allowed that would meet the many compliance standards the government and SGOs have set up for various industries.
As a device on the network, that message archiver is also backed up, along with the Exchange box/cluster. Backups of any legitimacy involve a local and at least one offsite location. This is harder with tape drives, which is why tape drives should go away and die forever, but there should always be at least 3-4 geographic points from where emails can reside. For all of those to disappear isn't believable.
[ link to this | view in chronology ]
Re: Re: Mail server blues
I think I see your error. ;-)
[ link to this | view in chronology ]
Re: Re: Mail server blues
The tape backup drive that the emails were probably stored on, (aside from those that were lost on the computer itself), was recycled as part of a rolling back-up system - (probably every 6 months).
If you don't like that, blame Congress, not the IRS, for not providing funding and support for a proper archival system as opposed to a short-term backup system, which they currently have - though since, of course, government funding for anything other than the military is currently being cut, YMMV.
They MIGHT be able to restore some data from the recycled back-up tape, but don't put any money on it...
This is NOT the fault of the IRS at all - which a lot of people won't want or like, so don't expect any of this to become part of the main narrative...
[ link to this | view in chronology ]
Re: Re: Re: Mail server blues
What really gets me is that this is the IRS - the same organization that requires me to maintain tax information for SEVEN YEARS. If they can claim "lost by stupidity" and be just fine, why can't the rest of us?
[ link to this | view in chronology ]
Re: Re: Re: Mail server blues
No, this problem is squarely on the shoulders of the IRS, not Congress. Congress has given them PLENTY of money to fix these issues -- on the order of billions of dollars.
The IRS blew through all that money in a couple of projects that ended in failure.
[ link to this | view in chronology ]
Because the janitors would be the only ones working..
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Typical and Far Too Common
In order to make a backup, I have to plug an external drive into my machine and manually run my backup software, rather than having an automated backup to the network. I start a obscure piece of software that I have been directed to use to make backups. I then start the process, which basically locks me out of actually using my machine until the backup is complete. There is a strong encryption requirement on any data put onto an external drive, which means that making a backup is nearly an all-day affair to allow my (somewhat underpowered) machine to crank through all the encryption. I can't leave the software to run unattended, though, because it is not quite smart enough to handle that. I can finally use my machine again once the whole process is completed.
I usually line up some work to do in the lab or on my other computers when I intend to run a backup on my primary machine. Otherwise, the day tends to drag.
I had to jump through hoops just to be able to do this. You see, I was not issued the external drive. Most of the machines are set up to *not* allow backups to external drives. If I did not have an external drive, I would have to use optical media. We also only recently (within the last two years) received DVD burners as standard in our machines. I suppose they think we are just supposed to back up to an assload of CDs, like anyone will do that with any regularity.
The trouble is, the backup and restore process should be easy. That's how you get people to actually make the backups they're supposed to! That's how you engineer things so that the records aren't lost when a hard drive dies. However, making the process easy would make sense and be efficient, so there is little chance of it happening.
On another note, I am not convinced that the emergency backup alluded to will be of any use. If the IRS system is set up similar to mine, then they probably only have 1995 levels of email storage (25MB or so) on the server that would be backed up at any one time. Either that or the files would mysteriously disappear before they were restored.
[ link to this | view in chronology ]
Re: Typical and Far Too Common
As for going into older tapes to get the data. Yes, that is entirely possible. However, if those tapes were being recycled and reused (common practice in a LOT of businesses) then the data could have been destroyed at any point.
[ link to this | view in chronology ]
Re: Re: Typical and Far Too Common
[ link to this | view in chronology ]
Re: Typical and Far Too Common
However, these are emails we're talking about. The backups for them would not be off o your machine, but the exchange server. Surely, they aren't backing up that server using USB drives.
[ link to this | view in chronology ]
Everyone who thinks there are backups are wrong
Commenter Chad Holbrook hit the nail on the head. It has already been well established that Lerner saved her email to a PST file on her local computer because their Exchange mailbox size limit is so low. The hard drive on her workstation crashed, was disposed of, and was replaced. Her workstation was NOT being backed up.
Yes, those emails were on the Exchange server for a period of time, but backup tapes get recycled and overwritten.
Those emails are irretrievably lost.
[ link to this | view in chronology ]
Re: Everyone who thinks there are backups are wrong
[ link to this | view in chronology ]
Re: Re: Everyone who thinks there are backups are wrong
Oh noes, my .pst files just vanished in a puff of logic!
[ link to this | view in chronology ]
Re: Everyone who thinks there are backups are wrong
Except now the IRS says they're backed up, but too hard to get to.
[ link to this | view in chronology ]
Re: Everyone who thinks there are backups are wrong
[ link to this | view in chronology ]
Re: Everyone who thinks there are backups are wrong
[ link to this | view in chronology ]
Backups. We don't need no stinkin' backups.
The office recovered from the quake using data from the disk packs in his cab.
As to Tim's post: I'm a Democrat who has supported Obama from early on (though break with him on several issues of TechDirt focus). Despite my general support, I agree with everything Tim wrote--something does not smell right here. They do need to recover those emails no matter what is on them.
[ link to this | view in chronology ]
Re: Backups. We don't need no stinkin' backups.
[ link to this | view in chronology ]
Re: Re: Backups. We don't need no stinkin' backups.
This stuff is waaaaaay more simple than people are led to believe. The idea of data residing primarily on a local client instead of a server and having the responsibility for backups foisted upon common users is laughable to the extreme....
[ link to this | view in chronology ]
Re: Re: Re: Backups. We don't need no stinkin' backups.
[ link to this | view in chronology ]
Re: Re: Re: Backups. We don't need no stinkin' backups.
[ link to this | view in chronology ]
Backups & DR
[ link to this | view in chronology ]
Heck, I'll do it for them. Dear IRS, I volunteer. I'll do it pro bono.
[ link to this | view in chronology ]
Re:
As far as I am aware, unless things have changed, it is very unlikely for the government to accept your volunteer/pro-bono work. There are some places that might take your Volunteer work, but they aren't many. The government assumes a liability when dealing with volunteers, and being risk adverse, they tend not to like it.
Of course, you could go to work for a contractor, which means someone else gets paid for your volunteer work. But that would require the standard government contracting process, which would likely make it not worth your time or effort.
[ link to this | view in chronology ]
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Love the disparaging remarks about Fox news
[ link to this | view in chronology ]
Re: Love the disparaging remarks about Fox news
Don't forget breathe, now....
[ link to this | view in chronology ]
Re: Love the disparaging remarks about Fox news
That's a false dichotomy. Just because FNC is abominable doesn't imply that CNN is not. Television news generally tends to be truly awful.
"Anyone who doesn't watch Fox is basically getting the same story from 100 places."
This is demonstrably untrue. I don't watch FNC, but I'm certainly not getting the same story from all of the news sources I do pay attention to.
"Fox is the only one showing the "other side".
You think there's only two sides to any given news story? If so, then I think that you need to check your own blinders before pointing out the ones on other people.
[ link to this | view in chronology ]
Re: Re: Love the disparaging remarks about Fox news
[ link to this | view in chronology ]
Sometimes the software used is even different - imagine that.
One scenario would be to do machine backups periodically. These machine images have to be restored in toto (as opposed to file by file). If that machine is part of a complex infrastructure, the corresponding infrastructure also has to be restored in order for that machine to function.
For example, I do yearly DR tests on a small environment. That environment requires 12 virtual systems and two networks to set up. During a DR test, those new machines and networks are provisioned, then the DR information is loaded in a documented order. Finally, both system and user tests are run to complete the drill. Even with a small environment such as this, it takes 6 hours from start to finish.
DR backups recover what's required for business continuity. This may NOT be everything. If everything goes as planned, you recover from a DR snapshot, then use off-site backups to bring the DR snapshot up to date. If everything works as planned, you lose 1 day's worth of information. Yes, you can do incremental backups more frequently (mirrored via a MAN) if your business requirements dictate it.
Now, imagine this scenario. Backups have failed, so all you have are DR snapshots. If you have the DR snapshot that contains the mail in question, you're in luck. You still have to set up enough of the infrastructure to bring up the environment from the DR snapshot. If you don't have a DR snapshot that contains the mail in question (depending on full or incremental backups to completely restore the environment), then you're out of luck.
Still, it can be a major undertaking to recover single pieces of information from a DR system.
[ link to this | view in chronology ]
We can solve this...
Gov: Wahh wah wahh wah wah!
Requestor: But the NSA grabs everything and promises it will only look at relevant stuff. We're just doing the same. We promise we'll only look at Lerner's emails. Trust us.
Gov: Wah wahh wah wahh wahh wah!
Requestor: But our promise is as good as the NSA's. You accept promises from the NSA. Look, we didn't even have our fingers crossed. You can trust us.
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Missing the Point...
MS-Exchange, if you use Arcserve backup or Symantec - you have two methods of backup - whole database or item by item. If you're really, really cheap, you use Windows backup to a flat file, and then backup that flat file backup of the entire database.
If your boss is really cheap and the procurement process is a nightmare, you probably also run Exchange 2003 with some limitations on maximum database size.
"Prior to SP2, each database was limited to 16 GB of total physical size. SP2 increases the licensed database size limit for Exchange Server 2003 Standard Edition from 16 GB to 75 GB; the default configured database size limit will be 18 GB"
So... now the boss won't spring for fancy backup software or bigger servers, or upgrade the Exchange software or hardware - which by the way is a project, and has to be planned and budgeted like most projects including man-hours.
So you have a backup that's permanently archived every year-end (not too cheap) but to recover that year's emails (assuming 1.5GB is a year or more of email, so January each year has not slipped off the abck end).
You will have to:
Create a domain controller first - create one of those on the network, then take it offline and put it on this private network.
Create a new Exchange server as part of this private network, join the domain.
Load the backup tape onto this private domain - oh wait, now the backup physical server with tape drive has to be transferred to the private network temporarily.
You've restores the email database, now a client on the private network, running Outlook, has to connect to the Exchange server and copy the emails down to a PST.
Repeat for each year relevant to the investigation.
What do you mean you don't have Exchange 2003 install media any more, or SP2? it was available for download when Joe set up the servers in 2004.
What do you mean you need 3 free machines to do this, plus a SCSI card for the tape drive, or you have to take our backup process offline?
This assumes to the backup software has not changed since the tape was made or the current product can read the old stuff. If they switched to VMware and VeeAm, did they think to keep the old tape hardware? (There's plies of reel tapes and punch cards out there in archiveland which are equally useful).
Whenever I dealt with clients with this sort of issue, the general rule was - your backup of Exchange is ONLY useful if the building burns down and you need to pick up where you left off. Anything else will be incredibly painful, expensive, and slow.
[ link to this | view in chronology ]
Re: Missing the Point...
I can also USE everything in them because everything in them is in open formats.
Once you marry yourself to proprietary formats, protocol, software, hardware, you're guaranteeing yourself future headaches. So don't do that. Ever.
[ link to this | view in chronology ]
[ link to this | view in chronology ]
I totally don't believe in conspiracies, but here's proof of a conspiracy...
[ link to this | view in chronology ]
Re: I totally don't believe in conspiracies, but here's proof of a conspiracy...
Sure. There's this text that start with "by". If you see "Timothy Geigner" next to that, then don't read the story.
[ link to this | view in chronology ]
Re: Re: I totally don't believe in conspiracies, but here's proof of a conspiracy...
[ link to this | view in chronology ]
Re: Re: Re: I totally don't believe in conspiracies, but here's proof of a conspiracy...
Either way, you're wrong, so feel free to go away....
[ link to this | view in chronology ]
Re: Re: Re: Re: I totally don't believe in conspiracies, but here's proof of a conspiracy...
[ link to this | view in chronology ]
And that is secured by an unbelievably feeble XML front-door that anyone could access..POOF! all your iSongs are gone forever.....
[ link to this | view in chronology ]
[ link to this | view in chronology ]
"It's too hard" isn't an excuse; this is (however silly the directions it goes sometimes) a Congressional investigation - *make* it easier.
Or do it anyway, chalk it up as a learning experience, and use a better backup system.
[ link to this | view in chronology ]
I'm pretty sure destruction of evidence is a crime. Possibly a felony charge. Our government is staffed by felons.
[ link to this | view in chronology ]
[ link to this | view in chronology ]
[ link to this | view in chronology ]