Your Tax Dollars At Work: How Commerce Dept. Spent $2.7 Million Cleaning Out Two Malware-Infected Computers
from the burning-a-hole-in-taxpayers'-pockets dept
The cyber-Pearl Harbor is upon us and the only way to defeat it is to sink our own ships at the first sign of invasion. This is the sort of thing that happens when the legislators and advisors with the loudest voices value paranoia over rational strategy. The Department of Commerce, aided by a tragicomic string of errors, managed to almost stamp out its malware problem.
The Commerce Department's Economic Development Administration spent almost half of its IT budget last year to remediate a cyber attack that barely happened.Also included in the mass destruction were cameras and TVs. It wasn't just cyber-paranoia that led to this hardware cull. There was plenty of miscommunication too, along with the usual doses of bureaucratic clumsiness. The Inspector General's report breaks down the chain of missteps, which all began with a response team member grabbing the wrong network info.
EDA's drastic steps to limit the damage by shutting down much of the access to the main Herbert Hoover Building network ended up costing the agency more than $2.7 million to clean up and reconfigure its network and computers. The IG said the bureau destroyed more than $170,000 in IT equipment, including desktop computers, printers, keyboards and mice.
In an effort to identify infected components, DOC CIRT’s (Dept. of Commerce Computer Incident Response Team) incident handler requested network logging information. However, the incident handler unknowingly requested the wrong network logging information... Instead of providing EDA a list of potentially infected components, the incident handler mistakenly provided EDA a list of 146 components within its network boundary. Accordingly, EDA believed it faced a substantial malware infection.Yes. Much like "Reply" and "Reply All" will both get the job done, only one is the correct choice when firing off a devastating critique of your soon-to-be-former coworkers. The same goes for network logs. One shows you the correct info. The other "indicates" that more than half the EDA's computers are suffering from a malware infection.
DOC CIRT did try to get this fixed, pointing out the error to the handling team and re-running the analysis using the correct network log. Turns out, the original estimate was slightly off.
The HCHB network staff member then performed the appropriate analysis identifying only two components exhibiting the malicious behavior in US-CERT’s alert.This new data in hand, a notification was sent out ostensibly to clear things up, but this too was mishandled so badly someone unfamiliar with bureaucratic ineptitude might be inclined to suspect sabotage.
DOC CIRT’s second incident notification did not clearly explain that the first incident notification was inaccurate. As a result, EDA continued to believe a widespread malware infection was affecting its systems.For five weeks, things went from bad to worse to comically tragic to tragically comic to full-scale computercide. Looking at its list (2 components), DOC CIRT asked the EDA to attempt containment by reimaging the infected items. Looking at its list (146 components), the EDA responded that reimaging half its devices would be "unfeasible." Taking a look at the EDA's list (from the first, mistaken network log analysis), DOC CIRT assumed the EDA had received additional analysis indicating the malware had spread, and changed its recommendations accordingly.
Specifically, the second incident notification began by stating the information previously provided about the incident was correct. EDA interpreted the statement as confirmation of the first incident notification, when DOC CIRT’s incident handler simply meant to confirm EDA was the agency identified in US-CERT’s alert. Nowhere in the notification or attachment does the DOC CIRT incident handler identify that there was a mistake or change to the previously provided information.
Although the incident notification’s attachment correctly identified only 2 components exhibiting suspicious behavior—not the 146 components that DOC CIRT initially identified—the name of the second incident notification’s attachment exactly matched the first incident notification’s attachment, obscuring the clarification.
Finally, both departments were on the same (but entirely wrong) page and scaled up the response accordingly. A copy went to the DHS, stating that "over 50%" of the EDA's devices were infected. The DHS then accepted this without seeking independent confirmation. The NSA cranked out its own concerned report, quoting heavily from the DHS report (which was still in draft form), both of which were based on DOC CIRT's first erroneous report. This went undetected for over a year, until the OIG informed the involved agencies of its findings in December 2012.
The end result? The EDA and DOC CIRT worked together, attempting to head off a "severe" malware threat before it spread to other connected government computers. Despite gathering more information from outside consultants that indicated the malware was neither "persistent" nor a threat to migrate, the two agencies began destroying devices in May of 2012, finally stopping three months later when the "break stuff" budget had been exhausted.
Fortunately for the agencies, taxpayers and the surviving equipment (valued at over $3 million), the OIG's findings were brought to the agencies' attention before the fiscal year began and a new "break stuff" budget approved. All in all, the EDA spent over $2.7 million fighting a malware "infection" confined to two computers.
There's nothing in this report that makes the EDA look good. A chart on page 8 shows the EDA has persistently ignored the OIG's recommendations on agency computer security, with some assessments going back as far as 2006. It's no surprise it managed to (along with the Dept. of Commerce's response team) transform a 2-computer infection into a nearly $3 million catastrophe.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: commerce department, malware, miscommunication
Reader Comments
Subscribe: RSS
View by: Time | Thread
[ link to this | view in chronology ]
Re:
How the holy fuckin' cow peripherials like mouse and keyboard got destroyed in the process, if not for the sake of "destroying" equipment.
Probably someone needed stuff and panicked destruction of misreported equipment were the most easiest way.
1. Look up who the fuck handled the two reports.
2. Check who did the destroying of the equipment
3. Check the new equipment vendors
I bet there would be some connection between 1-2 or 1-3.
[ link to this | view in chronology ]
Re: Re:
I actually hope that the equipment was destroyed as part of some backroom deal, and not out of stupidity. If the latter is true, than those responsible should be fired, and banned from ever touching a computer again!
Seriously, in a case of a malware infection, the most you ever do is format and reinstall, unless the computer in question is junk, and you wanted to get rid of it anyway.
[ link to this | view in chronology ]
Re: Re: Re:
The official response on hijackthis logs is that if you have had certain backdoors, the only way to be certain of avoiding a reinfection is destroying the equipment.
[ link to this | view in chronology ]
Re: Re: Re: Re:
"Despite gathering more information from outside consultants that indicated the malware was neither "persistent" nor a threat to migrate"
Yes, there are some extremely nasty malware variants that are persistant to the (sorta-)hardware level that lodge themselves in the (not-)ROM of a motherboard or other device. But those types were not involved in this incident - it was boring standard malware and email spam. A reimaging of the infected machines would've solved the problem.
[ link to this | view in chronology ]
Re: Re: Re: Re:
By the way, the only way to wipe a hard drive is using old Uinux/Linux tools like dd: dd if = /dev/zero of = /dev/sda, or something similar. The standard windows format command wont do, and can leave malware in hidden portions of the disk (other partitions, boot sector, partition table, etc).
[ link to this | view in chronology ]
Re: Re: Re: Re: Re:
There are Windows tools to do this as well (there's even a port of dd for those who like to kick it old-school). A half hour in a bulk eraser does a pretty good job, too.
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Re:
Sorry for the spelling derp, its Unix not Uinux.
[ link to this | view in chronology ]
Re: Re: Re: Re:
[ link to this | view in chronology ]
hiding something?
[ link to this | view in chronology ]
Re: hiding something?
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Please tell me ...
The only thing that could make the story better is if the original 2 infected computers still remain operational and unmolested by re-imaging.
[ link to this | view in chronology ]
The effects of ignorance
[ link to this | view in chronology ]
[ link to this | view in chronology ]
They blew up $170.000 in equipment. Where's the rest of the money?
Something smells fishy here. I'm guessing the money went to some contractor that was also conveniently a friend or a family member of someone at the top.
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re: Re:
:)
[ link to this | view in chronology ]
Re: Re:
[ link to this | view in chronology ]
Re: Re: Re:
Surely at some point they'd have said "we haven't found any malware on these computers"
[ link to this | view in chronology ]
Re: Re: Re: Re:
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re: Re:
[ link to this | view in chronology ]
Re: Re:
https://www.youtube.com/watch?v=HtTUsOKjWyQ
[ link to this | view in chronology ]
I blame economists: "Economic Development Administration"
http://www.techdirt.com/articles/20130605/06245023322/china-once-again-using-censorship-el sewhere-to-justify-oppressive-great-firewall-china.shtml#c24
(I see that my point was so effective that Mike hisself made a very rare response to my post. Heh, heh.)
[ link to this | view in chronology ]
Re: I blame economists: "Economic Development Administration"
"that Google being taxed by Germany is used as justification by the Chinese government to do censorship over their people"
FTFY
[ link to this | view in chronology ]
Re: I blame economists: "Economic Development Administration"
[ link to this | view in chronology ]
Re: Re: I blame economists: "Economic Development Administration"
[ link to this | view in chronology ]
Re: I blame economists: "Economic Development Administration"
Basically, you're not looking to say anything truthful or correct, you're merely looking for a response. So, by your criteria, if you were to say that it's all right to rape little kids, and you got responses denouncing you for this bullshit, you would still say that your comment is effective, merely for receiving responses.
Basically, to you, there is no bad PR.
[ link to this | view in chronology ]
"The EDA's overreaction is, well, a little alarming. Although not entirely to blame�the Department of Commerce's initial communication with EDA grossly overstated the severity of the problem (though corrected its error the following day)�the EDA systematically reacted in the worst possible way. The agency demonstrated serious technical misunderstandings�it shut down its e-mail servers because some of the e-mails on the servers contained malware, even though this posed no risk to the servers themselves�and a general sense of alarmism.
The malware that was found was common stuff. There were no signs of persistent, novel infections, nor any indications that the perpetrators were nation-states rather than common-or-garden untargeted criminal attacks. The audit does, however, note that the EDA's IT infrastructure was so badly managed and insecure that no attacker would need sophisticated attacks to compromise the agency's systems."
So this wasn't even an attack. This was on the level of your common variety malware worms and phishing spam stuff that they were destroying HARDWARE over.
Also note: The NSA in all their infinite wisdom (because afterall THEY should know because they monitor and know EVERYTHING) chimed in to report that they were "concerned" about this. And they should be trusted with keeping everyone's data and using it appropriately?
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re: Re:
Well that would work.
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Re:
I haven't found your three effective in quite a while.
[ link to this | view in chronology ]
Re: Re:
[ link to this | view in chronology ]
[ link to this | view in chronology ]
One Mil
[ link to this | view in chronology ]
Destruction of brains
If you read the report cited, and the Ars article, it was because the department head did not realize/know/care that it was 'miscommunication' and incorrect information.
They destroyed nearly half of their systems for not understanding the information, and going insane over it.
The term "going nuclear" fits this one. They wasted 2.7 million dollars out of paranoid stupidity.
Those responsible will probably be promoted with a raise. Don't laugh-it's probable.
[ link to this | view in chronology ]
http://www.youtube.com/watch?v=zvfD5rnkTws
[ link to this | view in chronology ]
Build your own secret OS and apps, morons.
[ link to this | view in chronology ]
Now I know....
http://www.youtube.com/watch?v=zvfD5rnkTws
[ link to this | view in chronology ]
[ link to this | view in chronology ]
fix root of problem
[ link to this | view in chronology ]
[ link to this | view in chronology ]
cost
cost to hire a cheap illegal mexican to put it in with a camera watching him 50$
look on govts face spending 2.7 million
PRICELESS
for everything else there is facebook
[ link to this | view in chronology ]