Your Tax Dollars At Work: How Commerce Dept. Spent $2.7 Million Cleaning Out Two Malware-Infected Computers

from the burning-a-hole-in-taxpayers'-pockets dept

The cyber-Pearl Harbor is upon us and the only way to defeat it is to sink our own ships at the first sign of invasion. This is the sort of thing that happens when the legislators and advisors with the loudest voices value paranoia over rational strategy. The Department of Commerce, aided by a tragicomic string of errors, managed to almost stamp out its malware problem.

The Commerce Department's Economic Development Administration spent almost half of its IT budget last year to remediate a cyber attack that barely happened.

EDA's drastic steps to limit the damage by shutting down much of the access to the main Herbert Hoover Building network ended up costing the agency more than $2.7 million to clean up and reconfigure its network and computers. The IG said the bureau destroyed more than $170,000 in IT equipment, including desktop computers, printers, keyboards and mice.
Also included in the mass destruction were cameras and TVs. It wasn't just cyber-paranoia that led to this hardware cull. There was plenty of miscommunication too, along with the usual doses of bureaucratic clumsiness. The Inspector General's report breaks down the chain of missteps, which all began with a response team member grabbing the wrong network info.
In an effort to identify infected components, DOC CIRT’s (Dept. of Commerce Computer Incident Response Team) incident handler requested network logging information. However, the incident handler unknowingly requested the wrong network logging information... Instead of providing EDA a list of potentially infected components, the incident handler mistakenly provided EDA a list of 146 components within its network boundary. Accordingly, EDA believed it faced a substantial malware infection.
Yes. Much like "Reply" and "Reply All" will both get the job done, only one is the correct choice when firing off a devastating critique of your soon-to-be-former coworkers. The same goes for network logs. One shows you the correct info. The other "indicates" that more than half the EDA's computers are suffering from a malware infection.

DOC CIRT did try to get this fixed, pointing out the error to the handling team and re-running the analysis using the correct network log. Turns out, the original estimate was slightly off.
The HCHB network staff member then performed the appropriate analysis identifying only two components exhibiting the malicious behavior in US-CERT’s alert.
This new data in hand, a notification was sent out ostensibly to clear things up, but this too was mishandled so badly someone unfamiliar with bureaucratic ineptitude might be inclined to suspect sabotage.
DOC CIRT’s second incident notification did not clearly explain that the first incident notification was inaccurate. As a result, EDA continued to believe a widespread malware infection was affecting its systems.

Specifically, the second incident notification began by stating the information previously provided about the incident was correct. EDA interpreted the statement as confirmation of the first incident notification, when DOC CIRT’s incident handler simply meant to confirm EDA was the agency identified in US-CERT’s alert. Nowhere in the notification or attachment does the DOC CIRT incident handler identify that there was a mistake or change to the previously provided information.

Although the incident notification’s attachment correctly identified only 2 components exhibiting suspicious behavior—not the 146 components that DOC CIRT initially identified—the name of the second incident notification’s attachment exactly matched the first incident notification’s attachment, obscuring the clarification.
For five weeks, things went from bad to worse to comically tragic to tragically comic to full-scale computercide. Looking at its list (2 components), DOC CIRT asked the EDA to attempt containment by reimaging the infected items. Looking at its list (146 components), the EDA responded that reimaging half its devices would be "unfeasible." Taking a look at the EDA's list (from the first, mistaken network log analysis), DOC CIRT assumed the EDA had received additional analysis indicating the malware had spread, and changed its recommendations accordingly.

Finally, both departments were on the same (but entirely wrong) page and scaled up the response accordingly. A copy went to the DHS, stating that "over 50%" of the EDA's devices were infected. The DHS then accepted this without seeking independent confirmation. The NSA cranked out its own concerned report, quoting heavily from the DHS report (which was still in draft form), both of which were based on DOC CIRT's first erroneous report. This went undetected for over a year, until the OIG informed the involved agencies of its findings in December 2012.

The end result? The EDA and DOC CIRT worked together, attempting to head off a "severe" malware threat before it spread to other connected government computers. Despite gathering more information from outside consultants that indicated the malware was neither "persistent" nor a threat to migrate, the two agencies began destroying devices in May of 2012, finally stopping three months later when the "break stuff" budget had been exhausted.

Fortunately for the agencies, taxpayers and the surviving equipment (valued at over $3 million), the OIG's findings were brought to the agencies' attention before the fiscal year began and a new "break stuff" budget approved. All in all, the EDA spent over $2.7 million fighting a malware "infection" confined to two computers.

There's nothing in this report that makes the EDA look good. A chart on page 8 shows the EDA has persistently ignored the OIG's recommendations on agency computer security, with some assessments going back as far as 2006. It's no surprise it managed to (along with the Dept. of Commerce's response team) transform a 2-computer infection into a nearly $3 million catastrophe.

Hide this

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: commerce department, malware, miscommunication


Reader Comments

Subscribe: RSS

View by: Time | Thread


  • icon
    Akari Mizunashi (profile), 9 Jul 2013 @ 6:18am

    I'm betting the equipment was destroyed with $35,000 hammers, supervised by $100,000 consultants.

    link to this | view in chronology ]

    • icon
      Anonymous Howard (profile), 9 Jul 2013 @ 8:17am

      Re:

      I tend to agree with you on this.

      How the holy fuckin' cow peripherials like mouse and keyboard got destroyed in the process, if not for the sake of "destroying" equipment.

      Probably someone needed stuff and panicked destruction of misreported equipment were the most easiest way.

      1. Look up who the fuck handled the two reports.
      2. Check who did the destroying of the equipment
      3. Check the new equipment vendors

      I bet there would be some connection between 1-2 or 1-3.

      link to this | view in chronology ]

      • identicon
        Anonymous Coward, 9 Jul 2013 @ 9:11am

        Re: Re:

        Good heavens, I think I knew more about IT when I was 5 years old than these people.

        I actually hope that the equipment was destroyed as part of some backroom deal, and not out of stupidity. If the latter is true, than those responsible should be fired, and banned from ever touching a computer again!

        Seriously, in a case of a malware infection, the most you ever do is format and reinstall, unless the computer in question is junk, and you wanted to get rid of it anyway.

        link to this | view in chronology ]

        • identicon
          Anonymous Coward, 9 Jul 2013 @ 9:57am

          Re: Re: Re:

          Well, if it had a backdoor installed it could be hard to close the hole.

          The official response on hijackthis logs is that if you have had certain backdoors, the only way to be certain of avoiding a reinfection is destroying the equipment.

          link to this | view in chronology ]

          • icon
            Josh in CharlotteNC (profile), 9 Jul 2013 @ 10:24am

            Re: Re: Re: Re:

            I think you missed this part:

            "Despite gathering more information from outside consultants that indicated the malware was neither "persistent" nor a threat to migrate"

            Yes, there are some extremely nasty malware variants that are persistant to the (sorta-)hardware level that lodge themselves in the (not-)ROM of a motherboard or other device. But those types were not involved in this incident - it was boring standard malware and email spam. A reimaging of the infected machines would've solved the problem.

            link to this | view in chronology ]

          • identicon
            Anonymous Coward, 9 Jul 2013 @ 10:37am

            Re: Re: Re: Re:

            The only way a backdoor can survive a hard-drive wipe is if it is resident in bios, in that case re-flashing the bios should clear it out, the truly paranoid may want to replace the bios chip. There is rarely, if ever, a reason to destroy working equipment.

            By the way, the only way to wipe a hard drive is using old Uinux/Linux tools like dd: dd if = /dev/zero of = /dev/sda, or something similar. The standard windows format command wont do, and can leave malware in hidden portions of the disk (other partitions, boot sector, partition table, etc).

            link to this | view in chronology ]

            • icon
              John Fenderson (profile), 9 Jul 2013 @ 10:40am

              Re: Re: Re: Re: Re:

              the only way to wipe a hard drive is using old Uinux/Linux tools like dd


              There are Windows tools to do this as well (there's even a port of dd for those who like to kick it old-school). A half hour in a bulk eraser does a pretty good job, too.

              link to this | view in chronology ]

              • identicon
                Anonymous Coward, 9 Jul 2013 @ 10:47am

                Re: Re: Re: Re: Re: Re:

                Of course, but dd is the first thing I thought of, since I use it so often, its such a nifty little program.

                Sorry for the spelling derp, its Unix not Uinux.

                link to this | view in chronology ]

          • identicon
            Anonymous Coward, 9 Jul 2013 @ 12:49pm

            Re: Re: Re: Re:

            or you know, replace the hard drive.

            link to this | view in chronology ]

  • identicon
    Anonymous Coward, 9 Jul 2013 @ 7:52am

    hiding something?

    The kind of paranoia that drives such crazy destruction and money wasting is usually a result of someone who is desperately trying to keep something hidden.

    link to this | view in chronology ]

  • icon
    Rikuo (profile), 9 Jul 2013 @ 7:53am

    I blame out_of_the_blue on this. (S)He's the person who woke up one morning, saw someone had done a search on Google and lost his/her mind (if they even had one to begin with is a question best left for philosophers).

    link to this | view in chronology ]

  • icon
    Trelly (profile), 9 Jul 2013 @ 7:56am

    Please tell me ...

    ... that the two infected computers were on the list of 146 components that were smashed.

    The only thing that could make the story better is if the original 2 infected computers still remain operational and unmolested by re-imaging.

    link to this | view in chronology ]

  • icon
    TaCktiX (profile), 9 Jul 2013 @ 8:06am

    The effects of ignorance

    We allow computers to be everpresent while not informing actual users of how to use them in anything but the most basic contexts. If someone who WASN'T CIRT (and preferably in EDA itself) had had a tech-inclined brain at all they would have thrown up a red flag about inconsistencies in this situation and dug up the same thing the IG did without the accompanying waste.

    link to this | view in chronology ]

  • identicon
    Anonymous Coward, 9 Jul 2013 @ 8:12am

    They should have spent that money on facebook likes instead.

    link to this | view in chronology ]

  • identicon
    Anonymous Coward, 9 Jul 2013 @ 8:12am

    Uh...where did the money go?

    They blew up $170.000 in equipment. Where's the rest of the money?

    Something smells fishy here. I'm guessing the money went to some contractor that was also conveniently a friend or a family member of someone at the top.

    link to this | view in chronology ]

    • icon
      Capitalist Lion Tamer (profile), 9 Jul 2013 @ 8:24am

      Re:

      P.16 has the breakdown of the $2.7 million. A little over $1.5 million went to two contractors and another $1 million+ to "temporary infrastructure, pending long-term solution."

      link to this | view in chronology ]

      • identicon
        Anonymous Coward, 9 Jul 2013 @ 8:34am

        Re: Re:

        So...the rule now is that we read the sources?

        :)

        link to this | view in chronology ]

      • identicon
        Anonymous Coward, 9 Jul 2013 @ 8:43am

        Re: Re:

        The two contractors should now be charged with war profiteering since this is a "cyber-war" and all.

        link to this | view in chronology ]

        • identicon
          Call me Al, 9 Jul 2013 @ 8:55am

          Re: Re: Re:

          While I wouldn't go so far as to have them arrested I'd certainly question their competence.

          Surely at some point they'd have said "we haven't found any malware on these computers"

          link to this | view in chronology ]

  • icon
    Coogan (profile), 9 Jul 2013 @ 8:17am

    Is anybody else envisioning a bunch of nerds running around the EDA offices double-time with axes and baseballs bats while the Benny Hill music plays in the background?

    link to this | view in chronology ]

  • This comment has been flagged by the community. Click here to show it
    identicon
    out_of_the_blue, 9 Jul 2013 @ 8:19am

    I blame economists: "Economic Development Administration"

    The overweening conceit of economists is that they understand everything, and it's all connected, such as Mike held yesterday: that Google being taxed by Germany causes censorship in China!

    http://www.techdirt.com/articles/20130605/06245023322/china-once-again-using-censorship-el sewhere-to-justify-oppressive-great-firewall-china.shtml#c24

    (I see that my point was so effective that Mike hisself made a very rare response to my post. Heh, heh.)

    link to this | view in chronology ]

    • icon
      Rikuo (profile), 9 Jul 2013 @ 8:35am

      Re: I blame economists: "Economic Development Administration"

      "that Google being taxed by Germany causes censorship in China!"

      "that Google being taxed by Germany is used as justification by the Chinese government to do censorship over their people"

      FTFY

      link to this | view in chronology ]

    • identicon
      Anonymous Coward, 9 Jul 2013 @ 8:38am

      Re: I blame economists: "Economic Development Administration"

      Effective =/= Correct or Truthful at all

      link to this | view in chronology ]

      • icon
        Rikuo (profile), 9 Jul 2013 @ 8:41am

        Re: Re: I blame economists: "Economic Development Administration"

        Above was me, somehow I got signed out between comments.

        link to this | view in chronology ]

    • icon
      Rikuo (profile), 9 Jul 2013 @ 8:44am

      Re: I blame economists: "Economic Development Administration"

      "I see that my point was so effective"

      Basically, you're not looking to say anything truthful or correct, you're merely looking for a response. So, by your criteria, if you were to say that it's all right to rape little kids, and you got responses denouncing you for this bullshit, you would still say that your comment is effective, merely for receiving responses.
      Basically, to you, there is no bad PR.

      link to this | view in chronology ]

  • identicon
    Anonymous Coward, 9 Jul 2013 @ 8:27am

    From the ArsTechnica article on the same story...

    "The EDA's overreaction is, well, a little alarming. Although not entirely to blame—the Department of Commerce's initial communication with EDA grossly overstated the severity of the problem (though corrected its error the following day)—the EDA systematically reacted in the worst possible way. The agency demonstrated serious technical misunderstandings—it shut down its e-mail servers because some of the e-mails on the servers contained malware, even though this posed no risk to the servers themselves—and a general sense of alarmism.

    The malware that was found was common stuff. There were no signs of persistent, novel infections, nor any indications that the perpetrators were nation-states rather than common-or-garden untargeted criminal attacks. The audit does, however, note that the EDA's IT infrastructure was so badly managed and insecure that no attacker would need sophisticated attacks to compromise the agency's systems."

    So this wasn't even an attack. This was on the level of your common variety malware worms and phishing spam stuff that they were destroying HARDWARE over.

    Also note: The NSA in all their infinite wisdom (because afterall THEY should know because they monitor and know EVERYTHING) chimed in to report that they were "concerned" about this. And they should be trusted with keeping everyone's data and using it appropriately?

    link to this | view in chronology ]

  • identicon
    Anonymous Coward, 9 Jul 2013 @ 8:42am

    I could of fixed this for free with a little one-two-three punch known as AVG-AdAware-SpyBot.

    link to this | view in chronology ]

    • icon
      akp (profile), 9 Jul 2013 @ 9:11am

      Re:

      I think you meant rkill-Malwarebytes-MSE.

      I haven't found your three effective in quite a while.

      link to this | view in chronology ]

      • identicon
        Anonymous Coward, 9 Jul 2013 @ 10:24am

        Re: Re:

        I agree with you. I haven't tried rkill though. I will have to take a look.

        link to this | view in chronology ]

  • identicon
    Rich, 9 Jul 2013 @ 9:10am

    That's funny. Where I work, in our classified labs, they even put "secret" labels on the mice!

    link to this | view in chronology ]

  • identicon
    Anonymous Coward, 9 Jul 2013 @ 9:17am

    One Mil

    I would of fixed it for only one million and given them a guarantee too!

    link to this | view in chronology ]

  • identicon
    FM Hilton, 9 Jul 2013 @ 9:45am

    Destruction of brains

    "I actually hope that the equipment was destroyed as part of some backroom deal, and not out of stupidity. If the latter is true, than those responsible should be fired, and banned from ever touching a computer again!"

    If you read the report cited, and the Ars article, it was because the department head did not realize/know/care that it was 'miscommunication' and incorrect information.

    They destroyed nearly half of their systems for not understanding the information, and going insane over it.

    The term "going nuclear" fits this one. They wasted 2.7 million dollars out of paranoid stupidity.

    Those responsible will probably be promoted with a raise. Don't laugh-it's probable.

    link to this | view in chronology ]

  • identicon
    Anonymous Coward, 9 Jul 2013 @ 9:52am

    Weird Al's Virus Alert song comes to mind with their reaction.

    http://www.youtube.com/watch?v=zvfD5rnkTws

    link to this | view in chronology ]

  • identicon
    UberSmart, 9 Jul 2013 @ 10:16am

    A federal budget of trillions and these guys use Windows computers.

    Build your own secret OS and apps, morons.

    link to this | view in chronology ]

  • icon
    Berenerd (profile), 9 Jul 2013 @ 10:22am

    Now I know....

    ...why this song was written...


    http://www.youtube.com/watch?v=zvfD5rnkTws

    link to this | view in chronology ]

  • identicon
    6, 9 Jul 2013 @ 10:48am

    Well, that is embarrasing.

    link to this | view in chronology ]

  • icon
    aldestrawk (profile), 9 Jul 2013 @ 2:46pm

    fix root of problem

    The root of the problem is clearly deep-set paranoia brought on by working in a building named after Herbert Hoover. The only true fix will be to tear down this building, build a new one, and never name anything after Herbert Hoover again.

    link to this | view in chronology ]

  • icon
    btrussell (profile), 9 Jul 2013 @ 3:36pm

    $1.35 million sounds about right. That is what I figure sony owes me for the root-kit they installed on MY computer.

    link to this | view in chronology ]

  • identicon
    Guardian, 9 Jul 2013 @ 6:55pm

    cost

    NEW hard drive....80$
    cost to hire a cheap illegal mexican to put it in with a camera watching him 50$

    look on govts face spending 2.7 million
    PRICELESS
    for everything else there is facebook

    link to this | view in chronology ]


Follow Techdirt
Essential Reading
Techdirt Deals
Report this ad  |  Hide Techdirt ads
Techdirt Insider Discord

The latest chatter on the Techdirt Insider Discord channel...

Loading...
Recent Stories

This site, like most other sites on the web, uses cookies. For more information, see our privacy policy. Got it
Close

Email This

This feature is only available to registered users. Register or sign in to use it.