NSA's Massive Utah Datacenter Having Serious Electrical Problems: Has Already Had 10 Fiery Explosions

from the frying-our-data dept

It's no secret that government computing projects tend to be something of a boondoggle for government contractors. There are way too many stories of projects that are massively over-budget while never actually working. It's almost to be expected these days. Still, when it came to the NSA's infamous Utah data center they've been building over the past few years, you would have thought that maybe these guys would plan things out a little more carefully. While we can mock the NSA for the lies and misleading statements they make to the public, most people agree that they do have pretty damn good technical skills.

But, no, it appears that the Bluffdale data center is a complete mess. The data center, which was supposed to open up last month, has apparently been massively delayed due to major electrical problems -- and we're not just talking about some issues with not having enough power, but with setting stuff on fire:
According to the Wall Street Journal, the data center's electrical problems include "arc failures," a.k.a. "a flash of lightning inside a 2-foot box," which results in fiery explosions, melted metal and circuit failure. More terrifying, this has happened ten times, most recently on September 25, reports the WSJ, which reviewed project documents and reports and talked to contractors involved. The report blames the NSA "fast tracking" the Utah project and thus bypassing "regular quality controls in design and construction." Whoops.
Whoops indeed. Apparently the NSA was in such a rush to store all our data that it almost burned down its own data center. Good thing they're getting a tax break on all that electricity they're using.

Also, it appears that there's a fair bit of sniping going on, as some people claim they knew this was going to happen all along, while others say they have no idea why it's happening:
Worse, it sounds from the WSJ's reporting as if the contractors — architectural firm KlingStubbins which designed the electrical system, along with construction companies Balfour Beatty Construction, DPR Construction and Big-D Construction Corp — are still scrambling to figure out what's causing the problems. The Army Corps of Engineers sent its "Tiger Team" to sort things out this summer but they were unable to pinpoint exactly what's wrong.

"The problem, and we all know it, is that they put the appliances too close together," a person familiar with the database construction told FORBES, describing the arcs as creating "kill zones." "They used wiring that's not adequate to the task. We all talked about the fact that it wasn't going to work."
So, while the NSA has all sorts of code-breaking specialists, it appears that they're a bit understaffed on electrical engineers... Meanwhile, how long until peeved Iranian government officials pretend that they did this in response to the NSA creating Stuxnet to mess with their nuclear processing powers...
Hide this

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: arc failures, bluffdale, datacenter, electrical problems, nsa, nsa surveillance, utah


Reader Comments

Subscribe: RSS

View by: Time | Thread


  1. identicon
    Alt0, 8 Oct 2013 @ 12:19pm

    Looks like they will have to settle for one or two "hops" for a while.

    link to this | view in thread ]

  2. identicon
    zem, 8 Oct 2013 @ 12:27pm

    Who would of thunk it

    Seams

    liar, liar, pants on fire

    is true after all

    link to this | view in thread ]

  3. icon
    Richard (profile), 8 Oct 2013 @ 12:27pm

    Bluffdale

    But, no, it appears that the Bluffdale data center is a complete mess.


    Ah - I get it the bluffdale data centre is not their real data centre - the clue is

    link to this | view in thread ]

  4. icon
    Richard (profile), 8 Oct 2013 @ 12:28pm

    Re: Bluffdale

    ...in the name....

    link to this | view in thread ]

  5. icon
    limbodog (profile), 8 Oct 2013 @ 12:29pm

    Wonder if this is at all like when Iran's nuclear fuel rod centrifuge issues.

    link to this | view in thread ]

  6. identicon
    Anonymous Coward, 8 Oct 2013 @ 12:30pm

    Hey Republicans, this is a clear sign God doesn't want NSA to have this much surveillance power, either. How about you vote/protest against it?

    link to this | view in thread ]

  7. identicon
    Anonymous Coward, 8 Oct 2013 @ 12:30pm

    Guess this is what happens when you concentrate 10 million watts of unconstitutional power in one location.

    link to this | view in thread ]

  8. identicon
    Anonymous Coward, 8 Oct 2013 @ 12:36pm

    Would any of the component be marked 'Made in China' perchance?

    All the contractors and Army people won't have the correct clearances for an operational NSA facility, so that whole place probably can't accomplish anything until they are all gone...

    link to this | view in thread ]

  9. identicon
    Anonymous Coward, 8 Oct 2013 @ 12:37pm

    It's a sign that the surveillance state was never meant to be.

    link to this | view in thread ]

  10. icon
    Internet Zen Master (profile), 8 Oct 2013 @ 12:38pm

    Would never happen... I think...

    Anyone want to bet that the NSA might conduct an "investigation" and then claim that all these problems and failures were the result of sabotage by none other than... wait for it... EDWARD SNOWDEN. And if anyone asks questions about their investigative methods, they will respond with: "Trust us. We're the NSA. Now stop asking questions."

    link to this | view in thread ]

  11. identicon
    Anonymous Coward, 8 Oct 2013 @ 12:49pm

    So they are taking short cuts to typical standards...sounds like their MO. Hopefully all the 'just metadata' get wiped.

    link to this | view in thread ]

  12. identicon
    Crusty the Ex-Clown, 8 Oct 2013 @ 12:50pm

    Code breakers. Heh.

    "So, while the NSA has all sorts of code-breaking specialists, it appears that they're a bit understaffed on electrical engineers.."

    Maybe they're trying to break the NEC.....

    link to this | view in thread ]

  13. identicon
    Anonymous Coward, 8 Oct 2013 @ 12:52pm

    i reckon it's all that meta data gone up in smoke!!

    link to this | view in thread ]

  14. icon
    Corwin (profile), 8 Oct 2013 @ 1:04pm

    New Leak!

    So THIS is the program they call EPICFAIL!

    The ones about the (DUMBUSERS).

    link to this | view in thread ]

  15. icon
    Transmitte (profile), 8 Oct 2013 @ 1:13pm

    Utah Data Center = Tax payer quagmire.

    Too bad no one will have to answer for this mess. More to the point, too bad this place is even in existence.

    link to this | view in thread ]

  16. icon
    Tom Betz (profile), 8 Oct 2013 @ 1:22pm

    This is a feature, not a bug

    Anyone who has been paying attention knows that the NSA and all the other three-letter agencies have since the Reagan administration been little more than vacuum hoses designed to siphon money out of taxpayers' pockets into the bank accounts of the connected ultra-wealthy who own the contracting companies. All this money being spent on surveillance isn't about security at all. That's just the lie told to justify it to outsiders. Screwups like this just mean more money for the contractors that fix them, usually the same ones who screwed up in the first place.

    link to this | view in thread ]

  17. icon
    Blatant Coward (profile), 8 Oct 2013 @ 1:24pm

    The industrial wiring code is not a good one to break.

    link to this | view in thread ]

  18. icon
    Josh in CharlotteNC (profile), 8 Oct 2013 @ 1:32pm

    Re:

    The NSA: home of groundbreaking crypto and ground breaking electronics.

    or

    Yo dawg, I heard you like breaking and wires so I put some breaks in your wiring, so you can break ground in your datacenter after you've broken ground on your data center.

    link to this | view in thread ]

  19. identicon
    William Munny, 8 Oct 2013 @ 1:39pm

    NSA's plan to eliminate 90% of their system admins is looking smarter by the moment.

    link to this | view in thread ]

  20. identicon
    Anonymous Coward, 8 Oct 2013 @ 1:58pm

    Burn, motherfucker, BURN!

    link to this | view in thread ]

  21. identicon
    Anonymous Coward, 8 Oct 2013 @ 2:00pm

    2 foot box x 10 wow.. small 'problem'

    have you seen the size of that complex ??

    10 2 foot boxes flashing over, is probably LESS than what I would have expect from a place that large.

    it is NOT very much..

    it's a SMALL switch board that sits inside a 2 foot box, and ANY switch will arc with switched wile carrying current.
    Thus the "flashes of lightning".

    They are not talking about wiring meltdown, or switchboard fires.

    In fact only 10 small boxes have failed (or might have failed) is next to nothing..

    But just keep feeding us background noise, that way any real story is lost in the noise.

    link to this | view in thread ]

  22. identicon
    Anonymous Coward, 8 Oct 2013 @ 2:06pm

    Goes to show... all the money in the world does not solve incompetence.

    link to this | view in thread ]

  23. identicon
    Anonymous Coward, 8 Oct 2013 @ 2:19pm

    Re: 2 foot box x 10 wow.. small 'problem'

    " a.k.a. "a flash of lightning inside a 2-foot box," which results in fiery explosions, melted metal and circuit failure.

    10 times is symptom of a badly specified system, and is not a minor problem.

    link to this | view in thread ]

  24. identicon
    Anonymous Coward, 8 Oct 2013 @ 2:21pm

    Re:

    I'm surprised they haven';t created a Doc Brown Time Paradox yet.

    link to this | view in thread ]

  25. icon
    Richard (profile), 8 Oct 2013 @ 2:29pm

    "...frying-our-data..."

    Don't know whether to lament my wasted tax dollars or celebrate the delayed abuses of my civil rights. Guess I'll just sit here and weep in my champagne.

    link to this | view in thread ]

  26. icon
    That One Guy (profile), 8 Oct 2013 @ 2:34pm

    When even Zeus and Thor are against you...

    It might be time to pack it up and admit defeat.

    link to this | view in thread ]

  27. icon
    John Fenderson (profile), 8 Oct 2013 @ 2:40pm

    Re: 2 foot box x 10 wow.. small 'problem'

    They are not talking about wiring meltdown, or switchboard fires.


    Just an arc event that happened a number of times and destroyed $100,000 worth of equipment each time it did.

    It's only almost $1,000,000 of destroyed equipment. No story here.

    link to this | view in thread ]

  28. identicon
    PRMan, 8 Oct 2013 @ 2:44pm

    Re:

    I wish God would shut down the NSA.

    link to this | view in thread ]

  29. identicon
    Anonymous Coward, 8 Oct 2013 @ 3:23pm

    Re: Re: 2 foot box x 10 wow.. small 'problem'

    no, sub-station meltdowns or fires, cable tray fires, feeder runs melting is a major problem. Some arcing in 10 small switching boxes, not so much.

    lets just face some basic facts first please.

    this is no Tacoma Narrows bridge.

    link to this | view in thread ]

  30. identicon
    Anonymous Coward, 8 Oct 2013 @ 3:26pm

    Re: Re: 2 foot box x 10 wow.. small 'problem'

    if the wiring is sub-standard, then why are the switchboards arcing over. And why not cable fires ?

    Why only 10, and they are really small switchboards indeed most are the size of a large room.

    again, this is a lost in noise non-story, but it does have NSA in it....

    link to this | view in thread ]

  31. identicon
    Anonymous Coward, 8 Oct 2013 @ 3:44pm

    Propaganda to pacify the world.

    link to this | view in thread ]

  32. identicon
    Anonymous Coward, 8 Oct 2013 @ 3:48pm

    Star Trek

    So, not only does the NSA have a Star Trek bridge, they also have the Explosive Instrumentation to go with it!

    link to this | view in thread ]

  33. identicon
    Anonymous Coward, 8 Oct 2013 @ 3:53pm

    Re: Made in XXXX

    More Likely the Components are All marked "Proudly Made in USA".

    Was there ever a time that the Best stuff was made in America?

    link to this | view in thread ]

  34. identicon
    Anonymous Coward, 8 Oct 2013 @ 3:56pm

    NSA,: the home of Broken Standards

    Encryption standards were not the only standards bankdoored

    link to this | view in thread ]

  35. identicon
    Anonymous Coward, 8 Oct 2013 @ 4:18pm

    Re:

    Who knew they were going to eliminate them by electrocution though?

    link to this | view in thread ]

  36. icon
    art guerrilla (profile), 8 Oct 2013 @ 4:34pm

    Re:

    of course, the very un-funny point about your joke, is that repugs, faux news propagandists, xtian water carriers, etc, *would* make that exact 'argument', were it to be some 'liberal' building of some sort (social security bldg? planned parenthood HQ?) that had MUCH less worse issues...

    (un)funny how that doesn't work when the shoe is on the other foot...

    (10!? 10 arc-fires ! dog damn, tesla woulda been proud!)

    a faux-biblical quote comes to mind:
    none so blind, as those who will not see...

    art guerrilla
    aka ann archy
    eof

    link to this | view in thread ]

  37. icon
    DB (profile), 8 Oct 2013 @ 5:00pm

    My guess is that they are using DC power distribution.

    With AC power, the voltage crosses zero 120 times a second and a spark tends to self-extinguish. The ionized air that might sustain a spark gets pushed in the opposite direction, and the voltage doesn't rise fast enough to start a fresh arc.

    With DC power, there is no zero cross and the ionized air stays in place, sustaining an arc.

    A switch or relay that has a AC voltage rating of 250V might only have a DC voltage rating 24V. Specialized high voltage DC relays typically combine several techniques to extinguish the arc that forms when contacts open. They always have contacts that open at high speed, and they usually use a 'blowout' mechanism. There might be several contacts in series, opened simultaneously, to effectively increase the opening speed. The might have the contacts in a vacuum or under high gas pressure. The blow-out might be a "magnetic blowout" which uses a magnetic field to push the arc into a longer path, a blast of compressed air, or even a blank shotgun shell to blow the ionized path away from the contacts.

    Back to the point: design engineers that are used to HVAC systems usually get it wrong when they assume that they can use their knowledge with much lower voltage DC systems.

    link to this | view in thread ]

  38. identicon
    Anonymous Coward, 8 Oct 2013 @ 5:09pm

    I'm going to make the guess they got scared with the Snowden stuff flying around and the Amish proposal and decided to hurry it up before the money ran out. Don't worry about the quality control, we'll fix that later type thing. I think they were in a hurry to get it up and running while still having boodles of money to throw at it.

    link to this | view in thread ]

  39. icon
    Bergman (profile), 8 Oct 2013 @ 5:23pm

    Building codes exist for a reason

    Sure, they can feel like useless bureaucratic red tape, but they really do exist for good reasons.

    As the NSA is learning, just because you legally CAN skip building to code, doesn't mean you SHOULD!

    link to this | view in thread ]

  40. icon
    JMT (profile), 8 Oct 2013 @ 5:50pm

    Right people for the job?

    "...architectural firm KlingStubbins which designed the electrical system..."

    Well there's your problem!

    link to this | view in thread ]

  41. icon
    Chronno S. Trigger (profile), 8 Oct 2013 @ 8:27pm

    Re: Re: Re: 2 foot box x 10 wow.. small 'problem'

    " a.k.a. "a flash of lightning inside a 2-foot box," which results in fiery explosions, melted metal and circuit failure."

    You seem to be missing parts. If the description of your electrical failure involves the words "fiery explosions" then you're doing something wrong.

    If these people cannot be trusted to wire power correctly, they cannot be trusted to spy on the entirety of the United States. This is not a non-story, this is just more proof that these assholes cannot follow the rules.

    link to this | view in thread ]

  42. identicon
    Anonymous Coward, 8 Oct 2013 @ 9:58pm

    Fire, Water, Burn.

    The box, the box, the box is on fire
    The box, the box, the box is on fire
    The box, the box, the box is on fire
    We don't need no water let the motherfucker burn
    Burn motherfucker burn

    link to this | view in thread ]

  43. icon
    Chris-Mouse (profile), 9 Oct 2013 @ 4:16am

    So the NSA's expertise at code breaking extends to breaking electrical building codes as well.

    link to this | view in thread ]

  44. identicon
    Pragmatic, 9 Oct 2013 @ 6:07am

    Meanwhile, how long until peeved Iranian government officials pretend that they did this in response to the NSA creating Stuxnet to mess with their nuclear processing powers...


    About five minutes before some drunk-with-power politician announces that they are in fact responsible and this is why we should nuke 'em from orbit. Just to make sure.

    link to this | view in thread ]

  45. identicon
    Anonymous Coward, 10 Oct 2013 @ 8:35am

    'Murica!

    link to this | view in thread ]

  46. icon
    Anonymous Howard (profile), 11 Oct 2013 @ 2:29am

    C'mon

    "Meanwhile, how long until peeved Iranian government officials pretend that they did this in response to the NSA creating Stuxnet to mess with their nuclear processing powers..."

    -10 points from Mike for Unnecessary Ad Hominem.

    link to this | view in thread ]

  47. identicon
    Anstee Electrical, 27 May 2014 @ 4:23am

    Electrician in Knoxfield

    Hey! nice article our company Anstee Electrical Provide best electrician in Knoxfield For other detail visit here http://www.ansteeelectrical.com.au/

    link to this | view in thread ]

  48. identicon
    venki, 9 Jul 2014 @ 4:46am

    NSA itself having electricity problem

    Hello,
    Good job! I can't belive that NSA itself has serious electricity issues and they are giving tax free for the electricity they consuming, why can't they go for better ways such as inverters,ups or solar system etc. to solve these type of issues.
    Thanks & Regards,
    venki

    link to this | view in thread ]

  49. identicon
    pca6661, 28 Nov 2014 @ 8:18pm

    NSA can't run a data center?

    I work in a carrier hotel. We don't have arc fires ever let alone on a regular basis.

    link to this | view in thread ]

  50. icon
    Tanner Andrews (profile), 3 Jun 2015 @ 4:58am

    Site is Called ``Bumblehive'' for a Reason

    The ``hive'' comes from Utah, the beehive state. As for bumble, well, some people thought it referred to a variety of bee. No such luck, it refers to the operators of the site.

    link to this | view in thread ]


Follow Techdirt
Essential Reading
Techdirt Deals
Report this ad  |  Hide Techdirt ads
Techdirt Insider Discord

The latest chatter on the Techdirt Insider Discord channel...

Loading...
Recent Stories

This site, like most other sites on the web, uses cookies. For more information, see our privacy policy. Got it
Close

Email This

This feature is only available to registered users. Register or sign in to use it.