Data Center Outage Puts Spotlight On Power Systems

from the the-other-grid dept

This week's outage of a major San Francisco data center has prompted a lot of discussion about the tech industry's massive energy requirements, and whether or not the existing energy infrastructure will continue to prove satisfactory. Although we blamed excessive hubris for the crash, some are pointing fingers at the utility PG&E, for letting the underlying power outage happen. Of course, this doesn't explain why 365 Main's extensive energy backup system failed to kick in as it was supposed to. Either way, it's likely that continued investments into energy systems are in order. IBM has been investing in technology that will reduce the energy demands of data centers, but the trend is helping low-tech firms as well: yesterday, engine maker Cummins reported strong earnings, due in part to the sale of generators to data centers. All of this is further evidence that tech firms are increasingly forced to get down and dirty with tangible, physical goods in order to stay competitive.
Hide this

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: energy
Companies: cummins, pg&e


Reader Comments

Subscribe: RSS

View by: Time | Thread


  • identicon
    SPR, 27 Jul 2007 @ 9:39am

    Emergency Power Backup Systems

    This is ALWAYS included in any significant IT structure requiring ready access for critical data. Apparently they did not believe their data was critical.

    link to this | view in chronology ]

    • identicon
      Bob, 27 Jul 2007 @ 9:44am

      Re: Emergency Power Backup Systems

      Didn't you read the article? They HAD an energy backup system. It just failed to "kick in".

      link to this | view in chronology ]

  • identicon
    Overcast, 27 Jul 2007 @ 9:46am

    Funny now how people 'expect' machines to work right all the time...

    It's an expectation that's only result will be disappointment.

    link to this | view in chronology ]

  • identicon
    Bigmunny, 27 Jul 2007 @ 9:57am

    Testing

    Regular testing of the system can help prevent the failed to "kick in" issue.

    link to this | view in chronology ]

    • identicon
      inc, 27 Jul 2007 @ 1:11pm

      Re: Testing

      This is true. The data center I work in regularly tests the power conditioning, batteries and generators to ensure they are going to work when we need them. It's important that the batteries have capacity for all the critical systems, including air conditioning, and that the generators come online automatically. Also an important factor is to have contract for guaranteed fuel for that generator. When Hurricane Wilma knocked out power for 2 weeks in our DC that generator was the employee of the month.

      link to this | view in chronology ]

  • identicon
    Enrico Suarve, 27 Jul 2007 @ 10:00am

    Not just the data centers

    I don't remember who posted it but in the previous thread one commentor was spot on with their analysis, that generators can spring into action all they want, but if you have an area wide power outage all they are going to do is reduce risks of data corruption from cold booting

    Basically if the entire grid goes down (as it did in this case) your connection to the data center is still only as good as the power to the first public network component not connected to your generators (i.e. still dead in the water)

    You're just a very expensive diesel powered data island

    A truly resilient solution would have to involve two completely seperate grids (and similarly seperated backup lines), or better - two very seperate locations

    link to this | view in chronology ]

  • identicon
    Unknowledgeable Geek, 27 Jul 2007 @ 10:23am

    Data Centers

    A lot of data centers I deal with are on two seperate grids. They have multiple internet lines coming in. The have multiple backup generators. I don't know what crappy center this was in San Fran, but I bet it was the cheapest option.

    link to this | view in chronology ]

  • identicon
    Wes, 27 Jul 2007 @ 10:42am

    Generation

    Backup generation with UPS can give you 100% uptime. Our company has a contract with an large tech firm for such a system. They do complex benchmarks that require constant power for over 6 months at a time. The systems exist, in this case it seems like there was a catastrophic failure at either the trasfer switches or the load study was incorrectly done.

    Businesses have a tendency to add components to a electrical system without considering the output of the electrical generators and the extra load.

    link to this | view in chronology ]

  • identicon
    Anonymous Coward, 27 Jul 2007 @ 11:45am

    Data Center Outage

    A good data center also stores it's data in more than one place.

    link to this | view in chronology ]

  • identicon
    Anonymous Coward Too, 27 Jul 2007 @ 11:57am

    Data Center Outage and Redundant Power

    My DC has redundant feeds from the street, redundant pipes, racks and racks of backup batteries. Didn't do a bit of good when a huge spike came in off the street and *vaporized* the emergency switching gear. I'm not kidding.

    No batteries, no redundant grid, no amount of testing will guarantee a no-impact failover. I've seen multiple outages at multiple sites over 20 years. The answer is IT DEPENDS. With hugs power feeds it gets complicated fast.

    You can take cheap shots if you want, doesn't mean you know anything.

    link to this | view in chronology ]

    • Re: Data Center Outage and Redundant Power

      An appropriate sized REGULATOR(s) before the panels and even UPS's are meant to absorb and take out all power spikes from city power. Sometimes, we forget to add them to our systems. Their only purpose is to prevent power spikes from continuing on to your infrastructure...

      link to this | view in chronology ]

  • identicon
    Chris Maresca, 27 Jul 2007 @ 12:50pm

    According to some insiders....

    ... the problem was not that backup failed, but that they are using 90% of the incoming PGE capacity, which is far more than the generators could sustain, something like 15% more.

    The end result was a complete power meltdown.

    Chris.

    link to this | view in chronology ]

  • identicon
    500CID, 27 Jul 2007 @ 1:01pm

    Outage cause

    Outage was due to failed Basler protection relay(s)

    link to this | view in chronology ]

  • identicon
    sean, 28 Jul 2007 @ 1:52pm

    http://seancasaidhe.wordpress.com

    This is entirely their own stupid fault. Anyone who depends entirely on telecoms for their survival had better be operating out of two or more physically diverse locations. UPS and gennies are just going to keep your servers running (although I understand that even this failed, showing a remarkably cavalier attitude on behalf of Technorati, Craiglists et. al. toward ensuring that your BCP works) - to keep your telecoms up you need to be on different telecoms networks AND different power networks, just altogether in a different city, state or country.
    I wonder if I can get a hugely overpaid consulting job advising them on the basics of continuity planning?

    link to this | view in chronology ]

  • identicon
    John W. Peace II CEcD, 5 Aug 2007 @ 8:24am

    Safest Place on Earth for Data Centers!!!

    Natural disasters, power outages, terrorism, not a problem if you locate your mission critical operations in the Safest Place on Earth!

    Take a look at:

    http://www.safestplaceonearth.net/

    link to this | view in chronology ]

  • identicon
    John Peace, 3 Jul 2009 @ 3:54pm

    Safest Place on Earth for Data Centers!

    Looking for the safest place to locate Data Centers?

    http://www.safestplaceonearth.net/

    Locate @ the power source with little risk and cheaper electrical rates!

    link to this | view in chronology ]


Follow Techdirt
Essential Reading
Techdirt Deals
Report this ad  |  Hide Techdirt ads
Techdirt Insider Discord

The latest chatter on the Techdirt Insider Discord channel...

Loading...
Recent Stories

This site, like most other sites on the web, uses cookies. For more information, see our privacy policy. Got it
Close

Email This

This feature is only available to registered users. Register or sign in to use it.