Washington Post Shows That The Newspaper Business Isn't Doomed

Data Center Outage Puts Spotlight On Power Systems

from the the-other-grid dept

Fri, Jul 27th 2007 8:24am — Joseph Weisenthal

This week's outage of a major San Francisco data center has prompted a lot of discussion about the tech industry's massive energy requirements, and whether or not the existing energy infrastructure will continue to prove satisfactory. Although we blamed excessive hubris for the crash, some are pointing fingers at the utility PG&E, for letting the underlying power outage happen. Of course, this doesn't explain why 365 Main's extensive energy backup system failed to kick in as it was supposed to. Either way, it's likely that continued investments into energy systems are in order. IBM has been investing in technology that will reduce the energy demands of data centers, but the trend is helping low-tech firms as well: yesterday, engine maker Cummins reported strong earnings, due in part to the sale of generators to data centers. All of this is further evidence that tech firms are increasingly forced to get down and dirty with tangible, physical goods in order to stay competitive.

Filed Under: energy
Companies: cummins, pg&e

18 Comments

If you liked this post, you may also be interested in...

Reader Comments

Subscribe: RSS

View by: Time | Thread

SPR, 27 Jul 2007 @ 9:39am

Emergency Power Backup Systems
This is ALWAYS included in any significant IT structure requiring ready access for critical data. Apparently they did not believe their data was critical.
[ link to this | view in thread ]
Bob, 27 Jul 2007 @ 9:44am

Re: Emergency Power Backup Systems
Didn't you read the article? They HAD an energy backup system. It just failed to "kick in".
[ link to this | view in thread ]
Overcast, 27 Jul 2007 @ 9:46am

Funny now how people 'expect' machines to work right all the time...

It's an expectation that's only result will be disappointment.
[ link to this | view in thread ]
Bigmunny, 27 Jul 2007 @ 9:57am

Testing
Regular testing of the system can help prevent the failed to "kick in" issue.
[ link to this | view in thread ]
Enrico Suarve, 27 Jul 2007 @ 10:00am

Not just the data centers
I don't remember who posted it but in the previous thread one commentor was spot on with their analysis, that generators can spring into action all they want, but if you have an area wide power outage all they are going to do is reduce risks of data corruption from cold booting

Basically if the entire grid goes down (as it did in this case) your connection to the data center is still only as good as the power to the first public network component not connected to your generators (i.e. still dead in the water)

You're just a very expensive diesel powered data island

A truly resilient solution would have to involve two completely seperate grids (and similarly seperated backup lines), or better - two very seperate locations
[ link to this | view in thread ]
Unknowledgeable Geek, 27 Jul 2007 @ 10:23am

Data Centers
A lot of data centers I deal with are on two seperate grids. They have multiple internet lines coming in. The have multiple backup generators. I don't know what crappy center this was in San Fran, but I bet it was the cheapest option.
[ link to this | view in thread ]
Wes, 27 Jul 2007 @ 10:42am

Generation
Backup generation with UPS can give you 100% uptime. Our company has a contract with an large tech firm for such a system. They do complex benchmarks that require constant power for over 6 months at a time. The systems exist, in this case it seems like there was a catastrophic failure at either the trasfer switches or the load study was incorrectly done.

Businesses have a tendency to add components to a electrical system without considering the output of the electrical generators and the extra load.
[ link to this | view in thread ]
SPR, 27 Jul 2007 @ 11:32am

Re: Re: Emergency Power Backup Systems
It's called "TESTING"!!
[ link to this | view in thread ]
Anonymous Coward, 27 Jul 2007 @ 11:45am

Data Center Outage
A good data center also stores it's data in more than one place.
[ link to this | view in thread ]
Anonymous Coward Too, 27 Jul 2007 @ 11:57am

Data Center Outage and Redundant Power
My DC has redundant feeds from the street, redundant pipes, racks and racks of backup batteries. Didn't do a bit of good when a huge spike came in off the street and *vaporized* the emergency switching gear. I'm not kidding.

No batteries, no redundant grid, no amount of testing will guarantee a no-impact failover. I've seen multiple outages at multiple sites over 20 years. The answer is IT DEPENDS. With hugs power feeds it gets complicated fast.

You can take cheap shots if you want, doesn't mean you know anything.
[ link to this | view in thread ]
death breath, 27 Jul 2007 @ 12:44pm

Re: Data Center Outage
no they don't the keep it on one floppy disk
[ link to this | view in thread ]
Chris Maresca, 27 Jul 2007 @ 12:50pm

According to some insiders....
... the problem was not that backup failed, but that they are using 90% of the incoming PGE capacity, which is far more than the generators could sustain, something like 15% more.

The end result was a complete power meltdown.

Chris.
[ link to this | view in thread ]
500CID, 27 Jul 2007 @ 1:01pm

Outage cause
Outage was due to failed Basler protection relay(s)
[ link to this | view in thread ]
inc, 27 Jul 2007 @ 1:11pm

Re: Testing
This is true. The data center I work in regularly tests the power conditioning, batteries and generators to ensure they are going to work when we need them. It's important that the batteries have capacity for all the critical systems, including air conditioning, and that the generators come online automatically. Also an important factor is to have contract for guaranteed fuel for that generator. When Hurricane Wilma knocked out power for 2 weeks in our DC that generator was the employee of the month.
[ link to this | view in thread ]
sean, 28 Jul 2007 @ 1:52pm

http://seancasaidhe.wordpress.com
This is entirely their own stupid fault. Anyone who depends entirely on telecoms for their survival had better be operating out of two or more physically diverse locations. UPS and gennies are just going to keep your servers running (although I understand that even this failed, showing a remarkably cavalier attitude on behalf of Technorati, Craiglists et. al. toward ensuring that your BCP works) - to keep your telecoms up you need to be on different telecoms networks AND different power networks, just altogether in a different city, state or country.
I wonder if I can get a hugely overpaid consulting job advising them on the basics of continuity planning?
[ link to this | view in thread ]
John W. Peace II CEcD, 5 Aug 2007 @ 8:24am

Safest Place on Earth for Data Centers!!!
Natural disasters, power outages, terrorism, not a problem if you locate your mission critical operations in the Safest Place on Earth!

Take a look at:

http://www.safestplaceonearth.net/
[ link to this | view in thread ]
Paul Rice, former Security Architect Engineer of t, 16 Sep 2007 @ 7:02pm

Re: Data Center Outage and Redundant Power
An appropriate sized REGULATOR(s) before the panels and even UPS's are meant to absorb and take out all power spikes from city power. Sometimes, we forget to add them to our systems. Their only purpose is to prevent power spikes from continuing on to your infrastructure...
[ link to this | view in thread ]
John Peace, 3 Jul 2009 @ 3:54pm

Safest Place on Earth for Data Centers!
Looking for the safest place to locate Data Centers?

http://www.safestplaceonearth.net/

Locate @ the power source with little risk and cheaper electrical rates!
[ link to this | view in thread ]