Blame Murphy's Law, Excessive Hubris For Blackouts In SF
from the or-blame-the-PR-people,-your-choice dept
Lots of San Francisco-based web sites (Craigslist, Six Apart, Yelp, and Technorati, among others) have been experiencing some problems today, after power outages in the city took down 365 Main, a major hosting facility there. While the power company says it doesn't yet know what's caused the outages, we have a pretty good idea: a 365 Main press release that went out this morning, bragging about the two years of continuous uptime one of its customers has had since moving to the data center. And when these guys invoke Murphy's Law, they don't do it by half, either. The release also brags about the center's "unique billing system in which 365 Main only charges customers for the exact amount of power that is used" -- so presumably today will be free. But what really sealed their fate was this paragraph:"To ensure uptime for key tenants such as RedEnvelope, 365 Main provides modern power and cooling infrastructure. The company's San Francisco facility includes two complete back-up systems for electrical power to protect against a power loss. In the unlikely event of a cut to a primary power feed, the state-of-the-art electrical system instantly switches to live back-up generators, avoiding costly downtime for tenants and keeping the data center continuously running."Good to see those backup systems are working!
![Hide This Hide this](https://ii.techdirt.com/s/t/i/topclose.png)
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: data centers
Companies: 365 main, redenvelope
Reader Comments
Subscribe: RSS
View by: Time | Thread
Drunk and disorderly... ?
http://valleywag.com/tech/breakdowns/a-drunk-employee-kills-all-of-the-websites-you-care-abo ut-282021.php
In that article it says that a "shitfaced drunk" employee did the damage to 40+ racks of equipment.
I dunno, but I think I'll take that article with a grain of salt for the time being.
[ link to this | view in chronology ]
Rats
Russia's Baikonur Kosmodrome also had a serious problem with rats chewing cables at the time, and they dealt with the problem by keeping cats.
[ link to this | view in chronology ]
Murphy's Law - a correction
Quoted from The Signature of God by John Dalmas
[ link to this | view in chronology ]
Could be related. =)
[ link to this | view in chronology ]
Bragging like that is almost as stupid as...
Moral: God doesn't like braggers.
BMR777
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Re: Excessive Hubris
[ link to this | view in chronology ]
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Duh...
[ link to this | view in chronology ]
That wasn't Murphy's Law...
[ link to this | view in chronology ]
Excessive BS is more like it
[ link to this | view in chronology ]
well duh!
(emphasis mine)
The power goes out and they trust on electrical systems to switch to back up generators?
No wonder it went wrong. :-)
[ link to this | view in chronology ]
Re: well duh!
[ link to this | view in chronology ]
Re: well duh!
Something clearly went wrong however. I had a client once who had a similar failure, the batteries switched over to the generator which started just fine and all was OK, but a few minutes later the generator stopped
Cause: whichever plank had installed the generator had wired up its electric diesel pump to the mains only, instead of to the generators own electrical output.... it drank itself dry ;0)
[ link to this | view in chronology ]
Murphy was an optimist
[ link to this | view in chronology ]
Askign for it.
[ link to this | view in chronology ]
No mirroring?
[ link to this | view in chronology ]
[ link to this | view in chronology ]
backups?
[ link to this | view in chronology ]
[ link to this | view in chronology ]
you think you're so smart...
No batteries, no redundant grid, no amount of testing will guarantee a no-impact failover. I've seen multiple outages at multiple sites over 20 years. The answer is IT DEPENDS. With hugs power feeds it gets complicated fast.
It is incredibly ironic that the PR folks put out a release like that the same day... but really, most hosting sites say the same kinds of things. You can take cheap shots if you want, doesn't mean you know anything. I often enjoy the hubris of the media!
[ link to this | view in chronology ]
reply
[ link to this | view in chronology ]