Is It Better Or Worse That GoDaddy's Massive Downtime Wasn't The Result Of A DDoS Attack?
from the i'd-say-worse... dept
As you may have heard, yesterday web host/domain registrar GoDaddy had some serious downtime, taking tons of sites down. Much of the news coverage focused on a single hacker who tried to take credit for bringing them down. However, now that GoDaddy is back, it insists the problem was entirely internal and not the result of a hack. The company was quite explicit on that point:The service outage was not caused by external influences. It was not a "hack" and it was not a denial of service attack (DDoS). We have determined the service outage was due to a series of internal network events that corrupted router data tables. Once the issues were identified, we took corrective actions to restore services for our customers and GoDaddy.com. We have implemented measures to prevent this from occurring again.While my first reaction to all of this was to wonder who would still use GoDaddy, my second question is to wonder whether GoDaddy looks better or worse if it was its own fault that the service went down so broadly. Mistakes happen, but a company like GoDaddy survives on its ability not to make mistakes at that level. I guess, in the end, it's just yet another reminder of why people might want to look for alternatives.
At no time was any customer data at risk or were any of our systems compromised.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: ddos, downtime, hacking, internal problems
Companies: godaddy
Reader Comments
Subscribe: RSS
View by: Time | Thread
One Here
[ link to this | view in chronology ]
Re: One Here
One tried to scam me this morning.
[ link to this | view in chronology ]
If the domain name was OWNED by GoDaddy, purchased and hosted elsewhere, the site still went down and displayed a GoDaddy error message.
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re: Re:
[ link to this | view in chronology ]
Typical (lame) network guys not taking responsibility for causing a problem (or creating the situation for the problem to occur), and then swooping in as the hero.
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Re:
They don't seem to the types that would do haxxorz that don't have a political protest motivation.
[ link to this | view in chronology ]
Forgive me for I have sinned...
BTW, Is Netflix down now? I can't resolve it... Sadness :(
[ link to this | view in chronology ]
Re: Forgive me for I have sinned...
[ link to this | view in chronology ]
If you admit that a hack took place, you now have to investigate whether any information was also taken during the hack, which is costly and a lengthy process. This, in fact, will continue bad press for a long time. By giving a we screwed up routing excuse, they can then dismiss accountability until a investigation has been conducted. This gives them time to research to give a valid RFO, and delay any bad press in the meantime. It pretty much makes sense from both a legal standpoint, and a PR standpoint imho.
Being a network guy, I understand that these problems can arise, but the length of the outage tends to cast doubts on what exactly was going on. If this was a major network overhaul gone completely bad, I can understand. It can take quite some time to rollback a major overhaul across multiple sites, but not enough information was given to sway me one way or another.
[ link to this | view in chronology ]
Choices...
vs
We are just inept at running our systems.
There is no real good way out of this, the upside is they can claim they weren't hacked so they look like the good guys... except they screwed their customers long and hard all on their own then...
Another reason to not use GoDaddy.
[ link to this | view in chronology ]
Re: Choices...
[ link to this | view in chronology ]
Re: Re: Choices...
We have a lone hacker taking credit vs GoDaddy, I do not like my odds with either side.
It reminds me of the Sony hacks, where Sony kept claiming they were never breached. They were doing stuff on their own, and only after the evidence reached stupid proportions did they finally admit they were hacked. Then they "found" a single text file with a well known motto to shift the blame to a group that never took credit for the hack. Then it came out how some of the hacks happened was because they skipped over even basic security concepts, and ignored people pointing out glaring flaws in the system.
Hacks in general happen, we all know this, but it seems to admit to them is worse than being hacked.
The only secure computer is one not connected to the net and not allowed human interaction.
If they came out and admitted we got hacked, and we have fixed the problem most people wouldn't think any less of them. It would be a good reminder to stay up to date with your security, and never just assume your secure.
Instead we get PR speak trying to save the company's reputation with doublespeak and hedging around the issues. Companies want to treat getting hacked as a corporate secret that no one can ever know about, and all that does it make us all less secure.
[ link to this | view in chronology ]
Re: Re: Re: Choices...
[ link to this | view in chronology ]
Re: Re: Re: Re: Choices...
And then someone over at ars mention in the comments of the story that a DDOS started the problem, then it all went sideways, and they are playing buzzword bingo to make it look like a little booboo from inside rather than the outside.
I take it with a grain a salt, because on the internet no one knows your a dog... but lots of people are suggesting this doesn't add up as explained unless they were not following even basic rules. Sort of like that cert company who claimed long and hard they weren't hacked, even as people were pulling up bogus certs generated from their systems.
[ link to this | view in chronology ]
[ link to this | view in chronology ]
[ link to this | view in chronology ]
GoDaddy is not immune to the problems of having a sizable network. Quite simply, it happens.
I know you want to kick them because you hate them and all, but come on Mike, can't you be reasonable and accept that shit happens sometimes?
[ link to this | view in chronology ]
Re:
Mike did not kick them, Mike pointed out a news story involving *gasp* tech and GoDaddy's statement that says it was not a hack.
Mike wondered how a mistake like this can happen in a company that is so big, to be kicking them he would have written a piece being much more critical of them and pointing out the massive amount of fail and underhanded tactics they employee. This was merely commentary on a single event and the "response" by GoDaddy.
[ link to this | view in chronology ]
Re: Re:
Probably because we expect a lot in spite of being human. It happens, heh.
[ link to this | view in chronology ]
Re: Re:
BGP errors are always cascading due to the nature of the protocol, so this is rather easy to show 5 times or more that this has happened on a global scale.
1. BGP leak on Dery Telecom Inc in Ontario
2. BGP leak on Telstra in Australia
3. BGP leak on SK Telecom in California
4. BGP leak on China Telecom in China
5. BGP leak on Evolva Telecom in Romania
[ link to this | view in chronology ]
Re:
Plan B? - What's that?
"Quite simply, it happens"
Interesting that it happens so little elsewhere, maybe other networks are designed for fault tolerance. Naaa - that can't be it.
"I know you want to kick them because you hate them and all"
Awww, isn't that special.
[ link to this | view in chronology ]
Re:
GoDaddy is not immune to the problems of having a sizable network. Quite simply, it happens.
I know you want to kick them because you hate them and all, but come on Mike, can't you be reasonable and accept that shit happens sometimes?
Please stop disrupting the Techdirt narrative. Any company that supported SOPA is evil, incompetent, should be boycotted and their services suck. Masnick is in a blind rage because after all of his self-congratulations about how he somehow had a stake in defeating SOPA, the end result was a big zero was all of the SOPA players side stepping and using private agreements to accomplish much the same thing.
[ link to this | view in chronology ]
Re: Re:
Would publicising, legalising, and setting in stone private agreements to leverage further harsher and narrow laws have been an improvement?
[ link to this | view in chronology ]
Re: Re:
[ link to this | view in chronology ]
Re:
PS, I hate GoDaddy too!
[ link to this | view in chronology ]
Re:
"GoDaddy is not immune to the problems of having a sizable network."
That's the thing, they shouldn't have just one network. When you're making something at this scale one of the design considerations is fault tolerance. The way to achieve that is by designing the system to be separate parts that do not rely on each other. No single change should EVER be able to affect all of their DNS infrastructure all at once like this, it should always require changing it in at least 2 places (for a global DNS infrastructure I'd require at least 6: one per continent, and we'll risk all the customers in Antarctica by lumping them in with someone else)
[ link to this | view in chronology ]
Re: Re:
[ link to this | view in chronology ]
Re: Re: Re:
In answer to Brad C, I can only say: Royal Bank of Scotland (RBS).
[ link to this | view in chronology ]
Re: Re: Re: Re:
[ link to this | view in chronology ]
[ link to this | view in chronology ]
[ link to this | view in chronology ]
[ link to this | view in chronology ]
[ link to this | view in chronology ]
MSM Spin
Josh Elliot mentioned the story while doing the news yesterday morning. I can't find a link.
Basically he said that GoDaddy had been taken down reportedly by the internet group Anonymous because they were angry about them supporting laws that could stop them from pirating movies and stuff.
Woohoo Disney! Way to spin a news report, even when you don't even have ANY of the facts!
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Offical RFO from GoDaddy
Basically, eBGP redistribution into iGP crashed the routers due to a bug in the OS. Due to customer traffic and routing traffic continually overloading the routers, they had to limit the traffic and delay recovery over a period of time causing the extended outage.
It's a pretty detailed RFO for end-users, and I give them credit for supplying such details.
[ link to this | view in chronology ]
downtime
[ link to this | view in chronology ]
[ link to this | view in chronology ]