Court To Twitter: No Time For Appeal, Hand Over Info Or You're In Contempt

Is It Better Or Worse That GoDaddy's Massive Downtime Wasn't The Result Of A DDoS Attack?

from the i'd-say-worse... dept

Tue, Sep 11th 2012 4:38pm — Mike Masnick

As you may have heard, yesterday web host/domain registrar GoDaddy had some serious downtime, taking tons of sites down. Much of the news coverage focused on a single hacker who tried to take credit for bringing them down. However, now that GoDaddy is back, it insists the problem was entirely internal and not the result of a hack. The company was quite explicit on that point:

The service outage was not caused by external influences. It was not a "hack" and it was not a denial of service attack (DDoS). We have determined the service outage was due to a series of internal network events that corrupted router data tables. Once the issues were identified, we took corrective actions to restore services for our customers and GoDaddy.com. We have implemented measures to prevent this from occurring again.

At no time was any customer data at risk or were any of our systems compromised.

While my first reaction to all of this was to wonder who would still use GoDaddy, my second question is to wonder whether GoDaddy looks better or worse if it was its own fault that the service went down so broadly. Mistakes happen, but a company like GoDaddy survives on its ability not to make mistakes at that level. I guess, in the end, it's just yet another reminder of why people might want to look for alternatives.

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: ddos, downtime, hacking, internal problems
Companies: godaddy

42 Comments

If you liked this post, you may also be interested in...

Reader Comments

Subscribe: RSS

View by: Time | Thread

A Dan (profile), 11 Sep 2012 @ 4:46pm

One Here
My employer's website uses GoDaddy. It's probably not a big deal, though, since almost nobody visits it.
[ link to this | view in thread ]
Anonymous Coward, 11 Sep 2012 @ 4:48pm

Funny thing, apparently sites that aren't even hosted by GoDaddy also went down.

If the domain name was OWNED by GoDaddy, purchased and hosted elsewhere, the site still went down and displayed a GoDaddy error message.
[ link to this | view in thread ]
Anonymous Coward, 11 Sep 2012 @ 4:50pm

I like how the outage started with "a series of internal network events", and then was fixed when "we took corrective actions".

Typical (lame) network guys not taking responsibility for causing a problem (or creating the situation for the problem to occur), and then swooping in as the hero.
[ link to this | view in thread ]
zos (profile), 11 Sep 2012 @ 4:52pm

After their support for SOPA my compassion meter is barely registering, and my "give a fuck" has apparently left the building. Honestly, the lulzyest part is that the media still thinks random twitter accounts with variations of the word "anonymous" in them, count as a reliable source.
[ link to this | view in thread ]
Anonymous Coward, 11 Sep 2012 @ 4:54pm

Re: One Here
Is your employer a Nigerian scam artist? Where do I apply?

One tried to scam me this morning.
[ link to this | view in thread ]
robofog (profile), 11 Sep 2012 @ 4:55pm

Forgive me for I have sinned...
I am still on GoDaddy... Not because I like them, but because I didn't cancel my webhosting before they auto-renewed my subscription... AND, I am also a tad lazy and don't feel like navigating their vomit style account pages.

BTW, Is Netflix down now? I can't resolve it... Sadness :(
[ link to this | view in thread ]
Anonymous Coward, 11 Sep 2012 @ 5:03pm

This was actually just being discussed on the NANOG mailing list.

If you admit that a hack took place, you now have to investigate whether any information was also taken during the hack, which is costly and a lengthy process. This, in fact, will continue bad press for a long time. By giving a we screwed up routing excuse, they can then dismiss accountability until a investigation has been conducted. This gives them time to research to give a valid RFO, and delay any bad press in the meantime. It pretty much makes sense from both a legal standpoint, and a PR standpoint imho.

Being a network guy, I understand that these problems can arise, but the length of the outage tends to cast doubts on what exactly was going on. If this was a major network overhaul gone completely bad, I can understand. It can take quite some time to rollback a major overhaul across multiple sites, but not enough information was given to sway me one way or another.
[ link to this | view in thread ]
That Anonymous Coward (profile), 11 Sep 2012 @ 5:13pm

Choices...
We were stupid and got hacked.
vs
We are just inept at running our systems.

There is no real good way out of this, the upside is they can claim they weren't hacked so they look like the good guys... except they screwed their customers long and hard all on their own then...

Another reason to not use GoDaddy.
[ link to this | view in thread ]
Chronno S. Trigger (profile), 11 Sep 2012 @ 5:13pm

Re:
If that's the case then I would guess that the DNS tables at GoDaddy were corrupted, or possibly the routing tables to the DNS servers. If you can get to GoDaddy's DNS servers, but they can't resolve a name, that's when you get the GoDaddy error page.
[ link to this | view in thread ]
Chronno S. Trigger (profile), 11 Sep 2012 @ 5:16pm

Re: Choices...
At least they had the balls to admit they screwed up, that's something. They could have said they got hacked, but their security kicked in and shut everything down before data could get stolen. That way they wouldn't have to do any actual investigation and they look like they have advanced security in place.
[ link to this | view in thread ]
TDR, 11 Sep 2012 @ 5:18pm

GoDaddy? How about NoDaddy? One would have to have a double lobotomy to willingly use their service these days.
[ link to this | view in thread ]
Anonymous Coward, 11 Sep 2012 @ 5:45pm

It was not a hack...because GoDaddy said so. And that's the bottom line!
[ link to this | view in thread ]
Anonymous Coward, 11 Sep 2012 @ 6:06pm

Cascade router failures can happen to almost anyone. It only takes a small change, and often that can cause every router in the network to degrade and finally stop passing data. When that happens, it's an incredible hard job to isolate the source and to reset each part of the network.

GoDaddy is not immune to the problems of having a sizable network. Quite simply, it happens.

I know you want to kick them because you hate them and all, but come on Mike, can't you be reasonable and accept that shit happens sometimes?
[ link to this | view in thread ]
That Anonymous Coward (profile), 11 Sep 2012 @ 6:20pm

Re: Re: Choices...
They still don't have to do any actual investigation.
We have a lone hacker taking credit vs GoDaddy, I do not like my odds with either side.

It reminds me of the Sony hacks, where Sony kept claiming they were never breached. They were doing stuff on their own, and only after the evidence reached stupid proportions did they finally admit they were hacked. Then they "found" a single text file with a well known motto to shift the blame to a group that never took credit for the hack. Then it came out how some of the hacks happened was because they skipped over even basic security concepts, and ignored people pointing out glaring flaws in the system.

Hacks in general happen, we all know this, but it seems to admit to them is worse than being hacked.
The only secure computer is one not connected to the net and not allowed human interaction.
If they came out and admitted we got hacked, and we have fixed the problem most people wouldn't think any less of them. It would be a good reminder to stay up to date with your security, and never just assume your secure.
Instead we get PR speak trying to save the company's reputation with doublespeak and hedging around the issues. Companies want to treat getting hacked as a corporate secret that no one can ever know about, and all that does it make us all less secure.
[ link to this | view in thread ]
That Anonymous Coward (profile), 11 Sep 2012 @ 6:28pm

Re:
Please name the names, times, and dates of the last 5 hosts who went dark because of a cascade router failure.

Mike did not kick them, Mike pointed out a news story involving *gasp* tech and GoDaddy's statement that says it was not a hack.

Mike wondered how a mistake like this can happen in a company that is so big, to be kicking them he would have written a piece being much more critical of them and pointing out the massive amount of fail and underhanded tactics they employee. This was merely commentary on a single event and the "response" by GoDaddy.
[ link to this | view in thread ]
abc gum, 11 Sep 2012 @ 6:29pm

Re:
"Cascade router failures can happen to almost anyone"

Plan B? - What's that?

"Quite simply, it happens"

Interesting that it happens so little elsewhere, maybe other networks are designed for fault tolerance. Naaa - that can't be it.

"I know you want to kick them because you hate them and all"

Awww, isn't that special.
[ link to this | view in thread ]
Anonymous Coward, 11 Sep 2012 @ 7:12pm

Re:
Cascade router failures can happen to almost anyone. It only takes a small change, and often that can cause every router in the network to degrade and finally stop passing data. When that happens, it's an incredible hard job to isolate the source and to reset each part of the network.

GoDaddy is not immune to the problems of having a sizable network. Quite simply, it happens.

I know you want to kick them because you hate them and all, but come on Mike, can't you be reasonable and accept that shit happens sometimes?

Please stop disrupting the Techdirt narrative. Any company that supported SOPA is evil, incompetent, should be boycotted and their services suck. Masnick is in a blind rage because after all of his self-congratulations about how he somehow had a stake in defeating SOPA, the end result was a big zero was all of the SOPA players side stepping and using private agreements to accomplish much the same thing.
[ link to this | view in thread ]
Anonymous Coward, 11 Sep 2012 @ 7:30pm

Re:
Yeah, I didn't buy the anon angle at all.

They don't seem to the types that would do haxxorz that don't have a political protest motivation.
[ link to this | view in thread ]
Anonymous Coward, 11 Sep 2012 @ 7:59pm

Re: Re:
You're obsessed with pushing this point on, aren't you?

Would publicising, legalising, and setting in stone private agreements to leverage further harsher and narrow laws have been an improvement?
[ link to this | view in thread ]
David, 11 Sep 2012 @ 8:11pm

Re: Re: Re: Choices...
Is there any possibly better way to approach this, though, other than (arguably) admitting a (possible) screw-up and taking flak still?
[ link to this | view in thread ]
David, 11 Sep 2012 @ 8:13pm

Re: Re:
Mike wondered how a mistake like this can happen in a company that is so big

Probably because we expect a lot in spite of being human. It happens, heh.
[ link to this | view in thread ]
Anonymous Coward, 11 Sep 2012 @ 8:35pm

Re:
I actually agree with this poster. It feels like TechDirt badly wanted to make hay out of this news, and the take of "Which is worse?" feels ham-handed.

PS, I hate GoDaddy too!
[ link to this | view in thread ]
Brad C (profile), 11 Sep 2012 @ 10:41pm

Re: Re:
GoDaddy had to be hosting the DNS for the domain, not just be the registrar. I don't do any web hosting with them, but I have a couple of personal domains that they run the DNS for and they went down. My company's DNS is registered there but we run our own DNS servers and those domains were unaffected.
[ link to this | view in thread ]
Brad C (profile), 11 Sep 2012 @ 10:53pm

Re:
"Cascade router failures can happen to almost anyone."
"GoDaddy is not immune to the problems of having a sizable network."

That's the thing, they shouldn't have just one network. When you're making something at this scale one of the design considerations is fault tolerance. The way to achieve that is by designing the system to be separate parts that do not rely on each other. No single change should EVER be able to affect all of their DNS infrastructure all at once like this, it should always require changing it in at least 2 places (for a global DNS infrastructure I'd require at least 6: one per continent, and we'll risk all the customers in Antarctica by lumping them in with someone else)
[ link to this | view in thread ]
The eejit (profile), 12 Sep 2012 @ 1:08am

Re:
You haven't seen some of the crazy shit that Sysadmins need to do sometimes to get things to work.
[ link to this | view in thread ]
That Anonymous Coward (profile), 12 Sep 2012 @ 1:11am

Re: Re: Re: Re: Choices...
You add more to the mix, like the simple fact they are a massive registrar.

And then someone over at ars mention in the comments of the story that a DDOS started the problem, then it all went sideways, and they are playing buzzword bingo to make it look like a little booboo from inside rather than the outside.

I take it with a grain a salt, because on the internet no one knows your a dog... but lots of people are suggesting this doesn't add up as explained unless they were not following even basic rules. Sort of like that cert company who claimed long and hard they weren't hacked, even as people were pulling up bogus certs generated from their systems.
[ link to this | view in thread ]
Seegras (profile), 12 Sep 2012 @ 3:12am

Re:
Sounds like a classical BGP routing table corruption. Happens all the time, everywhere. Something like three times a year at a typical ISP, but usually they get fixed within minutes to a few hours. But sometimes, every few years, something bigger happens, which takes hours to days to fix, and that's probably what happend there.
[ link to this | view in thread ]
Ninja (profile), 12 Sep 2012 @ 4:20am

I know, it's because SOPA didn't pass. Er...
[ link to this | view in thread ]
Anonymous Coward, 12 Sep 2012 @ 4:32am

Man I've heard some bull, but this one is really good. Huge operation like GoDaddy and no backup routers. Looks like a lot of people are going to get fired today!
[ link to this | view in thread ]
Anonymous Coward, 12 Sep 2012 @ 4:37am

GoDaddy's been hacked for years. There are all kind of back doors that let not reputable people put directories on peoples sites. I know from experience. One day I was editing my site and found a directory with an web page containing an login form for a bank. I didn't put it there and when I told GoDaddy they said it was my fault. I needed to change my password more. Out of there, haven't been hacked since.
[ link to this | view in thread ]
abc gum, 12 Sep 2012 @ 4:40am

Re: Re:
Are you as dumb as you sound?
[ link to this | view in thread ]
abc gum, 12 Sep 2012 @ 4:46am

Re: Re:
Please stop disrupting the troll narrative. Any post that criticized SOPA supporting corporations is evil, wrong and should be berated. SOPA supporting trolls are in a blind rage because after all of their self-congratulations and gloating about how they will eventually rule the world, the end result was a big zero as all of the SOPA players got bitch slapped and decided to side step the law and use private agreements to accomplish much the same thing.
[ link to this | view in thread ]
Wally (profile), 12 Sep 2012 @ 4:49am

Re: Forgive me for I have sinned...
Netflix streaming isn't down :-)
[ link to this | view in thread ]
quawonk, 12 Sep 2012 @ 5:13am

I believe you Godaddy. Just as much as I believe that you are genuinely against SOPA and not just for PR reasons from the boycott that didn't change shit since apparently people are still using Godaddy.
[ link to this | view in thread ]
Anonymous Coward, 12 Sep 2012 @ 6:29am

Re: Re:
Please name the names, times, and dates of the last 5 hosts who went dark because of a cascade router failure.

BGP errors are always cascading due to the nature of the protocol, so this is rather easy to show 5 times or more that this has happened on a global scale.
1. BGP leak on Dery Telecom Inc in Ontario
2. BGP leak on Telstra in Australia
3. BGP leak on SK Telecom in California
4. BGP leak on China Telecom in China
5. BGP leak on Evolva Telecom in Romania
[ link to this | view in thread ]
Niall (profile), 12 Sep 2012 @ 8:01am

Re: Re: Re:
I think you posted under the wrong comment.

In answer to Brad C, I can only say: Royal Bank of Scotland (RBS).
[ link to this | view in thread ]
Gwiz (profile), 12 Sep 2012 @ 9:21am

MSM Spin
I personally really don't care about GoDaddy, but the really intreasting thing in all this I saw was the brief mention on Good Morning America (ABC/Disney).

Josh Elliot mentioned the story while doing the news yesterday morning. I can't find a link.

Basically he said that GoDaddy had been taken down reportedly by the internet group Anonymous because they were angry about them supporting laws that could stop them from pirating movies and stuff.

Woohoo Disney! Way to spin a news report, even when you don't even have ANY of the facts!
[ link to this | view in thread ]
Gwiz (profile), 12 Sep 2012 @ 9:56am

Oof. I might have misspoke. I found the video for GMA's yesterday episode and couldn't find the mention. I think I may have mixed up coverage from my local ABC affiliate with the GMA coverage and the local newscaster was the one who stated it like that. Of course the local affiliate doesn't put the full newscasts online, so I won't ever find it now.
[ link to this | view in thread ]
abc gum, 12 Sep 2012 @ 4:32pm

Re: Re: Re: Re:
Ummm no, just changing the words found in a prior post for the lulz
[ link to this | view in thread ]
Anonymous Coward, 5 Oct 2012 @ 12:48pm

Offical RFO from GoDaddy
GoDaddy

Basically, eBGP redistribution into iGP crashed the routers due to a bug in the OS. Due to customer traffic and routing traffic continually overloading the routers, they had to limit the traffic and delay recovery over a period of time causing the extended outage.

It's a pretty detailed RFO for end-users, and I give them credit for supplying such details.
[ link to this | view in thread ]
Josh Gree, 29 Apr 2013 @ 10:02am

downtime
My website and business lives and dies by being available to my customers. DNS Services dnssvc.com offers a product that's so efficient at making sure my website doesn�t go down with a price to match. I will never leave my website unprotected again.
[ link to this | view in thread ]
Godaddy Reseller Hosting, 27 Jun 2013 @ 10:24pm

Are you as dumb as you sound?
[ link to this | view in thread ]