Hacked Recap

from the well,-that-was-fun dept

As mentioned over the weekend, we were briefly hacked on Saturday evening. We've put in a bit of time to figure out what happened, clean up the mess and correct the problems (and harden some other defenses as well). The short story is that we left open a big hole that we shouldn't have left open. Yay. We had certainly locked down most of the obvious holes, and people try to hack us on a semi-regular basis, with little success. But, if someone's persistent enough, they'll find a way. In this case, though, we made it a hell of a lot easier than we should have. This particular hacker tried hitting a whole bunch of different routes early Saturday morning, most of which got rejected (some people noticed his attempt to do a SQL injection via the comments -- that failed). However, he went on to try SQL injections just about everywhere and eventually found one where we hadn't properly escaped things, and bam, that's all it takes. As you probably know, this site has been around since 1998, and while we've dumped/updated most of the old code, and most of the new code is properly secured, there were still a little pieces left over from the ancient code -- and that's where the big vulnerabilities were. That's not an excuse. We should have caught it earlier (in fact, we actually had been testing some code to replace some of the vulnerabilities, but hadn't deployed it yet -- but, we now realize it wouldn't have blocked all the problems). But, it is what happened.

From there, the hacker got into part of the blog admin (don't want to get into too many details of how the blog backend works, but it actually involves two separate admins -- which are separate from other stuff we do). Then, he basically had pretty good access to doing some stuff (though not everything) on the blog. He poked around a bit, deleted a bunch of comments, deleted a whole ton of old story submissions (most of which were junk anyway -- so thanks!) and then replaced a few stories on the front page with his fancy "hacked!" claims.

After that, the story is pretty straightforward. Once we realized what happened, we put the old stories back in place and made sure to quickly toss up some more secure walls to keep him out of the admin. We also shut down comments and submissions for a while, even though we were pretty damn sure the vulnerability wasn't there (it wasn't), but we wanted to make sure. Then a few of us spent some time digging around to understand just what the guy did so we could retrace his steps and make sure we killed off the basic vulnerabilities. Considering that he tried to hit us from a bunch of different angles, this took a bit longer than expected. But, once we figured out the basics, it was just a matter of tracking down the actual holes in the code. It was a little frustrating, since we really thought we'd blocked out SQL injections -- but in the end, it turns out we didn't do it absolutely everywhere. Anyway, there's a fair amount of code to go through, so we've been going over it with a fine-tooth comb, and checking it twice, then locking it down again.

Finally, we've been restoring the lost comments (we're doing that right now, so they might not all be back yet), of which we believe we didn't lose any (there's a small chance that a very very small number of comments were lost). Restoring the lost submissions is a bit much at this point (as I said, most were junk anyway), so if you submitted stories late Friday or Saturday, and really think we should see them, perhaps submit them again.

On the whole, there's not that much to say, other than check your code carefully, folks. If there's a hole somewhere, eventually someone's gonna find it. Luckily, this guy didn't do much damage -- just a bit of vandalism -- and he kept a few of us from enjoying what had otherwise been quite nice weekends with our friends and families. But he got us to go over our code pretty carefully (and mentally kick ourselves a few times), and get in touch with our inner CSI detectives to track down exactly what happened.

Update: Well, that was just great. Less than half an hour after posting this, our network provider went down for nearly two hours, despite supposedly having all sorts of redundancies. It had nothing whatsoever to do with the hack, but was a bigger issue for the provider. However, it did slow down us restoring the comments, meaning that comments need to remain off for probably another few hours. This has really been a fun weekend.

Update 2: Comments are back. We did end up losing a few comments, mostly those right before the hack. Really sorry about that. If you said something really important and it's missing... say it again, please.

Filed Under: hacked, sql injection
Companies: floor64, techdirt