What has been killing my server?


Today I was at work when Barbelith went down. MySQL errors everywhere, the community in uproar, IMs and e-mails. And it wasn’t like I didn’t have enough to do. So I explore in more depth. First step, see what’s actually happening on the server – so I launch Terminal, ssh in to the Barbelith Superserver over at Pair, find the directory with my logs in and type in tail -f access-log. Immediately, I see each request coming into the server in roughly real-time, scrolling down the page like I’m looking at The Matrix. Unix is not my strong-point, so thanks to Simon for that little trick. It’s moving too fast for me visually get a grasp on what’s going on, but I start seeing some recurrent patterns after a minute or so – HTTrack, which I do a quick search for and turns out to be a piece of software that you run on your computer to download complete versions of someone’s website. Given that Barbelith contains nearly six hundred thousand posts across twenty five thousand threads (each paginating ever forty or so posts), this is not a small job. And given that the software is dragging down a bunch of pages each and every second, it’s not really a surprise that the MySQL server was having some trouble.

So I banned the user’s IP for a bit by adding a couple of lines to my .htaccess file and waited for the site to start working again. But no luck. Exploring the database through the PHPMyAdmin interface that Cal set up for me, I note that all the activity has resulted in one table in the database getting corrupted. So I dig around online a little longer, and work out how to login to MySQL directly through the Terminal and run a repair table command and hope for the best. It all seems to work. Everything’s back to normal. Cheers all around. I’m very proud of myself.

Except then half an hour later the site is down again. This time it’s so bad that people can’t even connect to my server at all. Every site that I run off the server is completely inaccessible to the outside world. plasticbag.org and Barbelith stop working obviously, but also other little-known ventures like Everything in Moderation and bought-for-fun-after-seeing-a-Penny Arcade strip-and-maybe-taking-the-joke-a-little-to-excess Cockthirsty.com are out of action. I can’t even ssh in to my server any more. I can’t send urgent support e-mails to my hosts, or receive replies to them. I am, to all intents and purposes, dead in the water.

I ring them up – half a world away – to find out what’s going on. They’re initially mystified – MySQL is running so hot it’s a wonder that the rack-mounts aren’t melting. When they try and login, the server basically falls over completely. A forced restart, and I hold my breath a little. When it comes back, they dig into the logs and it becomes immediately obvious to them what’s going on. Hundreds – thousands – of requests every minute for a file called mt-comments.cgi – the part of Movable Type that deals with incoming comments to my weblog. My entire site has been quite directly, and clearly spammed to death.

So I’ve had to make a short-term choice while I explore my options in more depth, between a site with no comments and no site at all – and I’m afraid the answer is no more comments, at least for the time being. I’d been thinking of looking into Akismet, but there’s simply no point. That still means that MySQL is going to be dealing with all this crap-peddling evil purpetrated by money-grubbing parasites, and that means regular meltdowns. I’ve come to wonder whether the problems I’ve had with MySQL errors on Barbelith over the last couple of years have been more to do with comment spam than anything else, and – while I want to make it clear that in no way do I blame Six Apart or Movable Type or anything and while I’m sure there’s a way out of this situation – it has started to feel like having the mt-comments.cgi script sitting on my server is like having a bullseye painted on my chest. In the meantime, any advice people have on how to deal with this kind of activity would be very much appreciated indeed. Would moving to Typekey authentication only help? Should I be looking into throttling on the server? Can anyone help? The e-mail address (I’m afraid) is tom at the name of this site – or you can write your own post and link to this one and I’ll find you via Technorati.