Some of you have probably noticed how quiet it’s got around here over the last week or two. I’m running up to a deadline at work and am pretty much thinking about nothing else right now. The weblog’s gone a bit to pot in the meantime. Apologies all around and hopefully I’ll have some more time on the weekend to catch up on everything that’s been going on.
Today I was at work when Barbelith went down. MySQL errors everywhere, the community in uproar, IMs and e-mails. And it wasn’t like I didn’t have enough to do. So I explore in more depth. First step, see what’s actually happening on the server – so I launch Terminal, ssh in to the Barbelith Superserver over at Pair, find the directory with my logs in and type in tail -f access-log. Immediately, I see each request coming into the server in roughly real-time, scrolling down the page like I’m looking at The Matrix. Unix is not my strong-point, so thanks to Simon for that little trick. It’s moving too fast for me visually get a grasp on what’s going on, but I start seeing some recurrent patterns after a minute or so – HTTrack, which I do a quick search for and turns out to be a piece of software that you run on your computer to download complete versions of someone’s website. Given that Barbelith contains nearly six hundred thousand posts across twenty five thousand threads (each paginating ever forty or so posts), this is not a small job. And given that the software is dragging down a bunch of pages each and every second, it’s not really a surprise that the MySQL server was having some trouble.
So I banned the user’s IP for a bit by adding a couple of lines to my .htaccess file and waited for the site to start working again. But no luck. Exploring the database through the PHPMyAdmin interface that Cal set up for me, I note that all the activity has resulted in one table in the database getting corrupted. So I dig around online a little longer, and work out how to login to MySQL directly through the Terminal and run a repair table command and hope for the best. It all seems to work. Everything’s back to normal. Cheers all around. I’m very proud of myself.
Except then half an hour later the site is down again. This time it’s so bad that people can’t even connect to my server at all. Every site that I run off the server is completely inaccessible to the outside world. plasticbag.org and Barbelith stop working obviously, but also other little-known ventures like Everything in Moderation and bought-for-fun-after-seeing-a-Penny Arcade strip-and-maybe-taking-the-joke-a-little-to-excess Cockthirsty.com are out of action. I can’t even ssh in to my server any more. I can’t send urgent support e-mails to my hosts, or receive replies to them. I am, to all intents and purposes, dead in the water.
I ring them up – half a world away – to find out what’s going on. They’re initially mystified – MySQL is running so hot it’s a wonder that the rack-mounts aren’t melting. When they try and login, the server basically falls over completely. A forced restart, and I hold my breath a little. When it comes back, they dig into the logs and it becomes immediately obvious to them what’s going on. Hundreds – thousands – of requests every minute for a file called mt-comments.cgi – the part of Movable Type that deals with incoming comments to my weblog. My entire site has been quite directly, and clearly spammed to death.
So I’ve had to make a short-term choice while I explore my options in more depth, between a site with no comments and no site at all – and I’m afraid the answer is no more comments, at least for the time being. I’d been thinking of looking into Akismet, but there’s simply no point. That still means that MySQL is going to be dealing with all this crap-peddling evil purpetrated by money-grubbing parasites, and that means regular meltdowns. I’ve come to wonder whether the problems I’ve had with MySQL errors on Barbelith over the last couple of years have been more to do with comment spam than anything else, and – while I want to make it clear that in no way do I blame Six Apart or Movable Type or anything and while I’m sure there’s a way out of this situation – it has started to feel like having the mt-comments.cgi script sitting on my server is like having a bullseye painted on my chest. In the meantime, any advice people have on how to deal with this kind of activity would be very much appreciated indeed. Would moving to Typekey authentication only help? Should I be looking into throttling on the server? Can anyone help? The e-mail address (I’m afraid) is tom at the name of this site – or you can write your own post and link to this one and I’ll find you via Technorati.
The second day of XTech and the first day of the conference proper (yesterday being tutorials) starts with a keynote from Paul Graham (see his Wikipedia entry) talking about whether or not the success of Silicon Valley might be replicated elsewhere – more specifically How American are Startups?. Suw Charman’s done a pretty solid near-perfect transcript of his talk and Graham’s subsequently written up the piece in two parts (How to be Silicon Valley and Why Start-ups Condense in America) but fundamentally his argument breaks down (to me at least) to these points:
Anyway, it would probably be fair to say that the reaction to the session has been mixed, although it’s more to do with his thoughts about American success and European problems than the points above (I suspect). Here’s one particularly astringent comment from Jeremy Keith:
It’s essentially a Thatcherite screed about why businesses should be able to get away with doing anything they want and treat employees like slaves … He also thinks that it won’t be long before Europe is all speaking one language namely, his … What. A. Wanker.
I think Jeremy’s gone a bit over the top, but I can completely understand why he reacted the way that he did. Paul’s piece felt extraordinarily American, in a semi-utopian libertarian free-market kind of way, and I have to admit it felt alien and strange and fairly abrasive. But there was also some pretty solid insights and a hell of a well-presented argument. By the end of the piece I was wondering, was this a political screed supported by good argument? Or was this a position that had been reached through experience that just happened to coincide with a particular political ideology.
I’ve spent about an hour thinking around this now, and have come to the conclusion that it’s probably the second of the two – an argument borne from experience but still an argument that needs to be heavily contextualised and derives from the particular environment that he operated within. The approach that Californians take to governance clearly works pretty well (for some interpretations of good), but that doesn’t make it a natural fact of the universe. It could be much more contingent than we tend to recognise.
Let me put it this way – one point that Graham made was about the role of government – basically intimating that regulation and government intervention was almost uniformly and universally a bad thing. But no government and no enlightened citizenry will be prepared to mutely accept the facts of their destiny on the basis of their weather (one of the aspects that Graham spells out as making a place attractive to the right kind of people). And all governments will try hard to make their environments more conducive to certain kinds of activity, including the US government. For example, one concern that I note that Paul Graham did not mention at all during his piece were simple start-up costs. He talked about companies started in garages, but probably didn’t realise that even garage space is pretty limited for large groups of people in metropolitan areas in Europe. This is a factor that probably has no effect in California outside the big cities, but is of massive importance across Europe. Property prices and costs are so extreme in parts of Britain that it’s almost immediately impractical for two or three people to try and start a little company. This is not in defiance of Graham’s talk, it’s simply ignored by it. And he ignores cultural differences, food costs and increased risks that mean that people are simply less comfortable making these kinds of decisions. If you want to create a culture where this kind of thing is possible, then these things need to be fixed. And that means work that needs to be funded and that means government one way or another.
And there’s another aspect which I found worrying – clearly it’s good for business to be able to hire and fire as you choose. And it’s also clearly not problematic for technology workers to lose their jobs if they’re working around Silicon Valley – it’s not like there’s a shortage of other places to work for. But the laws don’t only apply to the people with lots of job mobility and freedom – they also apply to people at the bottom of the food chain. Many European countries have decided to try and protect those people at the cost of some of their business flexibility. I’m not saying one option is more right or more wrong – I’m actually quite keen on the free market, and my time at the BBC rammed home to me some of the problems of working in an organisation that’s unionised to the point that it’s unable to fire people or restructure itself effectively in response to changing circumstances. But I think it’s important to at least recognise that the things that may make a Silicon Valley possible might also be partially founded on immigrant labour, crippled unions and a lack of support for people at the bottom of the pile. What’s good for Startups may not be good for all, and occasionally I got the impression that much of Graham’s stuff was describing the environmental factors that make Start-ups work as a uniform and perfect good in the world. I don’t buy that so much.
Having said all that, I have to be honest, I pretty much agree with all of his major points, and his thoughts on the right places for start-up activity got me thinking about places in the UK that would be good seed beds for an ecosystem of small and larger companies to operate together effectively. I’m not convinced that London is a good place for this kind of stuff at all, even though unfortunately all the money and all the business ends up there.
I’ve done some exploring around and found information on the top ten Computer Science departments in the UK and they are: Imperial, York, Oxford, UCL, King’s College London, Edinburgh, St Andrews, Cambridge, Glasgow and Bristol. Applying Graham’s criteria to those places, I’m afraid Scotland is probably mostly out as an ideal transportation hub and a home for rich technologists. I suspect London is simply too expensive and cripplingly scary for anyone other than people wanting to work for big businesses and media companies (it’s the New York of the UK, not the San Francisco). Which leaves Oxford, Cambridge, York and Bristol. I don’t know much about York, but my sense is that it’s a bit too far off the transport grid to be ideal, even though it has a large student population and is a relaxed and outdoorsy place. Oxford and Cambridge are obvious candidates, but I’m actually most interested in Bristol (coincidentally where I went to University) which is an hour and a half from Paddington and the Heathrow Express, is surrounded by beautiful landscape and opportunities to explore and has a 20,000-strong student population across the University of Bristol and the University of the West of England. It’s also not unmanageably expensive.
The other place that interests me a lot is Brighton. I don’t know whether or not it has an enormous technology contingent, but I’m hearing a lot about start-ups based out of there. It’s an hour from Central London, is extremely cosmopolitan and seems to have a lot of the characteristics that a start-up culture would require. I’d be really interested to get people’s thoughts about where and how we could get a more technology-focused start-up scene going in the UK. So feel free to leave a comment.
Addendum: For those interested, he also summarised the advantages and disadvantages of the US in the start-up space, and I’ve cut back the advantages to these helpful headlines which should give you the gist of his argument:
One small aside that I should probably bring to people’s attention (and get some feedback on) is the new Yahoo! homepage (Yahoo! UK version) which has launched in beta. Richard MacManus of the awesome Read/Write web has a good review of the changes to the page. There are a few little things I’m unsure about, but they’re basically trivial compared to the many significant improvements that have clearly been made. I’m interested to know what everyone else thinks:
I just thought I should mention that because of the ever-escalating amount of comment spam I’m getting on plasticbag.org nowadays, I’ve decided to make it the default behaviour to not publish them immediately on the site. If you’re a Typekey user, then you get a free pass – your comments will appear on the site immediately – but otherwise I’m afraid every single comment is going to get individually checked by me before it goes up. I’d really rather this wasn’t the case for a whole bunch of reasons (including my possible increased liability to legal threat) but I’m not comfortable any more having mother/daughter incest, bestiality and teen slut links on my site as a matter of course. So I’m cutting them off. It’ll mean a bit more work for me, and a less dynamic and exciting site, but I can’t really think of another solution that doesn’t end up being an ever-escalating arms race. You can read my earlier thoughts around this territory in this old post of mine: Trackback is Dead – Are Comments Dead Too?
I was looking at a post about the relative proportions of various planets today and it reminded me of a thought I’d had a couple of years ago and hadn’t ever explored in depth. It was based on a stupid theory of mine that I’d like you guys to disprove or correct or support. To start off with, look at the following picture from the above link:
So the most notable thing about this picture from my perspective (other than the size differential) is the difference in colour. More specifically, not that the colours are different, but that non-Earth planets are much more monotone. So my theory goes like this – we evolved on Earth, it’s important for us to be able to distinguish between certain types of substances and so our eyes are set up to better distinguish between the specific wavelengths of light that things on our planet reflect. Not a great moment of insight, I’m sure you’ll agree. Pushing it further though – this suggests that if we had evolved on Pluto, Jupiter or Venus, we’d perceive different wavelengths in more depth and with more variety. In fact, potentially if we’d been born on Mars, it would look to our eyes like a vibrant and colourful place, while Earth could look comparatively drab.
This could all be total balls for all I know. But what I wanted specifically to do was to get an image of Venus or Mars or whatever and kind of stretch the spectrum of colours across our visual spectrum, with the assumption that you’d get more Earth-like shades and a different sense of the landscape, even if (to the natives of this world) blue would no longer correspond to water. I’m assuming that someone’s done something like this before, but I can’t find anything good online about it, or a way to do such a thing in Photoshop. Anyone got any thoughts?