…. a multi-hour process halting on your biggest database table on your master database at 3am that throws all your database slaves out of whack. Perseverence is … rebuilding that huge table and index files from a backup (you always make backups before you do anything big right?) Happiness is … putting everything back together perfectly, getting the site back online and returning to one’s much more prosasic tasks (like launching IM ;)

Our sincere apologies to all our beloved members and fans who we let down last night.

And a note to anyone starting small with big plans. Once your database tables start getting above 10Gb your options for working with them become significantly reduced. Keep a keen eye open for the tables that grow much faster than others as they’ll no longer play nice like the little ones do.

Our tail is between our legs, but the site has been working fine now and it is definitely better than ever.