And So It Goes…

Well, I typed the word “reboot” into a server today, not expecting that I was going to cause myself and my team a combined total of 27 hours of work (between 4 people.)

Here’s what happened. We had about 40 websites, including simpli.biz, hosted on a RaQ 550 server that I’ve been wanting to get rid of for a long time now. Mooneer and I had been slowly moving the sites off of the RaQ 550 for the past week, planning to do a full-scale migration starting today and going through the end of next week. As of today, there were 26 websites still hosted there.

Well, the darn SMTP server kept crashing, and every day I was going without email and having to log in and restart the stupid SMTP server (and then get caught up with the email I had missed as it flowed in for the next hour.) Finally, today around 1PM, it crashed again and I decided I’d had enough. I rebooted the server through SSH.

It didn’t come back up.

I knew I was in trouble about an hour later. I’d rebooted it through SSH to begin with. After 15 minutes, I ran a hard reboot. About 20 minutes after that, I put in a ticket with AboveNet and had them unplug it and replug it. At 1:45PM, I headed to the datacenter to confirm that the server was, in fact, going through fsck. I figured I’d let it fsck while I ate lunch, and came back to find that it had stopped fsck’ing and was just sitting there like a total lump.

I rebooted. Nothing.

At that point (around 3PM), Mooneer came online for his shift and I realized we were going to have to take drastic action. I told him to grab our backup (made 14 hours previous, so it was pretty fresh) and we’d get to work restoring 26 websites. I then called Brandon and had him come into the datacenter to work the hardware end while I went home and helped Mooneer restore websites and email to meowcat (our new sexy dual Xeon server.) simpli.biz was moved to my ULTRA-POWERFUL AWESOME Celeron 800 server 😀 that I picked up for next to nothing (which also hosts this website and our helpdesk and ticket system.)

Well, the day wound down with Mooneer and I still restoring websites and email around 9PM PST (8 hours after this whole thing had started.) Just after 9, we finally got my email restored. I’d helped restore a couple websites, including simpli.biz and onthehouse.com (our two most hairy ones), and was working with customers to get them control panel access.

Why did it take almost 9 hours to restore 26 websites, you ask? Well, unfortunately, there is no easy way to convert sites from a RaQ 550 to a DirectAdmin-based server. We created all of the sites manually and slowly put back the public_html folders and the email inboxes for every person on the server, which turned out to be quite time-consuming. The lack of a migration utility meant we also had to change everyone’s passwords, so if you’re reading this and thinking “I haven’t been able to log into my email all day!” (George, that means you!), that’s why. (Email support@simpli.biz from an alternate email account, or call us, and we’ll get you your new password.)

So that’s how I spent Black Friday. And here I was really looking forward to getting cozy with Photoshop CS, DWMX2004, and a copy of Designing with Web Standards. Ah, well. There is no rest for the wicked [servers].