Well, I typed the word “reboot” into a server today, not expecting that I was going to cause myself and my team a combined total of 27 hours of work (between 4 people.)
Here’s what happened. We had about 40 websites, including simpli.biz, hosted on a RaQ 550 server that I’ve been wanting to get rid of for a long time now. Mooneer and I had been slowly moving the sites off of the RaQ 550 for the past week, planning to do a full-scale migration starting today and going through the end of next week. As of today, there were 26 websites still hosted there.
Well, the darn SMTP server kept crashing, and every day I was going without email and having to log in and restart the stupid SMTP server (and then get caught up with the email I had missed as it flowed in for the next hour.) Finally, today around 1PM, it crashed again and I decided I’d had enough. I rebooted the server through SSH.
It didn’t come back up.
I knew I was in trouble about an hour later. I’d rebooted it through SSH to begin with. After 15 minutes, I ran a hard reboot. About 20 minutes after that, I put in a ticket with AboveNet and had them unplug it and replug it. At 1:45PM, I headed to the datacenter to confirm that the server was, in fact, going through fsck. I figured I’d let it fsck while I ate lunch, and came back to find that it had stopped fsck’ing and was just sitting there like a total lump.
I rebooted. Nothing.
At that point (around 3PM), Mooneer came online for his shift and I realized we were going to have to take drastic action. I told him to grab our backup (made 14 hours previous, so it was pretty fresh) and we’d get to work restoring 26 websites. I then called Brandon and had him come into the datacenter to work the hardware end while I went home and helped Mooneer restore websites and email to meowcat (our new sexy dual Xeon server.) simpli.biz was moved to my ULTRA-POWERFUL AWESOME Celeron 800 server 
that I picked up for next to nothing (which also hosts this website and our helpdesk and ticket system.)
Well, the day wound down with Mooneer and I still restoring websites and email around 9PM PST (8 hours after this whole thing had started.) Just after 9, we finally got my email restored. I’d helped restore a couple websites, including simpli.biz and onthehouse.com (our two most hairy ones), and was working with customers to get them control panel access.
Why did it take almost 9 hours to restore 26 websites, you ask? Well, unfortunately, there is no easy way to convert sites from a RaQ 550 to a DirectAdmin-based server. We created all of the sites manually and slowly put back the public_html folders and the email inboxes for every person on the server, which turned out to be quite time-consuming. The lack of a migration utility meant we also had to change everyone’s passwords, so if you’re reading this and thinking “I haven’t been able to log into my email all day!” (George, that means you!), that’s why. (Email support@simpli.biz from an alternate email account, or call us, and we’ll get you your new password.)
So that’s how I spent Black Friday. And here I was really looking forward to getting cozy with Photoshop CS, DWMX2004, and a copy of Designing with Web Standards. Ah, well. There is no rest for the wicked [servers].
Bookmark in Del.icio.us
Technorati
Digg This!
Review on StumbleUpon
November 27th, 2004 at 2:53 am
DWMX2004? I can only assume that stands for Dreamweaver MX 2004. If you don’t wish to hear a rant, skip the comment. (If you wish to respond, don’t use that e-mail, it won’t work, use aim.)
Of all people… do yourself a favor and bust out vi. Use the specification itself as reference and your problem is solved. It will be exactly to standard because you design from it, it will be understandable because you wrote it all and it will likely be modular and will include data-presentation seperation if you design it right.
Dreamweaver is a clutch that is effecient in the short run, but in the long run… I think it’s a shot in the foot. There’s nothing like writing a bit of data and linking it into a CSS sheet and calling it a day.
November 27th, 2004 at 7:31 am
Heh, you’re absolutely right. Our web design guy loves DW because he
gets to move stuff around graphically. I look at the code it produces
and it is definitely "Because if it is not-Scottish, it's CRAP!">not-Scottish.
Other than vi, I would recommend picking up a copy of "http://tidy.sourceforge.net">Tidy, to clean up your code (and
automatically do all the mundane things we know we need to do, but just
can’t be bothered to, like closing tags, converting <b> to
<strong>, to  , etc).
November 27th, 2004 at 7:33 am
Oh for the love of all that is holy, the lack of a “preview” function is asnine.
That last bit should be “ to & #160;” (without the space after the amp).
November 27th, 2004 at 5:38 pm
That explains a lot:)
Since I can’t futz with the computer, I can spend some family time.