that sucked
My main server went down early last Thursday (2005-06-16) because of a disk failure. I've never liked Maxtor drives-- and that was what we lost. The replacement is a Seagate which I like a little better. Although my provider replaced the drive in a resonable amount of time, I wasted two and half days fucking around with Fedora and getting no where. The server gets Fedora Core 2 by default, which is inadequate for numerous reasons. Anyway, the previously successful upgrade didn't go so well this time and I ended up with an unbootable server and much, much more downtime than anticipated.
By Sunday afternoon I decided to ditch Fedora altogether and move to the new Debian stable (Sarge) 3.1 instead. Debian rules, which made this installation easy, especially considering my provider has a Debian-based rescue system (small network-booted distro) which made the install possible . Problem was I still couldn't boot when I was done.
Finally found out that my host not only has the Debian rescue system, but I have a serial console as well! This is fucking awesome. I configured my system and the bootloader for serial console, rebooted, watched the boot progress all the way up, starting with grub; identified the problem and had it fixed within 15 minutes.
Apache was running again by Sunday evening (2005-06-19) and mail was operational by the following morning. That makes for a total of about 36 hours downtime. The backup servers performed beautifully; handling about 750 emails in that time and keeping the dns records alive.
So it was an expensive lesson (as far as my time goes). But I learned: next time don't fuck around with Fedora; go straight to Debian. And use this serial console now that we know it exists because it's awesome.
