Oh, what a way to start Monday. So I was about to check-in some new code but Subversion server was not responding. Sure, I thought maybe it just needs rebooting… Happens now and then. I go into the server room (well it is just a closet) and reboot.

Clickety, clikety, click is not noise you want to hear from hard-disk where source code lives, yet, that’s what I heard. Machine is not booting up either. I pulled it apart took hard-disk out, put it into the USB enclosure to try to get data off just in case, but no luck. It is gone. Just like that… Now, I am thinking, we’ll see how good our backup is.

I knew backup procedure we have is fairly solid, but since I wrote about backup before, I knew that Murphy does not look favorable on those that preach about backup. They get hit the worst, so that was in the back(up) of my mind.

Now we have triple backup for this server. Two of these backups are off-site (Mozy and Syncplicity) and one is on site on Drobo-box. I first reach for Drobo to see is everything there. As you might expect, Murphy left his calling card, one week of backups was missing. Okey, not a big deal, but I wanted it all. I check Syncplicity and everything is there to the last minute and it downloads fairly fast. Whew, that was close. I check Mozy and everything is there too, but you can’t restore nothing from that place, restore speed is like 400 bytes-per-second…

So yesterday was spent re-building the server. It is good exercise, but tedious…

Couple of lessons here, and I’ll test Murphy again by preaching about backup:

• Automated backup is great, but something needs to monitor whether software that is doing it actually runs and has not crashed. That happened here where Drobo backup was 1 week behind because software crashed and nobody knew. It could have been worse.
• Online backups are great, BUT nothing beats in house backup. We will be adding another Drobo, this time Drobo FS to have redundant in house backup. We’ll be also actively monitoring whether software that is doing backups is actually running and doing its job.
• Online backups where you can’t access your files easily is worthless. We’ll be switching from Mozy to DropBox since we are already using DropBox and it is working really nice.

Luckily what could have turned into huge disaster was just minor annoyance. The source server is back up and running with nothing lost. Having real-time backup sync off-site using Syncplicity and DropBox really makes it so you don’t have to sweat when failures happen even if your local backup fails. And failures even multiple ones will happen. This HDD that failed was just 5 years old.

You can really never have enough backup… Stay backedup!