Yesterday, FUMBBL went down due to a power failure. Once power was restored and the servers were up and running again, it was fairly late (a bit past 1am or so) and I went to sleep. On waking up, I noticed that the site was being incredibly sluggish.

So, I spent some time trying to localise the issue and found that one of the hard drives that the database is running on was failing (read speeds of 1MB/sec, which is about 1% of what I expect from the drive). Further investigation showed multiple errors being reported by the drive self-test (SMART).

This meant that I rushed off to the local computer hardware store and picked up a replacement drive, which is now installed in the system. At the time of writing, the RAID array is recovering and will likely be doing so for the next few hours (the drive is 2TB running as part of a RAID5 array).

Bottom line is that the site should be functional again and because of the RAID rebuild will be slightly sluggish for a few hours (although not nearly as bad as it was while the failed drive was installed).

I apologise for any inconvenience this downtime has caused you guys.