UPDATE 2: We’re up to 55 hours, 20 minutes now.
UPDATE: Maybe I should disable the level 9 bzip compression. 36 hours is a bit long to dump 500GB of data to disk:
Performing backup of /export/home to /media/usb1/home.0 .. Checking external disk... fsck 1.39 (29-May-2006) /dev/sdd1: clean, 23/244203520 files, 12079391/488378000 blocks Mounting external disk... DUMP: Date of this level 0 dump: Sat Dec 25 10:45:34 2010 DUMP: Dumping /dev/mapper/VolGroup00-LogVol12 (/export/home) to /media/usb1/home.0 DUMP: Label: none DUMP: Writing 10 Kilobyte records DUMP: Compressing output at compression level 9 (bzlib) DUMP: mapping (Pass I) [regular files] DUMP: mapping (Pass II) [directories] DUMP: estimated 144080727 blocks. DUMP: Volume 1 started with block 1 at: Sat Dec 25 10:51:48 2010 DUMP: dumping (Pass III) [directories] DUMP: dumping (Pass IV) [regular files] DUMP: 0.23% done at 1102 kB/s, finished in 36:13
Hooray! The level 0 backup failed last night on the new 2T drive. But it’s not the new drive’s fault. It seems the news spool went to crap again. More than 2 million multiply-claimed inodes on a very dense, random-access, read and write-intensive ext3 filesystem (~125 million total inodes in only 2G). Don’t forget that not only does the news spool live here but the mail spool, cron spool, and at spool also.
So now I get to watch a fsck -f -y sort out the tangled mass of inode refs while innd, crond, atd, sendmail, and dovecot (that’s NNTP news, cron daemon, at daemon, mail transport agent, and IMAP server) are all down.
RedHat has a new LVM driver in their latest kernels but I’m 22 releases behind current because it breaks compatibility with older LVM pools and if I try to boot a newer kernel, it panics when it can’t scan the volume pool. The solution would be to move everything off to the 2T drive, erase the LVM metadata off the disks, and create a new LVM pool under the new kernel.
But that requires 3 days of site downtime (thankfully, Minecraft wouldn’t suffer at all being on a different server).
On the plus side, my desktop binary clock widget made a neat little box at 0757.05 this morning: