LVM over MDRAID on RHEL 5.5

UPDATE 2:  We’re up to 55 hours, 20 minutes now.

UPDATE:  Maybe I should disable the level 9 bzip compression.  36 hours is a bit long to dump 500GB of data to disk:

Performing backup of /export/home to /media/usb1/home.0 ..

Checking external disk...
fsck 1.39 (29-May-2006)
/dev/sdd1: clean, 23/244203520 files, 12079391/488378000 blocks
Mounting external disk...
  DUMP: Date of this level 0 dump: Sat Dec 25 10:45:34 2010
  DUMP: Dumping /dev/mapper/VolGroup00-LogVol12 (/export/home) to /media/usb1/home.0
  DUMP: Label: none
  DUMP: Writing 10 Kilobyte records
  DUMP: Compressing output at compression level 9 (bzlib)
  DUMP: mapping (Pass I) [regular files]
  DUMP: mapping (Pass II) [directories]
  DUMP: estimated 144080727 blocks.
  DUMP: Volume 1 started with block 1 at: Sat Dec 25 10:51:48 2010
  DUMP: dumping (Pass III) [directories]
  DUMP: dumping (Pass IV) [regular files]
  DUMP: 0.23% done at 1102 kB/s, finished in 36:13

Hooray!  The level 0 backup failed last night on the new 2T drive.  But it’s not the new drive’s fault.  It seems the news spool went to crap again.  More than 2 million multiply-claimed inodes on a very dense, random-access, read and write-intensive ext3 filesystem (~125 million total inodes in only 2G).  Don’t forget that not only does the news spool live here but the mail spool, cron spool, and at spool also.

So now I get to watch a fsck -f -y sort out the tangled mass of inode refs while innd, crond, atd, sendmail, and dovecot (that’s NNTP news, cron daemon, at daemon, mail transport agent, and IMAP server) are all down.

RedHat has a new LVM driver in their latest kernels but I’m 22 releases behind current because it breaks compatibility with older LVM pools and if I try to boot a newer kernel, it panics when it can’t scan the volume pool.  The solution would be to move everything off to the 2T drive, erase the LVM metadata off the disks, and create a new LVM pool under the new kernel.

But that requires 3 days of site downtime (thankfully, Minecraft wouldn’t suffer at all being on a different server).

On the plus side, my desktop binary clock widget made a neat little box at 0757.05 this morning:

Time reads as 0757.05 (7:57:05 A.M.)

Time reads as 0757.05 (7:57:05 A.M.)



  1. #1 by Chadwick on December 25, 2010 - 9:33 AM

    That is a neat plus side. So which direction does it read? Being a box, it can kind of go any way. For some reason, my natural inclination was to read from the bottom up.

    • #2 by Joshua on December 25, 2010 - 10:04 AM

      Half-right. It’s left-right, top down. The bottom row of boxes are ones, the next are twos, then fours, and the top are eights. You add down the column, starting from the top. So the first column is 0, the second is 7 (1 four, 1 two, and 1 one), the third is 5 (1 four, 0 twos, and 1 one), the fourth is 7 again, the fifth is 0 and the sixth is 5 again.

      This pattern is called Binary Coded Decimal (BCD) and is the pattern used by Real Time Clocks (RTCs) in most modern electronics. It’s very old (UNIVAC used BCD for basic arithmetic) and IBM still uses BCD for mainframe processors. Most other modern computers use pure binary. Almost all binary clocks and watches (like those on ThinkGeek) use BCD (but they also have a pure binary option that reads left-right, then bottom-up but that’s slower to read than the BCD format).

  2. #3 by Chadwick on December 25, 2010 - 9:36 AM

    Don’t forget that not only does the news spool live here but the mail spool, cron spool, and at spool also.

    By the time I got to the end of that sentence, “spool” read like a dirty word. Just one of those weird things.

    • #4 by Joshua on December 25, 2010 - 10:05 AM

      I could’ve written it better as “but the mail, cron, and at spools also live here.” But I didn’t ’cause I was in a hurry.

    • #5 by Joshua on December 25, 2010 - 10:06 AM

      And since this fsck has been running for an hour and 45 minutes now, I’m beginning to see spool as a dirty word for other reasons. 😛

  3. #6 by Joshua on December 25, 2010 - 10:11 AM

    LOL I should bitch about fsck more often. By the time I finished writing the comment about it taking an hour and 45 minutes, it had finished.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: