The last days of dustpuppy

This morning I was greeted by a root mailbox full of messages from the SMART monitors and MDRAID subsystem:

Return-Path: <webmin@dustpuppy.lakemasoniccenter.org>
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
	dustpuppy.lakemasoniccenter.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.6 required=5.0 tests=ALL_TRUSTED,AWL autolearn=ham
	version=3.2.5
Received: from dustpuppy.lakemasoniccenter.org (localhost.localdomain [127.0.0.1])
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8) with ESMTP id p4151oew029428
	for <root@dustpuppy.lakemasoniccenter.org>; Sun, 1 May 2011 00:01:50 -0500
Received: (from root@localhost)
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8/Submit) id p4151o8X029427;
	Sun, 1 May 2011 00:01:50 -0500
From: webmin@lakemasoniccenter.org
To: root@dustpuppy.lakemasoniccenter.org
Cc: 
Subject: Backup of / on dustpuppy.lakemasoniccenter.org
Date: Sun, 01 May 2011 00:01:49 -0500 (CDT)
Message-Id: <1304226109.28974.1@dustpuppy.lakemasoniccenter.org>

Backup of / on dustpuppy.lakemasoniccenter.org

  DUMP: Date of this level 0 dump: Sun May  1 00:00:26 2011
  DUMP: Dumping /dev/md5 (/) to /media/expansion/root_0
  DUMP: Label: none
  DUMP: Writing 10 Kilobyte records
  DUMP: mapping (Pass I) [regular files]
  DUMP: mapping (Pass II) [directories]
  DUMP: estimated 816168 blocks.
  DUMP: Volume 1 started with block 1 at: Sun May  1 00:00:35 2011
  DUMP: dumping (Pass III) [directories]
  DUMP: dumping (Pass IV) [regular files]
  DUMP: Closing /media/expansion/root_0
  DUMP: Volume 1 completed at: Sun May  1 00:01:42 2011
  DUMP: Volume 1 871680 blocks (851.25MB)
  DUMP: Volume 1 took 0:01:07
  DUMP: Volume 1 transfer rate: 13010 kB/s
  DUMP: 871680 blocks (851.25MB) on 1 volume(s)
  DUMP: finished in 66 seconds, throughput 13207 kBytes/sec
  DUMP: Date of this level 0 dump: Sun May  1 00:00:26 2011
  DUMP: Date this dump completed:  Sun May  1 00:01:42 2011
  DUMP: Average transfer rate: 13010 kB/s
  DUMP: DUMP IS DONE

Verifying backup ..

Backup succeeded
Return-Path: <root@dustpuppy.lakemasoniccenter.org>
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
	dustpuppy.lakemasoniccenter.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.6 required=5.0 tests=ALL_TRUSTED,AWL autolearn=ham
	version=3.2.5
Received: from dustpuppy.lakemasoniccenter.org (localhost.localdomain [127.0.0.1])
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8) with ESMTP id p414hCfp028140
	for <root@dustpuppy.lakemasoniccenter.org>; Sat, 30 Apr 2011 23:43:12 -0500
Received: (from root@localhost)
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8/Submit) id p414PQMJ026852;
	Sat, 30 Apr 2011 23:25:26 -0500
Date: Sat, 30 Apr 2011 23:25:26 -0500
Message-Id: <201105010425.p414PQMJ026852@dustpuppy.lakemasoniccenter.org>
From: mdadm monitoring <root@dustpuppy.lakemasoniccenter.org>
To: root@dustpuppy.lakemasoniccenter.org
Subject: Fail event on /dev/md0:dustpuppy.lakemasoniccenter.org

This is an automatically generated mail message from mdadm
running on dustpuppy.lakemasoniccenter.org

A Fail event had been detected on md device /dev/md0.

It could be related to component device /dev/sdb7.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid6] [raid5] [raid4] 
md3 : active raid5 sdd2[3] sdc2[2] sdb2[4](F) sda2[5](F) hda3[0]
      315628800 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md2 : active raid5 sdd3[3] sdc3[2] sdb3[1] sda3[4](S) hda5[0]
      97522176 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md9 : active raid5 sdd5[3] sdc5[2] sdb5[1] sda5[4](S) hda6[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md8 : active raid5 sdd6[3] sdc6[2] sdb6[4](F) sda6[5](F) hda7[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md0 : active raid5 sdd7[3] sdc7[2] sdb7[4](F) sda7[5](F) hda8[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md4 : active raid5 sdd8[3] sdc8[2] sdb8[1] sda8[4](S) hda9[0]
      4192512 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md7 : active raid5 sdd10[3] sdc10[2] sdb10[1] sda10[4](S) hda11[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md6 : active raid5 sdd11[3] sdc11[2] sdb11[1] sda11[4](S) hda12[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md1 : active raid5 sdd12[3] sdc12[2] sdb12[4](F) sda12[5](F) hda13[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md12 : active raid5 sdd13[3] sdc13[2] sdb13[4](F) sda13[5](F) hda14[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md10 : active raid5 sdd14[3] sdc14[2] sdb14[1] sda14[4](S) hda15[0]
      1035264 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md11 : active raid5 sdd15[3] sdc15[2] sdb15[1] sda15[4](S) hda16[0]
      529152 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md5 : active raid5 sdd9[3] sdc9[2] sdb9[1] sda9[4](S) hda10[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

unused devices: <none>
Return-Path: <root@dustpuppy.lakemasoniccenter.org>
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
	dustpuppy.lakemasoniccenter.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.6 required=5.0 tests=ALL_TRUSTED,AWL autolearn=ham
	version=3.2.5
Received: from dustpuppy.lakemasoniccenter.org (localhost.localdomain [127.0.0.1])
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8) with ESMTP id p414U8Mr027314
	for <root@dustpuppy.lakemasoniccenter.org>; Sat, 30 Apr 2011 23:30:08 -0500
Received: (from root@localhost)
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8/Submit) id p414U7AO027313;
	Sat, 30 Apr 2011 23:30:07 -0500
Date: Sat, 30 Apr 2011 23:30:07 -0500
Message-Id: <201105010430.p414U7AO027313@dustpuppy.lakemasoniccenter.org>
From: mdadm monitoring <root@dustpuppy.lakemasoniccenter.org>
To: root@dustpuppy.lakemasoniccenter.org
Subject: Fail event on /dev/md10:dustpuppy.lakemasoniccenter.org

This is an automatically generated mail message from mdadm
running on dustpuppy.lakemasoniccenter.org

A Fail event had been detected on md device /dev/md10.

It could be related to component device /dev/sdb14.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid6] [raid5] [raid4] 
md3 : active raid5 sdd2[3] sdc2[2] sdb2[4](F) sda2[5](F) hda3[0]
      315628800 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md2 : active raid5 sdd3[3] sdc3[2] sdb3[1] sda3[4](S) hda5[0]
      97522176 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md9 : active raid5 sdd5[3] sdc5[2] sdb5[1] sda5[4](S) hda6[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md8 : active raid5 sdd6[3] sdc6[2] sdb6[4](F) sda6[5](F) hda7[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md0 : active raid5 sdd7[3] sdc7[2] sdb7[4](F) sda7[5](F) hda8[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md4 : active raid5 sdd8[3] sdc8[2] sdb8[4](F) sda8[5](F) hda9[0]
      4192512 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md7 : active raid5 sdd10[3] sdc10[2] sdb10[4](F) sda10[5](F) hda11[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md6 : active raid5 sdd11[3] sdc11[2] sdb11[4](F) sda11[5](F) hda12[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md1 : active raid5 sdd12[3] sdc12[2] sdb12[4](F) sda12[5](F) hda13[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md12 : active raid5 sdd13[3] sdc13[2] sdb13[4](F) sda13[5](F) hda14[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md10 : active raid5 sdd14[3] sdc14[2] sdb14[4](F) sda14[5](F) hda15[0]
      1035264 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md11 : active raid5 sdd15[3] sdc15[2] sdb15[4](F) sda15[5](F) hda16[0]
      529152 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md5 : active raid5 sdd9[3] sdc9[2] sdb9[4](F) sda9[5](F) hda10[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

unused devices: <none>
Return-Path: <root@dustpuppy.lakemasoniccenter.org>
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
	dustpuppy.lakemasoniccenter.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.6 required=5.0 tests=ALL_TRUSTED,AWL autolearn=ham
	version=3.2.5
Received: from dustpuppy.lakemasoniccenter.org (localhost.localdomain [127.0.0.1])
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8) with ESMTP id p414hCg1028140
	for <root@dustpuppy.lakemasoniccenter.org>; Sat, 30 Apr 2011 23:43:12 -0500
Received: (from root@localhost)
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8/Submit) id p414PWfv026862;
	Sat, 30 Apr 2011 23:25:32 -0500
Date: Sat, 30 Apr 2011 23:25:32 -0500
Message-Id: <201105010425.p414PWfv026862@dustpuppy.lakemasoniccenter.org>
From: mdadm monitoring <root@dustpuppy.lakemasoniccenter.org>
To: root@dustpuppy.lakemasoniccenter.org
Subject: Fail event on /dev/md11:dustpuppy.lakemasoniccenter.org

This is an automatically generated mail message from mdadm
running on dustpuppy.lakemasoniccenter.org

A Fail event had been detected on md device /dev/md11.

It could be related to component device /dev/sdb15.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid6] [raid5] [raid4] 
md3 : active raid5 sdd2[3] sdc2[2] sdb2[4](F) sda2[5](F) hda3[0]
      315628800 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md2 : active raid5 sdd3[3] sdc3[2] sdb3[1] sda3[4](S) hda5[0]
      97522176 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md9 : active raid5 sdd5[3] sdc5[2] sdb5[1] sda5[4](S) hda6[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md8 : active raid5 sdd6[3] sdc6[2] sdb6[4](F) sda6[5](F) hda7[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md0 : active raid5 sdd7[3] sdc7[2] sdb7[4](F) sda7[5](F) hda8[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md4 : active raid5 sdd8[3] sdc8[2] sdb8[4](F) sda8[5](F) hda9[0]
      4192512 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md7 : active raid5 sdd10[3] sdc10[2] sdb10[4](F) sda10[5](F) hda11[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md6 : active raid5 sdd11[3] sdc11[2] sdb11[1] sda11[4](S) hda12[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md1 : active raid5 sdd12[3] sdc12[2] sdb12[4](F) sda12[5](F) hda13[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md12 : active raid5 sdd13[3] sdc13[2] sdb13[4](F) sda13[5](F) hda14[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md10 : active raid5 sdd14[3] sdc14[2] sdb14[1] sda14[4](S) hda15[0]
      1035264 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md11 : active raid5 sdd15[3] sdc15[2] sdb15[4](F) sda15[5](F) hda16[0]
      529152 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md5 : active raid5 sdd9[3] sdc9[2] sdb9[4](F) sda9[5](F) hda10[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

unused devices: <none>
Return-Path: <root@dustpuppy.lakemasoniccenter.org>
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
	dustpuppy.lakemasoniccenter.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.6 required=5.0 tests=ALL_TRUSTED,AWL autolearn=ham
	version=3.2.5
Received: from dustpuppy.lakemasoniccenter.org (localhost.localdomain [127.0.0.1])
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8) with ESMTP id p414hCfx028140
	for <root@dustpuppy.lakemasoniccenter.org>; Sat, 30 Apr 2011 23:43:12 -0500
Received: (from root@localhost)
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8/Submit) id p414PR4N026859;
	Sat, 30 Apr 2011 23:25:27 -0500
Date: Sat, 30 Apr 2011 23:25:27 -0500
Message-Id: <201105010425.p414PR4N026859@dustpuppy.lakemasoniccenter.org>
From: mdadm monitoring <root@dustpuppy.lakemasoniccenter.org>
To: root@dustpuppy.lakemasoniccenter.org
Subject: Fail event on /dev/md12:dustpuppy.lakemasoniccenter.org

This is an automatically generated mail message from mdadm
running on dustpuppy.lakemasoniccenter.org

A Fail event had been detected on md device /dev/md12.

It could be related to component device /dev/sdb13.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid6] [raid5] [raid4] 
md3 : active raid5 sdd2[3] sdc2[2] sdb2[4](F) sda2[5](F) hda3[0]
      315628800 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md2 : active raid5 sdd3[3] sdc3[2] sdb3[1] sda3[4](S) hda5[0]
      97522176 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md9 : active raid5 sdd5[3] sdc5[2] sdb5[1] sda5[4](S) hda6[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md8 : active raid5 sdd6[3] sdc6[2] sdb6[4](F) sda6[5](F) hda7[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md0 : active raid5 sdd7[3] sdc7[2] sdb7[4](F) sda7[5](F) hda8[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md4 : active raid5 sdd8[3] sdc8[2] sdb8[4](F) sda8[5](F) hda9[0]
      4192512 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md7 : active raid5 sdd10[3] sdc10[2] sdb10[4](F) sda10[5](F) hda11[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md6 : active raid5 sdd11[3] sdc11[2] sdb11[1] sda11[4](S) hda12[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md1 : active raid5 sdd12[3] sdc12[2] sdb12[4](F) sda12[5](F) hda13[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md12 : active raid5 sdd13[3] sdc13[2] sdb13[4](F) sda13[5](F) hda14[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md10 : active raid5 sdd14[3] sdc14[2] sdb14[1] sda14[4](S) hda15[0]
      1035264 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md11 : active raid5 sdd15[3] sdc15[2] sdb15[1] sda15[4](S) hda16[0]
      529152 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md5 : active raid5 sdd9[3] sdc9[2] sdb9[4](F) sda9[5](F) hda10[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

unused devices: <none>
Return-Path: <root@dustpuppy.lakemasoniccenter.org>
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
	dustpuppy.lakemasoniccenter.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.6 required=5.0 tests=ALL_TRUSTED,AWL autolearn=ham
	version=3.2.5
Received: from dustpuppy.lakemasoniccenter.org (localhost.localdomain [127.0.0.1])
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8) with ESMTP id p414hCfv028140
	for <root@dustpuppy.lakemasoniccenter.org>; Sat, 30 Apr 2011 23:43:12 -0500
Received: (from root@localhost)
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8/Submit) id p414PRVm026857;
	Sat, 30 Apr 2011 23:25:27 -0500
Date: Sat, 30 Apr 2011 23:25:27 -0500
Message-Id: <201105010425.p414PRVm026857@dustpuppy.lakemasoniccenter.org>
From: mdadm monitoring <root@dustpuppy.lakemasoniccenter.org>
To: root@dustpuppy.lakemasoniccenter.org
Subject: Fail event on /dev/md1:dustpuppy.lakemasoniccenter.org

This is an automatically generated mail message from mdadm
running on dustpuppy.lakemasoniccenter.org

A Fail event had been detected on md device /dev/md1.

It could be related to component device /dev/sdb12.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid6] [raid5] [raid4] 
md3 : active raid5 sdd2[3] sdc2[2] sdb2[4](F) sda2[5](F) hda3[0]
      315628800 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md2 : active raid5 sdd3[3] sdc3[2] sdb3[1] sda3[4](S) hda5[0]
      97522176 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md9 : active raid5 sdd5[3] sdc5[2] sdb5[1] sda5[4](S) hda6[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md8 : active raid5 sdd6[3] sdc6[2] sdb6[4](F) sda6[5](F) hda7[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md0 : active raid5 sdd7[3] sdc7[2] sdb7[4](F) sda7[5](F) hda8[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md4 : active raid5 sdd8[3] sdc8[2] sdb8[4](F) sda8[5](F) hda9[0]
      4192512 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md7 : active raid5 sdd10[3] sdc10[2] sdb10[4](F) sda10[5](F) hda11[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md6 : active raid5 sdd11[3] sdc11[2] sdb11[1] sda11[4](S) hda12[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md1 : active raid5 sdd12[3] sdc12[2] sdb12[4](F) sda12[5](F) hda13[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md12 : active raid5 sdd13[3] sdc13[2] sdb13[4](F) sda13[5](F) hda14[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md10 : active raid5 sdd14[3] sdc14[2] sdb14[1] sda14[4](S) hda15[0]
      1035264 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md11 : active raid5 sdd15[3] sdc15[2] sdb15[1] sda15[4](S) hda16[0]
      529152 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md5 : active raid5 sdd9[3] sdc9[2] sdb9[4](F) sda9[5](F) hda10[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

unused devices: <none>
Return-Path: <root@dustpuppy.lakemasoniccenter.org>
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
	dustpuppy.lakemasoniccenter.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.6 required=5.0 tests=ALL_TRUSTED,AWL autolearn=ham
	version=3.2.5
Received: from dustpuppy.lakemasoniccenter.org (localhost.localdomain [127.0.0.1])
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8) with ESMTP id p414WY26027531
	for <root@dustpuppy.lakemasoniccenter.org>; Sat, 30 Apr 2011 23:32:34 -0500
Received: (from root@localhost)
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8/Submit) id p414WYDm027530;
	Sat, 30 Apr 2011 23:32:34 -0500
Date: Sat, 30 Apr 2011 23:32:34 -0500
Message-Id: <201105010432.p414WYDm027530@dustpuppy.lakemasoniccenter.org>
From: mdadm monitoring <root@dustpuppy.lakemasoniccenter.org>
To: root@dustpuppy.lakemasoniccenter.org
Subject: Fail event on /dev/md2:dustpuppy.lakemasoniccenter.org

This is an automatically generated mail message from mdadm
running on dustpuppy.lakemasoniccenter.org

A Fail event had been detected on md device /dev/md2.

It could be related to component device /dev/sdb3.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid6] [raid5] [raid4] 
md3 : active raid5 sdd2[3] sdc2[2] sdb2[4](F) sda2[5](F) hda3[0]
      315628800 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md2 : active raid5 sdd3[3] sdc3[2] sdb3[4](F) sda3[5](F) hda5[0]
      97522176 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md9 : active raid5 sdd5[3] sdc5[2] sdb5[1] sda5[4](S) hda6[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md8 : active raid5 sdd6[3] sdc6[2] sdb6[4](F) sda6[5](F) hda7[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md0 : active raid5 sdd7[3] sdc7[2] sdb7[4](F) sda7[5](F) hda8[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md4 : active raid5 sdd8[3] sdc8[2] sdb8[4](F) sda8[5](F) hda9[0]
      4192512 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md7 : active raid5 sdd10[3] sdc10[2] sdb10[4](F) sda10[5](F) hda11[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md6 : active raid5 sdd11[3] sdc11[2] sdb11[4](F) sda11[5](F) hda12[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md1 : active raid5 sdd12[3] sdc12[2] sdb12[4](F) sda12[5](F) hda13[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md12 : active raid5 sdd13[3] sdc13[2] sdb13[4](F) sda13[5](F) hda14[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md10 : active raid5 sdd14[3] sdc14[2] sdb14[4](F) sda14[5](F) hda15[0]
      1035264 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md11 : active raid5 sdd15[3] sdc15[2] sdb15[4](F) sda15[5](F) hda16[0]
      529152 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md5 : active raid5 sdd9[3] sdc9[2] sdb9[4](F) sda9[5](F) hda10[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

unused devices: <none>
Return-Path: <root@dustpuppy.lakemasoniccenter.org>
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
	dustpuppy.lakemasoniccenter.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.6 required=5.0 tests=ALL_TRUSTED,AWL autolearn=ham
	version=3.2.5
Received: from dustpuppy.lakemasoniccenter.org (localhost.localdomain [127.0.0.1])
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8) with ESMTP id p414hCfr028140
	for <root@dustpuppy.lakemasoniccenter.org>; Sat, 30 Apr 2011 23:43:12 -0500
Received: (from root@localhost)
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8/Submit) id p414PRCV026858;
	Sat, 30 Apr 2011 23:25:27 -0500
Date: Sat, 30 Apr 2011 23:25:27 -0500
Message-Id: <201105010425.p414PRCV026858@dustpuppy.lakemasoniccenter.org>
From: mdadm monitoring <root@dustpuppy.lakemasoniccenter.org>
To: root@dustpuppy.lakemasoniccenter.org
Subject: Fail event on /dev/md3:dustpuppy.lakemasoniccenter.org

This is an automatically generated mail message from mdadm
running on dustpuppy.lakemasoniccenter.org

A Fail event had been detected on md device /dev/md3.

It could be related to component device /dev/sdb2.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid6] [raid5] [raid4] 
md3 : active raid5 sdd2[3] sdc2[2] sdb2[4](F) sda2[5](F) hda3[0]
      315628800 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md2 : active raid5 sdd3[3] sdc3[2] sdb3[1] sda3[4](S) hda5[0]
      97522176 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md9 : active raid5 sdd5[3] sdc5[2] sdb5[1] sda5[4](S) hda6[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md8 : active raid5 sdd6[3] sdc6[2] sdb6[4](F) sda6[5](F) hda7[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md0 : active raid5 sdd7[3] sdc7[2] sdb7[4](F) sda7[5](F) hda8[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md4 : active raid5 sdd8[3] sdc8[2] sdb8[4](F) sda8[5](F) hda9[0]
      4192512 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md7 : active raid5 sdd10[3] sdc10[2] sdb10[4](F) sda10[5](F) hda11[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md6 : active raid5 sdd11[3] sdc11[2] sdb11[1] sda11[4](S) hda12[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md1 : active raid5 sdd12[3] sdc12[2] sdb12[4](F) sda12[5](F) hda13[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md12 : active raid5 sdd13[3] sdc13[2] sdb13[4](F) sda13[5](F) hda14[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md10 : active raid5 sdd14[3] sdc14[2] sdb14[1] sda14[4](S) hda15[0]
      1035264 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md11 : active raid5 sdd15[3] sdc15[2] sdb15[1] sda15[4](S) hda16[0]
      529152 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md5 : active raid5 sdd9[3] sdc9[2] sdb9[4](F) sda9[5](F) hda10[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

unused devices: <none>
Return-Path: <root@dustpuppy.lakemasoniccenter.org>
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
	dustpuppy.lakemasoniccenter.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.6 required=5.0 tests=ALL_TRUSTED,AWL autolearn=ham
	version=3.2.5
Received: from dustpuppy.lakemasoniccenter.org (localhost.localdomain [127.0.0.1])
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8) with ESMTP id p414hCfl028140
	for <root@dustpuppy.lakemasoniccenter.org>; Sat, 30 Apr 2011 23:43:12 -0500
Received: (from root@localhost)
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8/Submit) id p414PVZK026861;
	Sat, 30 Apr 2011 23:25:31 -0500
Date: Sat, 30 Apr 2011 23:25:31 -0500
Message-Id: <201105010425.p414PVZK026861@dustpuppy.lakemasoniccenter.org>
From: mdadm monitoring <root@dustpuppy.lakemasoniccenter.org>
To: root@dustpuppy.lakemasoniccenter.org
Subject: Fail event on /dev/md4:dustpuppy.lakemasoniccenter.org

This is an automatically generated mail message from mdadm
running on dustpuppy.lakemasoniccenter.org

A Fail event had been detected on md device /dev/md4.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid6] [raid5] [raid4] 
md3 : active raid5 sdd2[3] sdc2[2] sdb2[4](F) sda2[5](F) hda3[0]
      315628800 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md2 : active raid5 sdd3[3] sdc3[2] sdb3[1] sda3[4](S) hda5[0]
      97522176 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md9 : active raid5 sdd5[3] sdc5[2] sdb5[1] sda5[4](S) hda6[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md8 : active raid5 sdd6[3] sdc6[2] sdb6[4](F) sda6[5](F) hda7[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md0 : active raid5 sdd7[3] sdc7[2] sdb7[4](F) sda7[5](F) hda8[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md4 : active raid5 sdd8[3] sdc8[2] sdb8[4](F) sda8[5](F) hda9[0]
      4192512 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md7 : active raid5 sdd10[3] sdc10[2] sdb10[4](F) sda10[5](F) hda11[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md6 : active raid5 sdd11[3] sdc11[2] sdb11[1] sda11[4](S) hda12[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md1 : active raid5 sdd12[3] sdc12[2] sdb12[4](F) sda12[5](F) hda13[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md12 : active raid5 sdd13[3] sdc13[2] sdb13[4](F) sda13[5](F) hda14[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md10 : active raid5 sdd14[3] sdc14[2] sdb14[1] sda14[4](S) hda15[0]
      1035264 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md11 : active raid5 sdd15[3] sdc15[2] sdb15[4](F) sda15[5](F) hda16[0]
      529152 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md5 : active raid5 sdd9[3] sdc9[2] sdb9[4](F) sda9[5](F) hda10[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

unused devices: <none>
Return-Path: <root@dustpuppy.lakemasoniccenter.org>
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
	dustpuppy.lakemasoniccenter.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.6 required=5.0 tests=ALL_TRUSTED,AWL autolearn=ham
	version=3.2.5
Received: from dustpuppy.lakemasoniccenter.org (localhost.localdomain [127.0.0.1])
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8) with ESMTP id p414hCfn028140
	for <root@dustpuppy.lakemasoniccenter.org>; Sat, 30 Apr 2011 23:43:12 -0500
Received: (from root@localhost)
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8/Submit) id p414PW3b026863;
	Sat, 30 Apr 2011 23:25:32 -0500
Date: Sat, 30 Apr 2011 23:25:32 -0500
Message-Id: <201105010425.p414PW3b026863@dustpuppy.lakemasoniccenter.org>
From: mdadm monitoring <root@dustpuppy.lakemasoniccenter.org>
To: root@dustpuppy.lakemasoniccenter.org
Subject: Fail event on /dev/md5:dustpuppy.lakemasoniccenter.org

This is an automatically generated mail message from mdadm
running on dustpuppy.lakemasoniccenter.org

A Fail event had been detected on md device /dev/md5.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid6] [raid5] [raid4] 
md3 : active raid5 sdd2[3] sdc2[2] sdb2[4](F) sda2[5](F) hda3[0]
      315628800 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md2 : active raid5 sdd3[3] sdc3[2] sdb3[1] sda3[4](S) hda5[0]
      97522176 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md9 : active raid5 sdd5[3] sdc5[2] sdb5[1] sda5[4](S) hda6[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md8 : active raid5 sdd6[3] sdc6[2] sdb6[4](F) sda6[5](F) hda7[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md0 : active raid5 sdd7[3] sdc7[2] sdb7[4](F) sda7[5](F) hda8[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md4 : active raid5 sdd8[3] sdc8[2] sdb8[4](F) sda8[5](F) hda9[0]
      4192512 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md7 : active raid5 sdd10[3] sdc10[2] sdb10[4](F) sda10[5](F) hda11[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md6 : active raid5 sdd11[3] sdc11[2] sdb11[1] sda11[4](S) hda12[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md1 : active raid5 sdd12[3] sdc12[2] sdb12[4](F) sda12[5](F) hda13[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md12 : active raid5 sdd13[3] sdc13[2] sdb13[4](F) sda13[5](F) hda14[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md10 : active raid5 sdd14[3] sdc14[2] sdb14[1] sda14[4](S) hda15[0]
      1035264 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md11 : active raid5 sdd15[3] sdc15[2] sdb15[4](F) sda15[5](F) hda16[0]
      529152 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md5 : active raid5 sdd9[3] sdc9[2] sdb9[4](F) sda9[5](F) hda10[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

unused devices: <none>
Return-Path: <root@dustpuppy.lakemasoniccenter.org>
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
	dustpuppy.lakemasoniccenter.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.6 required=5.0 tests=ALL_TRUSTED,AWL autolearn=ham
	version=3.2.5
Received: from dustpuppy.lakemasoniccenter.org (localhost.localdomain [127.0.0.1])
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8) with ESMTP id p414QFfV026870
	for <root@dustpuppy.lakemasoniccenter.org>; Sat, 30 Apr 2011 23:26:15 -0500
Received: (from root@localhost)
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8/Submit) id p414QFaZ026869;
	Sat, 30 Apr 2011 23:26:15 -0500
Date: Sat, 30 Apr 2011 23:26:15 -0500
Message-Id: <201105010426.p414QFaZ026869@dustpuppy.lakemasoniccenter.org>
From: mdadm monitoring <root@dustpuppy.lakemasoniccenter.org>
To: root@dustpuppy.lakemasoniccenter.org
Subject: Fail event on /dev/md6:dustpuppy.lakemasoniccenter.org

This is an automatically generated mail message from mdadm
running on dustpuppy.lakemasoniccenter.org

A Fail event had been detected on md device /dev/md6.

It could be related to component device /dev/sdb11.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid6] [raid5] [raid4] 
md3 : active raid5 sdd2[3] sdc2[2] sdb2[4](F) sda2[5](F) hda3[0]
      315628800 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md2 : active raid5 sdd3[3] sdc3[2] sdb3[1] sda3[4](S) hda5[0]
      97522176 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md9 : active raid5 sdd5[3] sdc5[2] sdb5[1] sda5[4](S) hda6[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md8 : active raid5 sdd6[3] sdc6[2] sdb6[4](F) sda6[5](F) hda7[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md0 : active raid5 sdd7[3] sdc7[2] sdb7[4](F) sda7[5](F) hda8[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md4 : active raid5 sdd8[3] sdc8[2] sdb8[4](F) sda8[5](F) hda9[0]
      4192512 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md7 : active raid5 sdd10[3] sdc10[2] sdb10[4](F) sda10[5](F) hda11[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md6 : active raid5 sdd11[3] sdc11[2] sdb11[4](F) sda11[5](F) hda12[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md1 : active raid5 sdd12[3] sdc12[2] sdb12[4](F) sda12[5](F) hda13[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md12 : active raid5 sdd13[3] sdc13[2] sdb13[4](F) sda13[5](F) hda14[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md10 : active raid5 sdd14[3] sdc14[2] sdb14[1] sda14[4](S) hda15[0]
      1035264 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md11 : active raid5 sdd15[3] sdc15[2] sdb15[4](F) sda15[5](F) hda16[0]
      529152 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md5 : active raid5 sdd9[3] sdc9[2] sdb9[4](F) sda9[5](F) hda10[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

unused devices: <none>
Return-Path: <root@dustpuppy.lakemasoniccenter.org>
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
	dustpuppy.lakemasoniccenter.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.6 required=5.0 tests=ALL_TRUSTED,AWL autolearn=ham
	version=3.2.5
Received: from dustpuppy.lakemasoniccenter.org (localhost.localdomain [127.0.0.1])
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8) with ESMTP id p414QFq7026874
	for <root@dustpuppy.lakemasoniccenter.org>; Sat, 30 Apr 2011 23:26:15 -0500
Received: (from root@localhost)
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8/Submit) id p414QFvQ026872;
	Sat, 30 Apr 2011 23:26:15 -0500
Date: Sat, 30 Apr 2011 23:26:15 -0500
Message-Id: <201105010426.p414QFvQ026872@dustpuppy.lakemasoniccenter.org>
From: mdadm monitoring <root@dustpuppy.lakemasoniccenter.org>
To: root@dustpuppy.lakemasoniccenter.org
Subject: Fail event on /dev/md7:dustpuppy.lakemasoniccenter.org

This is an automatically generated mail message from mdadm
running on dustpuppy.lakemasoniccenter.org

A Fail event had been detected on md device /dev/md7.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid6] [raid5] [raid4] 
md3 : active raid5 sdd2[3] sdc2[2] sdb2[4](F) sda2[5](F) hda3[0]
      315628800 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md2 : active raid5 sdd3[3] sdc3[2] sdb3[1] sda3[4](S) hda5[0]
      97522176 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md9 : active raid5 sdd5[3] sdc5[2] sdb5[1] sda5[4](S) hda6[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md8 : active raid5 sdd6[3] sdc6[2] sdb6[4](F) sda6[5](F) hda7[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md0 : active raid5 sdd7[3] sdc7[2] sdb7[4](F) sda7[5](F) hda8[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md4 : active raid5 sdd8[3] sdc8[2] sdb8[4](F) sda8[5](F) hda9[0]
      4192512 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md7 : active raid5 sdd10[3] sdc10[2] sdb10[4](F) sda10[5](F) hda11[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md6 : active raid5 sdd11[3] sdc11[2] sdb11[4](F) sda11[5](F) hda12[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md1 : active raid5 sdd12[3] sdc12[2] sdb12[4](F) sda12[5](F) hda13[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md12 : active raid5 sdd13[3] sdc13[2] sdb13[4](F) sda13[5](F) hda14[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md10 : active raid5 sdd14[3] sdc14[2] sdb14[1] sda14[4](S) hda15[0]
      1035264 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md11 : active raid5 sdd15[3] sdc15[2] sdb15[4](F) sda15[5](F) hda16[0]
      529152 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md5 : active raid5 sdd9[3] sdc9[2] sdb9[4](F) sda9[5](F) hda10[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

unused devices: <none>
Return-Path: <root@dustpuppy.lakemasoniccenter.org>
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
	dustpuppy.lakemasoniccenter.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.6 required=5.0 tests=ALL_TRUSTED,AWL autolearn=ham
	version=3.2.5
Received: from dustpuppy.lakemasoniccenter.org (localhost.localdomain [127.0.0.1])
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8) with ESMTP id p414hCft028140
	for <root@dustpuppy.lakemasoniccenter.org>; Sat, 30 Apr 2011 23:43:12 -0500
Received: (from root@localhost)
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8/Submit) id p414PRW0026860;
	Sat, 30 Apr 2011 23:25:27 -0500
Date: Sat, 30 Apr 2011 23:25:27 -0500
Message-Id: <201105010425.p414PRW0026860@dustpuppy.lakemasoniccenter.org>
From: mdadm monitoring <root@dustpuppy.lakemasoniccenter.org>
To: root@dustpuppy.lakemasoniccenter.org
Subject: Fail event on /dev/md8:dustpuppy.lakemasoniccenter.org

This is an automatically generated mail message from mdadm
running on dustpuppy.lakemasoniccenter.org

A Fail event had been detected on md device /dev/md8.

It could be related to component device /dev/sdb6.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid6] [raid5] [raid4] 
md3 : active raid5 sdd2[3] sdc2[2] sdb2[4](F) sda2[5](F) hda3[0]
      315628800 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md2 : active raid5 sdd3[3] sdc3[2] sdb3[1] sda3[4](S) hda5[0]
      97522176 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md9 : active raid5 sdd5[3] sdc5[2] sdb5[1] sda5[4](S) hda6[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md8 : active raid5 sdd6[3] sdc6[2] sdb6[4](F) sda6[5](F) hda7[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md0 : active raid5 sdd7[3] sdc7[2] sdb7[4](F) sda7[5](F) hda8[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md4 : active raid5 sdd8[3] sdc8[2] sdb8[4](F) sda8[5](F) hda9[0]
      4192512 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md7 : active raid5 sdd10[3] sdc10[2] sdb10[4](F) sda10[5](F) hda11[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md6 : active raid5 sdd11[3] sdc11[2] sdb11[1] sda11[4](S) hda12[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md1 : active raid5 sdd12[3] sdc12[2] sdb12[4](F) sda12[5](F) hda13[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md12 : active raid5 sdd13[3] sdc13[2] sdb13[4](F) sda13[5](F) hda14[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md10 : active raid5 sdd14[3] sdc14[2] sdb14[1] sda14[4](S) hda15[0]
      1035264 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md11 : active raid5 sdd15[3] sdc15[2] sdb15[1] sda15[4](S) hda16[0]
      529152 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

md5 : active raid5 sdd9[3] sdc9[2] sdb9[4](F) sda9[5](F) hda10[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

unused devices: <none>
Return-Path: <root@dustpuppy.lakemasoniccenter.org>
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
	dustpuppy.lakemasoniccenter.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.6 required=5.0 tests=ALL_TRUSTED,AWL autolearn=ham
	version=3.2.5
Received: from dustpuppy.lakemasoniccenter.org (localhost.localdomain [127.0.0.1])
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8) with ESMTP id p41503WX029027
	for <root@dustpuppy.lakemasoniccenter.org>; Sun, 1 May 2011 00:00:03 -0500
Received: (from root@localhost)
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8/Submit) id p415032L029018;
	Sun, 1 May 2011 00:00:03 -0500
Date: Sun, 1 May 2011 00:00:03 -0500
Message-Id: <201105010500.p415032L029018@dustpuppy.lakemasoniccenter.org>
From: mdadm monitoring <root@dustpuppy.lakemasoniccenter.org>
To: root@dustpuppy.lakemasoniccenter.org
Subject: Fail event on /dev/md9:dustpuppy.lakemasoniccenter.org

This is an automatically generated mail message from mdadm
running on dustpuppy.lakemasoniccenter.org

A Fail event had been detected on md device /dev/md9.

It could be related to component device /dev/sdb5.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid6] [raid5] [raid4] 
md3 : active raid5 sdd2[3] sdc2[2] sdb2[4](F) sda2[5](F) hda3[0]
      315628800 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md2 : active raid5 sdd3[3] sdc3[2] sdb3[4](F) sda3[5](F) hda5[0]
      97522176 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md9 : active raid5 sdd5[3] sdc5[2] sdb5[4](F) sda5[5](F) hda6[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md8 : active raid5 sdd6[3] sdc6[2] sdb6[4](F) sda6[5](F) hda7[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md0 : active raid5 sdd7[3] sdc7[2] sdb7[4](F) sda7[5](F) hda8[0]
      8385024 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md4 : active raid5 sdd8[3] sdc8[2] sdb8[4](F) sda8[5](F) hda9[0]
      4192512 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md7 : active raid5 sdd10[3] sdc10[2] sdb10[4](F) sda10[5](F) hda11[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md6 : active raid5 sdd11[3] sdc11[2] sdb11[4](F) sda11[5](F) hda12[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md1 : active raid5 sdd12[3] sdc12[2] sdb12[4](F) sda12[5](F) hda13[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md12 : active raid5 sdd13[3] sdc13[2] sdb13[4](F) sda13[5](F) hda14[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md10 : active raid5 sdd14[3] sdc14[2] sdb14[4](F) sda14[5](F) hda15[0]
      1035264 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md11 : active raid5 sdd15[3] sdc15[2] sdb15[4](F) sda15[5](F) hda16[0]
      529152 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

md5 : active raid5 sdd9[3] sdc9[2] sdb9[4](F) sda9[5](F) hda10[0]
      2095872 blocks level 5, 256k chunk, algorithm 2 [4/3] [U_UU]

unused devices: <none>
Return-Path: <root@dustpuppy.lakemasoniccenter.org>
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
	dustpuppy.lakemasoniccenter.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.4 required=5.0 tests=ALL_TRUSTED,AWL autolearn=ham
	version=3.2.5
Received: from dustpuppy.lakemasoniccenter.org (localhost.localdomain [127.0.0.1])
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8) with ESMTP id p4192kgr013454
	for <root@dustpuppy.lakemasoniccenter.org>; Sun, 1 May 2011 04:02:47 -0500
Received: (from root@localhost)
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8/Submit) id p4192TgB013202;
	Sun, 1 May 2011 04:02:29 -0500
Date: Sun, 1 May 2011 04:02:29 -0500
Message-Id: <201105010902.p4192TgB013202@dustpuppy.lakemasoniccenter.org>
To: root@dustpuppy.lakemasoniccenter.org
From: logwatch@lakemasoniccenter.org
Subject: Logwatch for dustpuppy.lakemasoniccenter.org (Linux)
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset="iso-8859-1"

 ################### Logwatch 7.3 (03/24/06) #################### 
        Processing Initiated: Sun May  1 04:02:28 2011
        Date Range Processed: yesterday
                              ( 2011-Apr-30 )
                              Period is day.
      Detail Level of Output: 0
              Type of Output: unformatted
           Logfiles for Host: dustpuppy.lakemasoniccenter.org
  ################################################################## 

 --------------------- Dovecot Begin ------------------------ 

 Dovecot disconnects:
    Inactivity: 4 Time(s)
    Logged out: 367 Time(s)
    in IDLE: 5 Time(s)
    no reason: 4 Time(s)

 **Unmatched Entries**
    dovecot: IMAP(imbrius): Connection closed: 3 Time(s)
    dovecot: IMAP(mom): Connection closed: 3 Time(s)

 ---------------------- Dovecot End ------------------------- 

 --------------------- ftpd-xferlog Begin ------------------------ 

 TOTAL KB OUT: 11663KB (11MB)

 ---------------------- ftpd-xferlog End ------------------------- 

 --------------------- httpd Begin ------------------------ 

 Connection attempts using mod_proxy:
    207.30.25.40 -> 207.30.25.40:7000: 1 Time(s)

 Requests with error response codes
    400 Bad Request
       HTTP/1.1: 2 Time(s)
    403 Forbidden
       /phpMyAdmin/scripts/setup.php: 2 Time(s)
       /phpmyadmin/scripts/setup.php: 2 Time(s)
    404 Not Found
       /favicon.ico: 5 Time(s)
       /jargon/anthropomorphization.html: 1 Time(s)
       /sites/lakemasoniccenter.org/files/Joe+Harkers+Emblem(1).jpg: 1 Time(s)
       /tiles/tiles/0/0.png: 2 Time(s)
       /tiles/tiles/0/1.png: 2 Time(s)
       /tiles/tiles/1/0.png: 2 Time(s)
       /tiles/tiles/1/1.png: 2 Time(s)
       /tiles/tiles/1/3.png: 2 Time(s)
       /tiles/tiles/2/0.png: 2 Time(s)
       /tiles/tiles/2/1/0/3.png: 2 Time(s)
       /tiles/tiles/2/1/1/1/3/1/0/0.png: 2 Time(s)
       /tiles/tiles/2/1/1/2.png: 2 Time(s)
       /tiles/tiles/2/2.png: 2 Time(s)
       /tiles/tiles/2/3.png: 2 Time(s)
       /tiles/tiles/3/0/0/0/2/3/3/2.png: 2 Time(s)
       /tiles/tiles/3/0/0/3.png: 2 Time(s)
       /tiles/tiles/3/1.png: 2 Time(s)
       /tiles/tiles/3/2.png: 2 Time(s)
       /tiles/tiles/3/3.png: 2 Time(s)

 ---------------------- httpd End ------------------------- 

 --------------------- Init Begin ------------------------ 

 Re-execs of init: 1 times

 ---------------------- Init End ------------------------- 

 --------------------- Kernel Begin ------------------------ 

 WARNING:  Kernel Errors Present
             res 51/40:00:8d:66:83/00:00:11:00:00/10 Emask 0x9 (media error) ...:  6 Time(s)
             res 51/40:00:9c:b6:5e/00:00:11:00:00/10 Emask 0x9 (media error) ...:  12 Time(s)
             res d0/d0:d0:d0:d0:d0/00:00:11:00:00/d0 Emask 0x12 (ATA bus error) ...:  1 Time(s)
             res d0/d0:d0:d0:d0:d0/d0:d0:d0:d0:d0/d0 Emask 0x12 (ATA bus error) ...:  11 Time(s)
             res d0/d0:d0:d0:d0:d0/d8:d8:d8:d8:d8/d0 Emask 0x12 (ATA bus error) ...:  1 Time(s)
             res d0/d0:d0:d0:d0:d0/ff:ff:ff:ff:ff/d0 Emask 0x12 (ATA bus error) ...:  1 Time(s)
             res d8/d8:d8:d8:d8:d8/d8:d8:d8:d8:d8/d8 Emask 0x12 (ATA bus error) ...:  5 Time(s)
             res ff/ff:ff:ff:d8:d8/d8:d8:d8:d8:d8/d8 Emask 0x12 (ATA bus error) ...:  1 Time(s)
             res ff/ff:ff:ff:ff:ff/ff:ff:ff:ff:ff/ff Emask 0x12 (ATA bus error) ...:  14 Time(s)
        Add. Sense: Unrecovered read error - auto reallocat ...:  3 Time(s)
    ata3: SError: { PHYRdyChg Com ...:  1 Time(s)
    ata3: SError: { PHYRdyChg } ...:  40 Time(s)
    ata4.00: error: { ICRC UNC IDNF ...:  34 Time(s)
    ata4.00: error: { UNC } ...:  18 Time(s)
    ata4: SError: { PHYRdyChg 10B ...:  8 Time(s)
    ata4: SError: { PHYRdyChg } ...:  53 Time(s)
    end_request: I/O error, dev sdb, sector ...:  24 Time(s)
    raid5:md12: read error not correctable  ...:  3 Time(s)
    sd 3:0:0:0: SCSI error: return code = 0 ...:  24 Time(s)
    sdb: Current [descriptor]: sense key: Medium Error ...:  3 Time(s)

 ---------------------- Kernel End ------------------------- 

 --------------------- Raid Begin ------------------------ 

 Apr 30 23:25:27 raid5: Disk failure on sdb13, disabling device. Operation continuing on 3 devices 
 Apr 30 23:25:27 raid5: Disk failure on sdb6, disabling device. Operation continuing on 3 devices 
 Apr 30 23:25:27 raid5: Disk failure on sdb2, disabling device. Operation continuing on 3 devices 
 Apr 30 23:25:27 raid5: Disk failure on sda13, disabling device. Operation continuing on 3 devices 
 Apr 30 23:25:27 raid5: Disk failure on sdb7, disabling device. Operation continuing on 3 devices 
 Apr 30 23:25:27 raid5: Disk failure on sdb12, disabling device. Operation continuing on 3 devices 
 Apr 30 23:25:27 raid5: Disk failure on sda7, disabling device. Operation continuing on 3 devices 
 Apr 30 23:25:27 raid5: Disk failure on sda12, disabling device. Operation continuing on 3 devices 
 Apr 30 23:25:28 raid5: Disk failure on sda6, disabling device. Operation continuing on 3 devices 
 Apr 30 23:25:28 raid5: Disk failure on sda2, disabling device. Operation continuing on 3 devices 
 Apr 30 23:25:28 raid5: Disk failure on sdb9, disabling device. Operation continuing on 3 devices 
 Apr 30 23:25:28 raid5: Disk failure on sda9, disabling device. Operation continuing on 3 devices 
 Apr 30 23:25:28 raid5: Disk failure on sdb8, disabling device. Operation continuing on 3 devices 
 Apr 30 23:25:28 raid5: Disk failure on sda8, disabling device. Operation continuing on 3 devices 
 Apr 30 23:25:28 raid5: Disk failure on sdb10, disabling device. Operation continuing on 3 devices 
 Apr 30 23:25:28 raid5: Disk failure on sda10, disabling device. Operation continuing on 3 devices 
 Apr 30 23:25:31 raid5: Disk failure on sdb15, disabling device. Operation continuing on 3 devices 
 Apr 30 23:25:31 raid5: Disk failure on sda15, disabling device. Operation continuing on 3 devices 
 Apr 30 23:26:14 raid5: Disk failure on sdb11, disabling device. Operation continuing on 3 devices 
 Apr 30 23:26:14 raid5: Disk failure on sda11, disabling device. Operation continuing on 3 devices 
 Apr 30 23:30:07 raid5: Disk failure on sdb14, disabling device. Operation continuing on 3 devices 
 Apr 30 23:30:07 raid5: Disk failure on sda14, disabling device. Operation continuing on 3 devices 
 Apr 30 23:32:34 raid5: Disk failure on sdb3, disabling device. Operation continuing on 3 devices 
 Apr 30 23:32:34 raid5: Disk failure on sda3, disabling device. Operation continuing on 3 devices 

 ---------------------- Raid End ------------------------- 

 --------------------- samba Begin ------------------------ 

 **Unmatched Entries**
 printing/print_cups.c:cups_connect(69)  Unable to connect to CUPS server localhost:631 - Connection refused : 118 Time(s)

 ---------------------- samba End ------------------------- 

 --------------------- Smartd Begin ------------------------ 

 Currently unreadable (pending) sectors detected:
 	/dev/sdb - 1 Time(s)
 	1 unreadable sectors detected

 Offline uncorrectable sectors detected:
 	/dev/sdb - 1 Time(s)
 	1 offline uncorrectable sectors detected

 Warnings:
 	Warning via mail to root: successful  - 5 Time(s)
 	Sending warning via mail to root ...  - 5 Time(s)

 **Unmatched Entries**
 Device: /dev/sdb, failed to read SMART Attribute Data 

 ---------------------- Smartd End ------------------------- 

 --------------------- SSHD Begin ------------------------ 

 Refused incoming connections:
       ::ffff:113.31.27.182 (::ffff:113.31.27.182): 1 Time(s)
       ::ffff:83.149.32.228 (::ffff:83.149.32.228): 2 Time(s)

 ---------------------- SSHD End ------------------------- 

 --------------------- XNTPD Begin ------------------------ 

 Total synchronizations 3 (hosts: 2)

 ---------------------- XNTPD End ------------------------- 

 --------------------- Disk Space Begin ------------------------ 

 Filesystem            Size  Used Avail Use% Mounted on
 /dev/md5              2.0G  839M  1.1G  45% /
 /dev/md9              7.8G  374M  7.0G   5% /opt
 /dev/md8              7.8G  3.2G  4.2G  44% /usr
 /dev/md11             501M   64M  412M  14% /usr/mailman
 /dev/md10             980M  804M  125M  87% /usr/local
 /dev/md7              2.0G   59M  1.8G   4% /tmp
 /dev/md4              3.9G  1.9G  1.8G  52% /var
 /dev/md12             2.0G  140M  1.7G   8% /var/log
 /dev/md6              2.0G   83M  1.8G   5% /var/cache
 /dev/md3              292G  228G   50G  83% /export/home
 /dev/md2               91G   16G   70G  19% /var/ftp
 /dev/md0              7.8G  351M  7.1G   5% /var/lib/mysql
 /dev/hda1             122M   24M   92M  21% /boot
 /dev/md1              2.0G   96M  1.8G   6% /var/spool
 /dev/sde1             1.8T  1.4T  409G  77% /media/expansion

 ---------------------- Disk Space End ------------------------- 

 ###################### Logwatch End ######################### 

Return-Path: <root@SavageChicken.lakemasoniccenter.org>
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
	dustpuppy.lakemasoniccenter.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.3 required=5.0 tests=ALL_TRUSTED,AWL autolearn=ham
	version=3.2.5
Received: from SavageChicken.lakemasoniccenter.org (SavageChicken.lakemasoniccenter.org [192.168.0.228])
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8) with ESMTP id p41926qT013037
	for <root@dustpuppy.lakemasoniccenter.org>; Sun, 1 May 2011 04:02:06 -0500
Received: from SavageChicken.lakemasoniccenter.org (Javacraft [127.0.0.1])
	by SavageChicken.lakemasoniccenter.org (8.13.8/8.13.8) with ESMTP id p41925tw028691
	for <root@SavageChicken.lakemasoniccenter.org>; Sun, 1 May 2011 04:02:05 -0500
Received: (from root@localhost)
	by SavageChicken.lakemasoniccenter.org (8.13.8/8.13.8/Submit) id p41925nt028647;
	Sun, 1 May 2011 04:02:05 -0500
Date: Sun, 1 May 2011 04:02:05 -0500
Message-Id: <201105010902.p41925nt028647@SavageChicken.lakemasoniccenter.org>
To: root@SavageChicken.lakemasoniccenter.org
From: logwatch@SavageChicken.lakemasoniccenter.org
Subject: Logwatch for savagechicken (Linux)
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset="iso-8859-1"

 ################### Logwatch 7.3 (03/24/06) #################### 
        Processing Initiated: Sun May  1 04:02:05 2011
        Date Range Processed: yesterday
                              ( 2011-Apr-30 )
                              Period is day.
      Detail Level of Output: 0
              Type of Output: unformatted
           Logfiles for Host: savagechicken
  ################################################################## 

 --------------------- SSHD Begin ------------------------ 

 Users logging in through sshd:
    imbrius:
       192.168.0.219: 1 time

 ---------------------- SSHD End ------------------------- 

 --------------------- Sudo (secure-log) Begin ------------------------ 

 ==============================================================================
 imbrius => root
 ------------------------------------------------------------------------------
 /usr/bin/yum reinstall less
 /usr/bin/yum search lesspipe

 ---------------------- Sudo (secure-log) End ------------------------- 

 --------------------- yum Begin ------------------------ 

 Packages Installed:
    less-436-7.el5.x86_64

 ---------------------- yum End ------------------------- 

 --------------------- Disk Space Begin ------------------------ 

 Filesystem            Size  Used Avail Use% Mounted on
 /dev/hda2              16G  1.1G   14G   7% /
 /dev/hda5              20G  3.7G   16G  20% /usr
 /dev/hdb3              45G  5.6G   37G  14% /opt
 /dev/hdb2              27G  802M   25G   4% /var
 /dev/hda1             122M   26M   90M  23% /boot

 ---------------------- Disk Space End ------------------------- 

 ###################### Logwatch End ######################### 

Return-Path: <root@dustpuppy.lakemasoniccenter.org>
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
	dustpuppy.lakemasoniccenter.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.6 required=5.0 tests=ALL_TRUSTED,AWL autolearn=ham
	version=3.2.5
Received: from dustpuppy.lakemasoniccenter.org (localhost.localdomain [127.0.0.1])
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8) with ESMTP id p414DngX026003
	for <root@dustpuppy.lakemasoniccenter.org>; Sat, 30 Apr 2011 23:13:49 -0500
Received: (from root@localhost)
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8/Submit) id p414Dnen026002
	for root; Sat, 30 Apr 2011 23:13:49 -0500
Date: Sat, 30 Apr 2011 23:13:49 -0500
From: root <root@dustpuppy.lakemasoniccenter.org>
Message-Id: <201105010413.p414Dnen026002@dustpuppy.lakemasoniccenter.org>
To: root@dustpuppy.lakemasoniccenter.org
Subject: SMART error (CurrentPendingSector) detected on host: dustpuppy.lakemasoniccenter.org

This email was generated by the smartd daemon running on:

   host name: dustpuppy.lakemasoniccenter.org
  DNS domain: lakemasoniccenter.org
  NIS domain: (none)

The following warning/error was logged by the smartd daemon:

Device: /dev/sdb, 1 Currently unreadable (pending) sectors

For details see host's SYSLOG (default: /var/log/messages).

You can also use the smartctl utility for further investigation.
No additional email messages about this problem will be sent.
Return-Path: <root@dustpuppy.lakemasoniccenter.org>
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
	dustpuppy.lakemasoniccenter.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.1 required=5.0 tests=ALL_TRUSTED,AWL,
	CONTAINS_HEALTH autolearn=no version=3.2.5
Received: from dustpuppy.lakemasoniccenter.org (localhost.localdomain [127.0.0.1])
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8) with ESMTP id p414hnqm028196
	for <root@dustpuppy.lakemasoniccenter.org>; Sat, 30 Apr 2011 23:43:49 -0500
Received: (from root@localhost)
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8/Submit) id p414hnta028195
	for root; Sat, 30 Apr 2011 23:43:49 -0500
Date: Sat, 30 Apr 2011 23:43:49 -0500
From: root <root@dustpuppy.lakemasoniccenter.org>
Message-Id: <201105010443.p414hnta028195@dustpuppy.lakemasoniccenter.org>
To: root@dustpuppy.lakemasoniccenter.org
Subject: SMART error (FailedHealthCheck) detected on host: dustpuppy.lakemasoniccenter.org

This email was generated by the smartd daemon running on:

   host name: dustpuppy.lakemasoniccenter.org
  DNS domain: lakemasoniccenter.org
  NIS domain: (none)

The following warning/error was logged by the smartd daemon:

Device: /dev/sdb, not capable of SMART self-check

For details see host's SYSLOG (default: /var/log/messages).

You can also use the smartctl utility for further investigation.
No additional email messages about this problem will be sent.
Return-Path: <root@dustpuppy.lakemasoniccenter.org>
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
	dustpuppy.lakemasoniccenter.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.6 required=5.0 tests=ALL_TRUSTED,AWL autolearn=ham
	version=3.2.5
Received: from dustpuppy.lakemasoniccenter.org (localhost.localdomain [127.0.0.1])
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8) with ESMTP id p414hmpZ028188
	for <root@dustpuppy.lakemasoniccenter.org>; Sat, 30 Apr 2011 23:43:48 -0500
Received: (from root@localhost)
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8/Submit) id p414hmqK028187
	for root; Sat, 30 Apr 2011 23:43:48 -0500
Date: Sat, 30 Apr 2011 23:43:48 -0500
From: root <root@dustpuppy.lakemasoniccenter.org>
Message-Id: <201105010443.p414hmqK028187@dustpuppy.lakemasoniccenter.org>
To: root@dustpuppy.lakemasoniccenter.org
Subject: SMART error (FailedOpenDevice) detected on host: dustpuppy.lakemasoniccenter.org

This email was generated by the smartd daemon running on:

   host name: dustpuppy.lakemasoniccenter.org
  DNS domain: lakemasoniccenter.org
  NIS domain: (none)

The following warning/error was logged by the smartd daemon:

Device: /dev/sda, unable to open device

For details see host's SYSLOG (default: /var/log/messages).

You can also use the smartctl utility for further investigation.
No additional email messages about this problem will be sent.
Return-Path: <root@dustpuppy.lakemasoniccenter.org>
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
	dustpuppy.lakemasoniccenter.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.6 required=5.0 tests=ALL_TRUSTED,AWL autolearn=ham
	version=3.2.5
Received: from dustpuppy.lakemasoniccenter.org (localhost.localdomain [127.0.0.1])
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8) with ESMTP id p414hnmm028204
	for <root@dustpuppy.lakemasoniccenter.org>; Sat, 30 Apr 2011 23:43:49 -0500
Received: (from root@localhost)
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8/Submit) id p414hnAe028203
	for root; Sat, 30 Apr 2011 23:43:49 -0500
Date: Sat, 30 Apr 2011 23:43:49 -0500
From: root <root@dustpuppy.lakemasoniccenter.org>
Message-Id: <201105010443.p414hnAe028203@dustpuppy.lakemasoniccenter.org>
To: root@dustpuppy.lakemasoniccenter.org
Subject: SMART error (FailedReadSmartData) detected on host: dustpuppy.lakemasoniccenter.org

This email was generated by the smartd daemon running on:

   host name: dustpuppy.lakemasoniccenter.org
  DNS domain: lakemasoniccenter.org
  NIS domain: (none)

The following warning/error was logged by the smartd daemon:

Device: /dev/sdb, failed to read SMART Attribute Data

For details see host's SYSLOG (default: /var/log/messages).

You can also use the smartctl utility for further investigation.
No additional email messages about this problem will be sent.
Return-Path: <root@dustpuppy.lakemasoniccenter.org>
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
	dustpuppy.lakemasoniccenter.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.6 required=5.0 tests=ALL_TRUSTED,AWL autolearn=ham
	version=3.2.5
Received: from dustpuppy.lakemasoniccenter.org (localhost.localdomain [127.0.0.1])
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8) with ESMTP id p414DnvK026009
	for <root@dustpuppy.lakemasoniccenter.org>; Sat, 30 Apr 2011 23:13:49 -0500
Received: (from root@localhost)
	by dustpuppy.lakemasoniccenter.org (8.13.8/8.13.8/Submit) id p414DnOU026007
	for root; Sat, 30 Apr 2011 23:13:49 -0500
Date: Sat, 30 Apr 2011 23:13:49 -0500
From: root <root@dustpuppy.lakemasoniccenter.org>
Message-Id: <201105010413.p414DnOU026007@dustpuppy.lakemasoniccenter.org>
To: root@dustpuppy.lakemasoniccenter.org
Subject: SMART error (OfflineUncorrectableSector) detected on host: dustpuppy.lakemasoniccenter.org

This email was generated by the smartd daemon running on:

   host name: dustpuppy.lakemasoniccenter.org
  DNS domain: lakemasoniccenter.org
  NIS domain: (none)

The following warning/error was logged by the smartd daemon:

Device: /dev/sdb, 1 Offline uncorrectable sectors

For details see host's SYSLOG (default: /var/log/messages).

You can also use the smartctl utility for further investigation.
No additional email messages about this problem will be sent.

Thankfully, I had a hot spare in the box connected to the new SATA controller.  Otherwise, the box would be down right now.  I’d be willing to bet that my drives are fine and that my SATA controller is dying.  ’cause it’d be odd to have two drives fail in the same way at the same time and they’re both connected to the same SATA controller.

So the plan is to shut the box down at some point and move the drives to the new SATA controller.

Given the recent slow failure of each component in the box, it seems like these may be the final days of dustpuppy, the AMD AthlonXP 2000+ machine that’s served me faithfully since 2002.

Advertisements

, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

  1. #1 by Chadwick on May 1, 2011 - 9:38 AM

    This is a sad-making event. But it has served you well over the years. So there’s also happy memories to look back on during the funeral.

    • #2 by Joshua on May 1, 2011 - 9:44 AM

      Yeah and I don’t intend to decommission it until I have a line on replacing it. If it completely failed today I could bring the websites up on SavageChicken but it doesn’t have the throughput for mail routing.

      I do have HomelyGosling that would handle the load easily. Maybe now that we have brand new 20 amp circuits I could look at getting that set up.

  2. #3 by Joshua on May 1, 2011 - 1:31 PM

    I got the drives moved over to the off-board controller. When I went to power it back up, the PSU went into bug-zapper mode. Made pretty blue sparks and nasty smells. So I had to drop my spare PSU in and now it’s working. The software RAID-5 is doing exactly what it was designed for and the machine booted just fine. I did have to manually rebuild the arrays, though. But meh. No data loss and that’s the key.

  3. #5 by Joshua on May 3, 2011 - 5:30 PM

    Nope. Dustpuppy is really dead. I ordered a prebuilt rackmount server from TigerDirect today to replace it. Two things: one – how do I memorialize dustpuppy and two – what do I name the new one. Since I’ve been naming machines after starships, I was thinking either Kitesfear (also a bird!), Petey or PostDatedCheckLoan.

    The comic strip that inspired the name “dustpuppy” is mostly defunct now. Shlock Mercenary, the one that gives us Kitesfear, Petey, and PostDatedCheckLoan, is still going.

    Any other ideas?

    • #6 by Joshua on May 3, 2011 - 5:32 PM

      For reference: the Kitesfear got blown up and Petey (full name PostDatedCheckLoan) went crazy and then self-destructed (but his secret wormhole copy was fine, if a little… off).

      • #7 by Chadwick on May 4, 2011 - 7:24 PM

        DOGGY!

        • #8 by Joshua on May 4, 2011 - 7:31 PM

          I was gonna use Florence but it’s already used for one of my work passwords 😛

        • #9 by Joshua on May 4, 2011 - 7:32 PM

          I could always just use “doggy”

          • #10 by Chadwick on May 4, 2011 - 8:16 PM

            It was just the first thing that came to mind while combining “Josh” and “Webcomics”.

            Hey, what’s the name of the crazy robot who helps disassemble robots?

          • #11 by Joshua on May 5, 2011 - 3:59 PM

            Hmm… I’ll have to look him up again. Though Sawtooth might be okay too. He’s the super-nice “landscaping” robot that looks like a giant ladybug.

          • #12 by Chadwick on May 5, 2011 - 7:48 PM

            EDGE!

          • #13 by Chadwick on May 5, 2011 - 8:05 PM

            And Blunt. BluntEdge?

  4. #14 by Joshua on May 3, 2011 - 6:59 PM

    The new box is a CybertronPC Quantum QBA2420. Here are the specs. I also bought one 4-drive SATA hotswap canister and one 3-drive SATA hotswap canister. One for the 3-disk RAID-5 that will hold the data and one for the 2-disk RAID-1 that will hold the OS and system software.

  5. #15 by Joshua on May 7, 2011 - 1:18 PM

    The new machine will be here on Tuesday. Also, I’m debating installing StumbleUpon and Morning Coffee on Horde instead of UglyDucking. Since Firefox 4 doesn’t run (correctly) on SPARC and StumbleUpon only half works, the list of tasks for which UglyDuckling is fit seems to be decreasing lately.

    • #16 by Chadwick on May 8, 2011 - 11:49 AM

      Firefox 4 doesn’t run correctly on SPARC? Any relation to the script errors and overflows that come with combining StumbleUpon 3.81 + FF4?

      • #17 by Joshua on May 8, 2011 - 2:12 PM

        Dunno but on SPARC, it locks up when loading pages. Pretty much any page that loads that ISN’T directly after the home page.

  6. #18 by Joshua on May 7, 2011 - 1:21 PM

    The good news is that I’ll be able to reuse the OS install DVD from SavageChicken since CentOS 6.0 will not be released for a long time yet.

  7. #19 by Joshua on May 9, 2011 - 8:39 AM

    The new machine will be here tomorrow. And I’ve decided on the name “kitesfear.” Here’s the new logo: http://i.imgur.com/YNKZ8.png

  8. #20 by Joshua on May 9, 2011 - 10:24 AM

     _     _ _____ _______ _______ _______ _______ _______ _______  ______
     |____/    |      |    |______ |______ |______ |______ |_____| |_____/
     |    \_ __|__    |    |______ ______| |       |______ |     | |    \_
    
    • #21 by Chadwick on May 9, 2011 - 6:40 PM

      So, tell us a bit about Kitesfear, for those of us who are unfamiliar with the comic, and far too lazy to look into this; bonus points for explaining how this reflects on the new system.

      • #22 by Joshua on May 10, 2011 - 8:26 PM

        I’ll answer as soon as I finish installing the OS.

      • #23 by Joshua on May 10, 2011 - 8:35 PM

        The Kitesfear was a Dragon-class cruiser owned by the mercenary team Tagon’s Toughs and captained by Kaff Tagon.

        The deck plans for her are here. Here is a good 3D rendering of her.

        As you can see, even though the ship was ostensibly named after a bird (the kite), you can also see that the webcomic author has a very punny sense of humor (in that that ship is a giant kite with a sphere in the middle).

        The forum thread discussing her particulars is here.

      • #24 by Joshua on May 10, 2011 - 8:45 PM

        As far as what happened to her and how it relates to the box, the Kitesfear exploded in a nice fireball after the Partnership Collective (a hivemind of snake-lawyer clones) planted bombs on it.

        If how it’s starting out is any indication, it’s not likely to blow up (the thing is SOLID!) but is likely to be rather picky about how the disks are labeled.

  9. #25 by Joshua on May 10, 2011 - 7:54 AM

    					       *
    				     *		     *
    			      *    		     	    
    			*	       			  *
    		  *			     * * *		
          	   *				   *	   *		*
          *     	 			  *	    *
    *					  *         *                *
          *					  *  	    *
    	   *				   * 	   *	        *
    	          *   			     * * *	  
    			*                                 *
    			      *      
                             	     *         	     *
                             	               *
     _     _ _____ _______ _______ _______ _______ _______ _______  ______
     |____/    |      |    |______ |______ |______ |______ |_____| |_____/
     |    \_ __|__    |    |______ ______| |       |______ |     | |    \_
    
  10. #26 by Joshua on May 11, 2011 - 1:00 PM

  11. #27 by Joshua on May 15, 2011 - 8:00 PM

    I still wanna have a pizza party / funeral for dustpuppy one of these days.

    • #28 by Phillip on May 15, 2011 - 11:19 PM

      I’d be up for that. And I think I’ll bring along the scotch from Christmas yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: