Good little sysadmin

I would consider myself a fairly competent UNIX sysadmin.  I make my share of boneheaded mistakes (like mounting one disk in a RAID-1 array read/write while the vendor CEs are trying to work on it), but I’ve never done anything so stupid that it’s actually cost a company money.

As part of my daily sysadminning rituals, I read the e-mails that get sent to root@dustpuppy by the logwatch and chkrootkit cron jobs.  Logwatch is a tool that monitors the system logs for anything out of the ordinary and summarizes everything else.  Kind of like a very high-level overview of the system activity in the past 24 hours with selective low-level details.  If there’s anything that needs checking into, I ssh into the box and take care of it.  Most days, there’s nothing unusual:

 ################### Logwatch 7.3 (03/24/06) ####################
        Processing Initiated: Tue Mar 30 04:02:13 2010
        Date Range Processed: yesterday
                              ( 2010-Mar-29 )
                              Period is day.
      Detail Level of Output: 0
              Type of Output: unformatted
           Logfiles for Host: dustpuppy.lakemasoniccenter.org
  ################################################################## 

 --------------------- httpd Begin ------------------------ 

 Requests with error response codes
    403 Forbidden
       /Bethel6IOJD: 2 Time(s)
    404 Not Found
       /favicon.ico: 17 Time(s)
       /getinvolved: 1 Time(s)
       /linked/interim%20ministry.pdf: 1 Time(s)
       /ykjfmdgniyxljop.html: 1 Time(s)
    503 Service Unavailable
       *: 180 Time(s)
       /drupal/cron.php: 24 Time(s)
       http://firstlutheranelca.org: 96 Time(s)
       http://lakemasoniccenter.org/: 360 Time(s)

 ---------------------- httpd End ------------------------- 

 --------------------- pam_unix Begin ------------------------ 

 runuser:
    Unknown Entries:
       session closed for user news: 1 Time(s)
       session opened for user news by (uid=0): 1 Time(s)

 runuser-l:
    Unknown Entries:
       session closed for user news: 49 Time(s)
       session opened for user news by (uid=0): 49 Time(s)

 sshd:
    Authentication Failures:
       unknown (121.131.210.82): 16 Time(s)
    Invalid Users:
       Unknown Account: 16 Time(s)

 su:
    Sessions Opened:
       (uid=0) -> imbrius: 1440 Time(s)
       (uid=0) -> mom: 1440 Time(s)

 sudo:
    Unknown Entries:
       auth could not identify password for [imbrius]: 1 Time(s)
       conversation failed: 1 Time(s)

 ---------------------- pam_unix End ------------------------- 

 --------------------- samba Begin ------------------------ 

 **Unmatched Entries**
 printing/print_cups.c:cups_connect(69)  Unable to connect to CUPS server /var/run/cups/cups.sock:631 - Connection refused : 54 Time(s)

 ---------------------- samba End ------------------------- 

 --------------------- Connections (secure-log) Begin ------------------------ 

 **Unmatched Entries**
 webmin[3395]: Timeout of session for root
 webmin[3395]: Timeout of session for root
 webmin[24743]: Successful login as root from 192.168.0.192 

 ---------------------- Connections (secure-log) End ------------------------- 

 --------------------- SSHD Begin ------------------------ 

 **Unmatched Entries**
 pam_succeed_if(sshd:auth): error retrieving information about user TeamSpeak : 1 time(s)
 pam_succeed_if(sshd:auth): error retrieving information about user suzuki : 1 time(s)
 pam_succeed_if(sshd:auth): error retrieving information about user db2inst1 : 1 time(s)
 pam_succeed_if(sshd:auth): error retrieving information about user domin : 1 time(s)
 pam_succeed_if(sshd:auth): error retrieving information about user cisco : 1 time(s)
 pam_succeed_if(sshd:auth): error retrieving information about user plcmspip : 2 time(s)
 pam_succeed_if(sshd:auth): error retrieving information about user PlcmSpIp : 1 time(s)
 pam_succeed_if(sshd:auth): error retrieving information about user ts : 1 time(s)
 pam_succeed_if(sshd:auth): error retrieving information about user test : 4 time(s)
 pam_succeed_if(sshd:auth): error retrieving information about user dasusr1 : 1 time(s)
 pam_succeed_if(sshd:auth): error retrieving information about user svn : 1 time(s)
 pam_succeed_if(sshd:auth): error retrieving information about user dream : 1 time(s)

 ---------------------- SSHD End ------------------------- 

 --------------------- Sudo (secure-log) Begin ------------------------ 

 ==============================================================================
 imbrius => root
 ------------------------------------------------------------------------------
 /sbin/fsck /dev/sdc1
 /sbin/fsck /dev/sdc1
 /bin/mount /media/usb
 /bin/umount /media/usb

 ---------------------- Sudo (secure-log) End ------------------------- 

 --------------------- Disk Space Begin ------------------------ 

 Filesystem            Size  Used Avail Use% Mounted on
 /dev/mapper/VolGroup00-LogVol00
                       2.0G  741M  1.2G  40% /
 /dev/mapper/VolGroup00-LogVol11
                       2.0G   76M  1.8G   4% /tmp
 /dev/mapper/VolGroup00-LogVol12
                       301G   87G  200G  31% /export/home
 /dev/mapper/VolGroup00-LogVol01
                       7.8G  4.7G  2.8G  63% /opt
 /dev/mapper/VolGroup00-LogVol02
                       7.8G  2.9G  4.5G  40% /usr
 /dev/mapper/VolGroup00-LogVol10
                       992M  794M  147M  85% /usr/local
 /dev/mapper/VolGroup00-LogVol03
                       3.9G  1.2G  2.6G  31% /var
 /dev/mapper/VolGroup00-LogVol09
                       7.8G  414M  7.0G   6% /var/lib/mysql
 /dev/mapper/VolGroup00-LogVol05
                        93G  8.5G   79G  10% /var/ftp
 /dev/mapper/VolGroup00-LogVol06
                       2.0G  229M  1.7G  13% /var/cache
 /dev/mapper/VolGroup00-LogVol08
                       2.0G  240M  1.7G  13% /var/spool
 /dev/mapper/VolGroup00-LogVol04
                       2.0G  292M  1.6G  16% /var/log
 /dev/mapper/VolGroup00-LogVol07
                       496M   56M  415M  12% /var/www
 /dev/hda1             122M   33M   84M  28% /boot
 /dev/sdc1             230G  230G     0 100% /media/usb

 ---------------------- Disk Space End ------------------------- 

 ###################### Logwatch End #########################

Explanation:

The “httpd” section tells me all the errors that the server sent.  The 403 is correct – that page is off-limits to visitors.  The 404’s are all routine – the /favicon.ico file doesn’t exist due to the drupal multisite configuration.  The middle two 404s are unfinished parts of the church website.  The last 404 is someone trying something dodgy with my server and getting nowhere.  The 503s aren’t really 503s – they’re hits on cron.php in the various drupal sites but for some reason, the server logs a 503 even when it doesn’t send one.  The drupal crons get run with no problems anyway.

The “pam_unix” section details system logins.  The “runuser” entries are caused by the “news” virtual user starting an interactive shell to handle the hourly news cron jobs.  The “sshd” section is usually chock-full of people trying to brute-force attack my server.  The “su” section is caused by two accounts (imbrius and mom) running fetchmail every minute.  Fetchmail runs under an su shell to the user whose mail is being fetched.  The “sudo” entry is from me fat-fingering my password.

The “samba” section is because samba is configured to look for a CUPS-configured printer on the system and there is none.  In fact, CUPS isn’t even installed.

The “Connections [secure-log]” section is from Webmin.  I log onto it every day to follow-up the previous night’s backup.  The timeout notices happen when I forget to log out.

The “SSHD” section is almost always full (sometimes 1500 pages worth) of people trying to bash their way into my box.

The “Sudo (secure-log) ” section lists all the commands that I’ve run under sudo.

The “disk space” section is self-explanatory.

I also check my “chkrootkit” output every day.  Chkrootkit is a script that checks for obvious evidence of a rootkit on the system.  The output consists of about 30 lines-worth of file listings (dotfiles that chkrootkit thinks look suspicious) and a few lines of “hidden processes.”  The dotfile listings are always in error (Perl uses dotfiles in /usr/share and chkrootkit doesn’t like that).  The hidden process listing is what I actually look at.  Normally, there’s one or two listed because the chkrootkit runs while the backups are running.  Today, there was an extra one:  tcpdump.  That’s not right.

I signed on to the box over SSH and ran a pgrep tcpdump.  It came back with one PID.  I ps -ef | grep $pid and found out that it was started on March 18!  Apparently, I started a tcpdump localhost capture, stopped it, backgrounded it, and dtached it.  And it’s been sitting there dormant for a week and three days.  So I sent it a HUP and it died gracefully.

Advertisements

, , , , , , , , ,

  1. #1 by Blake on February 10, 2012 - 4:29 AM

    Very help, thank you!

  2. #2 by Blake on February 10, 2012 - 4:34 AM

    Although I find a few different things:
    In SSHD:
    Invalid Users:
    Unknown Account: 131 Time(s)

    Failed logins from:
    1.202.249.106: 490 times
    46.4.61.76 (static.76.61.4.46.clients.your-server.de): 20 times
    124.232.131.45: 4 times
    125.211.221.117: 13 times
    184.107.179.242: 23 times

    Illegal users from:
    1.202.249.106: 13 times
    46.4.61.76 (static.76.61.4.46.clients.your-server.de): 9 times
    124.232.131.45: 2372 times
    125.211.221.117: 2 times

    What would you make of these?

    Thank You

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: