Reigning in the crons

I had to kill a wedged denyhosts process today on my server.  I noticed last night that a box got off over 500 ssh crack attempts in an hour.  Denyhosts is supposed to kick in and block that IP after 5 failed attempts in 3 minutes.  So I checked on the state of the program with an strace and it said it was waiting on a return from select(NULL, 0, 0, 0... and that told me that it was wedged.

So I sent it two HUPs (to be polite and give it a chance to unwedge before I took more drastic measures) and eventually had to send a TERM.  It terminated gracefully but left its lockfile in /var/lock/subsys which had to be removed manually.  Then I ran into another problem:  what runs the denyhosts daemon at startup?  I didn’t want to just run /usr/sbin/ --daemon -c /etc/denyhosts.cfg, even if that would start the daemon.  It would be out of control of the /sbin/service subsystem then.

So I did what any sane sysadmin would do, check the initscripts.  It wasn’t listed in /etc/init.d, so I checked /etc/rc.local.  Not there either.  So I did a find /etc/ -type f -exec grep -l '/usr/sbin/' \{\} \;. It pulled up some config files and RPM catalogs but the thing that caught my eye was the entry in /etc/init.d called “daemon-control.”

It turns out that since denyhosts is from EPEL and not native to RHEL, it doesn’t have a standard init script.  Instead, the /usr/share/denyhosts/ control script just gets copied into /etc/init.d.  Now that I knew what the init script was called, I was able to run /sbin/service daemon-control start and have it load the denyhosts daemon.

I figured that if I had that much trouble tracking down a minor tool, how much other crap is running out of cron and /etc/rc.* that needs to be checked into?

I started in /etc/cron.d.  Immediately, I found an outdated daily mailman crontab (mailman crons run from root’s personal crontab on my boxen) that I turned off (by removing the execute bit).  Next, I found something that could answer a question that’s been bugging me for a while now:  Why does the server suspend the network layer, force the incoming packets to buffer, unload the firewall rules, flush the networking drivers, and reload the firewall rules every 20 minutes?  This entry explained it:

green [root@dustpuppy cron.d]# cat portsentry
# Restart portsentry to reset the history file
# - & -
# Flush the entries added by portsentry in iptables or ipchains
# $Id: portsentry.cron,v 1.2 2003/09/17 15:53:04 dude Exp $

03,23,43 * * * *        root    /sbin/service portsentry restart >/dev/null && /sbin/service iptables restart >/dev/null
#03,23,43 * * * *       root    /sbin/service portsentry restart >/dev/null && /sbin/service ipchains restart >/dev/null

I changed it from every 20 minutes to every 24 hours.  I’ll be back later with the rest of my adventures in cron land.

  1. Leave a comment

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: