Being Mean to Minecraft

UPDATE 20110216 1942:  Whitelist is working and populated:

Whitelist is working

Whitelist is working

UPDATE 20110215 1943:  General and Permissions both work:

if /help shows the help message, then General is working

if /help shows the help message, then General is working

I seem to be having trouble downloading the Whitelist plugin – their webserver seems to be either down or overloaded at the moment.

UPDATE 20110214 1854:  Bukkit Lives!  I’m just a moron:  Minecraft doesn’t use Linux’s tcpd wrapper for net access so it does its own binding calls.  Naturally, 127.0.0.1 is already taken up by tcpd.  Once I told it what interface to listen on, it worked fine:

Bukkit Lives!  (And I'm an idiot!)

Bukkit Lives! (And I'm an idiot!)

And I’m running Minecraft at niceness 10 so it doesn’t take anything away from the rest of what dustpuppy’s running:

dustpuppy under load with Minecraft at niceness 10

dustpuppy under load with Minecraft at niceness 10

UPDATE 20110215 1845:  Initial tests with Bukkit aren’t very promising.  Although the server starts, it’s refusing connections.  I verified that the firewall ports are open and there are no AVC denials.  I’m researching now.

Packet analysis - I get a TCP/RST,ACK packet back - standard "Port Is Closed" message.

Packet analysis - I get a TCP/RST,ACK packet back - standard "Port Is Closed" message.

I wanted to post the contents of the homegrown cronjobs I mentioned before:

check_newkernel.sh:

#!/bin/bash
#
# Checks to see if the current kernel version matches the newest kernel image in /boot
#

NEWEST_IN_BOOT=`grep vmlinuz /etc/grub.conf | tail -n +2 | head -n 1 | cut -d"-"
-f2,3 | cut -d" " -f1`
#echo "DEBUG:  Newest available kernel image in /boot is $NEWEST_IN_BOOT"
CURRENT_VERSION=`uname -r`
#echo "DEBUG:  Current kernel release is $CURRENT_VERSION"

if [ "$NEWEST_IN_BOOT" != "$CURRENT_VERSION" ]
then
        mail -s "New kernel available" root@dustpuppy <<EOF

******************************
* A new kernel is available. *
* Please schedule a reboot.  *
******************************

Kernel release $NEWEST_IN_BOOT is available in /boot.
Current release is $CURRENT_VERSION.

-root on dustpuppy

EOF
fi

findaltroots.sh:

#!/bin/sh

for id in `awk 'FS=":" {if(($3 == 0 && $1 != "root" )) print $1}' /etc/passwd`
do
        mail -s "Root Access Alert" root@dustpuppy << EOF

********************************************************************************
*                                                                              *
*                ALERT! Login ID `echo ${id}` has uid 0                        *
*                `date "+Detected On Date :%D Time :%r"`                       *
*                                                                              *
********************************************************************************

EOF
done

lost+found-check.sh:

#!/bin/bash
#
# checks for files in lost+found directories

ALERT=`find / -type d -name 'lost+found' -not -empty -print 2>/dev/null`
LINES=`echo $ALERT | wc -l`

if [ $LINES > 0 ]
then
        for dir in $ALERT
        do
                mail -s "$dir not empty!" root <<EOF
****************************************
* $dir is not empty at `date`          *
****************************************

`ls -l $dir`

Please take care of these files.
-root@dustpuppy

EOF

        done
fi

UPDATE 20110213 1701:  Lest you think I’ve been slacking, I’ve declared dustpuppy done and in production and I’ve begun to copy the Minecraft server root over to dustpuppy from SavageChicken so I can play with Bukkit without risking the live Minecraft instance:

Me tarring up the Minecraft server root on SavageChicken

Me tarring up the Minecraft server root on SavageChicken

(green is SavageChicken and amber is dustpuppy)

UPDATE 20110211 1946: If I like the way my cron reports look in the morning, I’ll consider dustpuppy a finished peice and schedule the backups.
Here’s a quick cronjob audit:

Hourly

  • awstats (web stats data collector script)
  • drupal (CMS cron runner)
  • optimize_drupal_db.sh (homebrew script to run a quick table optimization under write lock for the website DB)

Daily

  • 00webalizer (webserver performance monitor data collector script)
  • 0anacron (checks to see if any cronjobs were missed in the last day)
  • 0logwatch (system log monitor data collector script)
  • certwatch (SSL / PKI cert management watchdog script)
  • check_newkernel.sh (homebrew script to check if the running kernel is the latest one available in /boot)
  • cups (Common Unix Printing System cron runner)
  • findaltroots.sh (homebrew script to report any entries in /etc/passwd that have a UID of 0 but aren’t called “root”)
  • logrotate (logfile rotation and cleanup script)
  • lost+found-check.sh (homebrew script to check for files in any lost+found directories)
  • makewhatis.cron (data collector script for quick-access one-line system reference)
  • mlocate.cron (filesystem indexing script)
  • prelink (GNU Dynamical Linker cache builder script)
  • rpm (package DB builder script)
  • sa-update.cron (spamassassin rules updater script)
  • squirrelmail.cron (tempfile cleanup script for SquirrelMail webmail interface)
  • tmpwatch (system tempfile cleanup script)

Weekly

  • 0anacron (checks to see if any cronjobs were missed in the last week)
  • 99-raid-check (checks to make sure the RAID devices are still in sync)
  • makewhatis.cron (data collector script for quick-access one-line system reference)

Monthly

  • 0anacron (checks to see if any cronjobs were missed in the last month)

Other Schedules

  • imbrius: every minute, pop RoadRunner mail into system mailbox
  • imbrius: every minute, generate new fortune for e-mail signature file

UPDATE 20110209 1745:  It’s official:  SavageChicken is no longer burdened with serving websites!  The traffic was successfully rerouted to dustpuppy and the http and mysql daemons on SavageChicken have been shut down and disabled:

(SavageChicken) $ sudo service httpd stop
[sudo] password for imbrius:
Stopping httpd:                                            [  OK  ]
(SavageChicken) $ sudo service mysqld stop
Stopping MySQL:                                            [  OK  ]
(SavageChicken) $ sudo chkconfig httpd off
(SavageChicken) $ sudo chkconfig mysqld off

And the system load looks MUCH better:

SavageChicken's top output without running httpd and mysqld

SavageChicken's top output without running httpd and mysqld

UPDATE 20110131 1618:  After completing moving everything but the mail routing, this is what the late afternoon system usage snapshot looks like:

Top is CPU, Middle is RAM, bottom is Network

Top is CPU, Middle is RAM, bottom is Network

UPDATE 20110129 1125:  Apache, MySQL, and drupal are working on SavageChicken and the website load has been transferred successfully from dustpuppy.  As of 1125 this morning, dustpuppy is no longer responding to HTTP requests.  (Apache is still running for Webmin and Ajaxterm but those are HTTPS):

Watching the HTTPD access_log on SavageChicken

Watching the HTTPD access_log on SavageChicken

Yes, that’s real internet traffic.  SavageChicken is now the webserver for the lakemasoniccenter.org and firstlutheranelca.org domains until further notice.

dustpuppy needs some attention.  I’m sick and tired of LVM taking a random crap all over the filesystems so I’m going to get rid of it.  To do that, I need to reinstall the OS.  To do that, I need to move the 3 websites, databases, e-mail routing, etc., etc., etc. elsewhere.  Where else?  Well, I have a kickass Sun Enterprise 420R that would handle that load without even batting an eye.  Sadly, it consumes more power than running a vacuum cleaner 24/7 and is unlikely to be put into production on this ancient wiring.

But I do have another server.  And it’s mostly unused.  Minecraft uses less than 25% CPU at peak load and around 50% of the system RAM.  So for a few days in February (probably between 3 and 5), I’ll be running 3 websites, the database backend for those websites, the FTP server, the NAS, the mailbox store, and the e-mail routing engine off of SavageChicken.  This will put it under stress – dustpuppy’s load average ends up at around 0.08-0.37 for a 24 hour test.  This is likely to have an impact on Minecraft but only during peak loads (>8 players at a time) or when players are more than 64 blocks apart and the system has to mmap large areas of the world from disk into memory.

Since SavageChicken has two puny IDE drives and is in no way equipped to handle NAS duty, I’ll be connecting Chad’s 120GB USB drive and my 250GB USB drive to use as the FTP store and NAS respectively.  Everything else (including websites, database, mailbox store, and mail routing engine) will run off of the slow IDE drives.  Well, slow compared to the SATA-150s in dustpuppy.  Fast compared to the USB1.1 external drives :P

So it won’t be ideal for anybody but if it causes too many issues, I’ll see about changing things around and maybe suspending some services like FTP until we get this resettled.

I do NOT anticipate needing to stop Minecraft at any time to transfer these services and hardware over so you shouldn’t notice any interruption in service.

About these ads

, ,

  1. #1 by Christa on January 27, 2011 - 4:33 PM

    Fair enough. I don’t think any of us have a right to complain. It’ll all be fine.

  2. #2 by Chadwick on January 27, 2011 - 5:46 PM

    I’ll complain! That thing’s 160GB (at least on paper).

  3. #3 by Chadwick on January 29, 2011 - 2:46 PM

    Well, good luck with the load balancing.

    • #4 by Joshua on January 29, 2011 - 5:37 PM

      Any impact to game performance so far? I did notice that at 5:30 this afternoon, the CPU was pegged but most of it was in IOWAIT which is attributable to the slow IDE drives. :-(

      • #5 by Chadwick on January 29, 2011 - 10:30 PM

        Well, it took a while to log in, though Minecraft.net was also having trouble all day, so it may have been a matter of authentication delays. Other than that, I didn’t notice much.

  4. #6 by Joshua on February 2, 2011 - 7:15 AM

    With any luck today I should be able to get Linux reinstalled on dustpuppy. Then later this week, I can move the database and webserver back off of SavageChicken so Minecraft will have free rein once again.

    • #7 by Chadwick on February 2, 2011 - 9:23 AM

      Awesome. When that time comes, it looks like we can give Bukkit a spin and see how she flies. I’ll be posting a bit about that on the Minecraft Server page shortly.

      • #8 by Joshua on February 2, 2011 - 9:29 AM

        Sweet! I’d love to get repairs and teleporting back!

      • #9 by Joshua on February 2, 2011 - 9:31 AM

        The plan is to clone the minecraft server over to dustpuppy and test there.

  5. #10 by Joshua on February 11, 2011 - 7:05 PM

    I know I promised that I’d fiddle with Bukkit tonight but I’ve still got a tiny bit of admin work to do. Not the smallest of which is setting up backups.

    I’ll try to get to it tomorrow. I’ve got an auto detailing appointment in the morning and I have to deacon at church in the afternoon. Sorry I’m being such a slacker but it’s been a busy week.

    • #11 by Chadwick on February 11, 2011 - 8:20 PM

      Oooh, detailing. Does that mean the thing with the girl is upcoming soon?

      • #12 by Joshua on February 11, 2011 - 8:26 PM

        Thursday. Before then, I need to get detailed too. Gonna be the beardy trimming and the hairs-cutting.

  6. #13 by Joshua on February 11, 2011 - 8:10 PM

    If it helps, Firefox wanted to change every instance of “cronjob” to “con job.” :P

  7. #14 by Joshua on February 13, 2011 - 12:22 PM

    Update: The dumps look good, the crons ran, and no more AVCS. Dustpuppy is back in business. This afternoon, I’ll copy Minecraft over and download Bukkit. :-)

  8. #15 by Joshua on February 13, 2011 - 5:46 PM

    2.5GB? WTF have you people been doing on this server??? :P (no worries, there’s still 40GB free on /opt on SavageChicken and 7.5GB free on /opt on dustpuppy)

    • #16 by Chadwick on February 13, 2011 - 6:07 PM

      We built a new city. I’ve been building walls!

      • #17 by Joshua on February 14, 2011 - 7:02 AM

        You are good at building walls :P

      • #18 by Andrew on February 14, 2011 - 9:54 AM

        I am proud of Moonglow. The city has come along beautifully. I hope it becomes a great hub for more exploring around the world as we make more fun projects. I just need to come up with some now… I ran out of ideas which is kinda sad….

  9. #19 by Joshua on February 14, 2011 - 7:00 AM

    UPDATE: The minecraft server snapshot tarball has been extracted to dustpuppy and the Minecraft server started successfully during the test run. This does mean that since the snapshot was taken last night, the Bukkit sandbox instance (which I will open up as soon as I get Bukkit installed and working) will be current only as of last night. Once we’re all satisfied with the way Bukkit works, I’ll take another snapshot, deploy Bukkit, and then swap that in as the production instance. This offers us maximum protection against Bukkit hosing the world and also minimizes downtime.

    The server takes diff backups every 5 minutes so I can undo any changes anyone does in haste. I also take nightly world-state backups that get saved off to disk (and eventually off to DVD+RW). But the fact that it’s daily means that if the world data gets hosed at 9 PM, the last available full restore is from 20 hours prior (differentials are only useful on intact data – corruption needs a full restore).

    I’ve tested the backup and restore strategy in the past and it does work well. It eats up an assload of disk space but I think it’s worth it. (And disk space is cheap anyway – you guys have SavageChicken’s 500GB disks all to yourselves :P)

  10. #20 by Joshua on February 14, 2011 - 7:03 AM

    You guys will notify me if the server needs updating, right? I haven’t signed on in a few days to check.

    • #21 by Chadwick on February 14, 2011 - 8:39 AM

      Notch’s been on vacation for two weeks, so there haven’t been any notable updates.

  11. #22 by Joshua on February 15, 2011 - 6:58 PM

    Now to start heaping on the plugins :-)

    • #23 by Chadwick on February 15, 2011 - 7:02 PM

      Yay! Have you considered the Hot-Swap plugin, which would let you make adjustments on the fly?

    • #26 by Joshua on February 15, 2011 - 7:15 PM

      Gen and Permissions have a lot of options to configure. I’ll be working on those first. MonsterHunt requires a database server so I’ll have to set up dustpuppy to use that. Likewise, I might as well set up dustpuppy to be the Minecraft DB overall so I can avoid the 2.2G daily world backups :P

  12. #27 by Matt on February 18, 2011 - 12:17 PM

    Not sure if Dorney mentioned this, but our friend Joe is tossing some old Hp ProLiant G4 servers… only a couple years old, but still pretty beefy. Downside is the power consumption, 1 amp plugged in but turned off. If you want to throw one or two into your basement, let me know!

    • #28 by Joshua on February 18, 2011 - 12:26 PM

      That sounds awesome but I’ve got no way to power them and nowhere to store them. Sorry. But thanks for the offer.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 48 other followers

%d bloggers like this: