Okay: I set the fstab errors= option to continue for all filesystems the last time I had to reboot. And now this morning, the spool filesystem ate itself and I found out that it was the only filesystem I DIDN’T set to errors=continue. 😦 Now it’s spent the last two hours resolving multiply-claimed inodes on a filesystem with over 700,000 files.
IMPORTANT!!
Okay, so Minecraft is still in beta. This is abundantly evident as my hole now has a front row seat to an iceskating rink:
Yup. That used to be my flower garden and large front window.
I think that sign is a bit misleading.
The tunnel is still good but now includes a swimming lane.
amoco
baleon
Phillip
Chadwick
London Burt
MHook
ottomadenedamotto
Christa
Matt
VisualHierarchy
Andrew D.
Around Teh Table 
#1 by Chadwick on November 1, 2010 - 6:55 PM
Wow.
#2 by Joshua on November 1, 2010 - 6:57 PM
I can try and fix it by restoring from a backup it made 2 days ago. Do you want me to try that?
#3 by Joshua on November 1, 2010 - 7:39 PM
It’s restoring now. I did make a copy of the world as-was in the glitched state in case we need to rollback.
#4 by Joshua on November 1, 2010 - 7:40 PM
Minecraft is on the same filesystem as steam so it’s got all the orangebox content to skip through before it gets to minecraft. Hooray linear-access backups.
#5 by Joshua on November 1, 2010 - 7:49 PM
Done with orangebox, now doing nessus…
#6 by Joshua on November 1, 2010 - 7:54 PM
DOne with nessus, now doing jboss
#7 by Joshua on November 1, 2010 - 8:03 PM
It found minecraft and is reading the headers. It should start writing in a minute.
#8 by Chadwick on November 1, 2010 - 8:03 PM
Yay!
#9 by Joshua on November 1, 2010 - 8:04 PM
Writing now.
#10 by Joshua on November 1, 2010 - 8:05 PM
Restore success. I’ll restart the minecraft server and see what we have.
#11 by Joshua on November 1, 2010 - 8:09 PM
Well it’s still glitched. It’s a code bug:
(dustpuppy) $ sudo less nohup.out Starting minecraft server: CentOS release 5.5 (Final) Today is Setting Orange, the 13rd day of The Aftermath in the YOLD 3176 18:25:18 up 17 min, 1 user, load average: 0.27, 0.73, 0.83 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT imbrius pts/0 marlene.lakemaso 18:16 0.00s 0.12s 0.09s -ksh No, we put him in there with her, then nail the crate shut. Then they will fall in love. Or die trying. - King Julien, Monkey Love java.io.IOException: Bad file descriptor at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:236) at java.io.BufferedInputStream.read1(BufferedInputStream.java:273) at java.io.BufferedInputStream.read(BufferedInputStream.java:334) at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:282) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:324) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:176) at java.io.InputStreamReader.read(InputStreamReader.java:184) at java.io.BufferedReader.fill(BufferedReader.java:153) at java.io.BufferedReader.readLine(BufferedReader.java:316) at java.io.BufferedReader.readLine(BufferedReader.java:379) at bw.run(SourceFile:79) 2010-11-01 18:25:19 [INFO] Starting minecraft server version 0.2.2_01 2010-11-01 18:25:19 [INFO] Loading properties#12 by Chadwick on November 1, 2010 - 8:10 PM
Right. So what does that mean, practically?
#13 by Chadwick on November 1, 2010 - 7:48 PM
Technically, Alpha. He’s set “getting damage and minecarts to work correctly in Multiplayer” as the bar to hit to declare it Beta.
#14 by Joshua on November 1, 2010 - 7:51 PM
Lol. Hopefully bugs like this get knocked out fairly soon.
#15 by Joshua on November 1, 2010 - 8:46 PM
Okay: regarding the lag, I don’t know why the server process is stuck. I don’t have the source code to debug it. If it’s still doing it tomorrow, I’ll try restarting the minecraft server process but for now, I’m getting tired of banging my head against it.
Sorry for providing substandard minecraft servage.
#16 by Chadwick on November 1, 2010 - 8:57 PM
Lol. It’s okay. Totally not your doing.
#17 by Chadwick on November 1, 2010 - 9:29 PM
Known item: http://notch.tumblr.com/post/1452872379/survival-mode-multiplayer-is-priority-1
#18 by Joshua on November 2, 2010 - 8:53 AM
I’m at workee so no minecraft atm but did you guys ever get the cube back in the ground?
#19 by Chadwick on November 2, 2010 - 9:25 AM
It got down to like 3-4 levels high before the server kicked me off, and Bjorn checked out ’cause it was too glitchy. So basically, it’s nearly there, and if the server behaves tonight, we’ll probably knock it out.
#20 by Joshua on November 2, 2010 - 7:16 PM
I set up a job to tar up the world nightly and copy it off to /home which does get backed up daily.
I also restarted the minecraft server process to address the day/night loop issue.
#21 by Phillip on November 2, 2010 - 8:12 PM
What’s the day/night problem? When I was playing earlier it seemed to be working fine.
#22 by Joshua on November 2, 2010 - 8:19 PM
Link here. It’s a bug with lighting. Causes a CPU leak. It sucks. 😦
#23 by Christa on November 2, 2010 - 8:21 PM
So is it going to be down for a while? Just so I don’t keep checking like the addict I am.
#24 by Chadwick on November 2, 2010 - 8:31 PM
Oh, so it’s not just me…
#25 by Joshua on November 3, 2010 - 7:13 AM
It should be back up. It was stuck in a loop again.
#26 by Chadwick on November 3, 2010 - 11:41 AM
Hmm…Still what I had before; it goes to log in, and times out after 60 seconds. Which doesn’t seem like the same behavior I was experiencing with the previous loop. But that might just be me.
#27 by Joshua on November 3, 2010 - 11:46 AM
Well it has the CPU pegged again. It was good as of 10:00 AM this morning. I don’t have a clue at this point. I’d really like to just tar up this whole install, download the latest server version, and let it create a new and hopefully more stable world. As far as changes, I can d/l a world editor and recreate all our houses on the new world if it comes to it.
#28 by Chadwick on November 3, 2010 - 12:49 PM
I’m not sure I’d object to that; assuming that the existing one got saved somewhere.
#29 by Joshua on November 3, 2010 - 3:11 PM
I looked into moving from the uniproc 1500MHz 32 bit box onto the quad-proc 450MHz 64 bit box for the minecraft server. But at the moment, multi-processing support is buggy at best (it picks 1 CPU and hogs 100% of the resource of that chip, doing nothing with the others). So I’ma leave it where it is for now, tar up the current server, d/l the latest version, and let it build a new world. Then we’ll revisit a map editor to rebuild our fortresses.
#30 by Chadwick on November 3, 2010 - 3:21 PM
I’ll just sit here and be sad in the meantime.
#31 by Joshua on November 3, 2010 - 3:22 PM
Which hopefully won’t be long.
#32 by Joshua on November 3, 2010 - 3:40 PM
It turns out I may have killed the server by not having a way to safely stop it other than by sending two SIGHUPs and a SIGTERM. But thanks to this SysV init script, I now have a way of safely starting and shutting down the server.
#33 by Joshua on November 3, 2010 - 3:43 PM
Apparently the 100% CPU bug was caused or exacerbated by my starting the server with nohup – the SIGHUPs would be blocked and the TERM would catch the server offguard.
#34 by Chadwick on November 3, 2010 - 4:06 PM
Huh. Good to know, I guess.
#35 by Joshua on November 5, 2010 - 7:58 AM
I’d like to thank Dagmar d’Surreal for the init script. It works excellently out of the box on RHEL 5.5. Far better than my attempt at writing one from scratch with
nohup. I thought about usingscreenbut I haven’t used it in forever since virtual terminals are all the rage now. 😛#36 by Joshua on November 4, 2010 - 7:57 AM
“What the deuce? That ain’t right!”
If, like Skipper, I could ask Kowalski for analysis, I would. But I can’t. At since I’m at work, I don’t have shell access to dustpuppy, only webmin. Webmin says “CPU load averages: 38.13 (1 mins) , 37.60 (5 mins) , 36.67 (15 mins).” In general for dustpuppy, any load average over 1 is heavy, over 3 is saturated, over 10 is suspicious, and over 20 is WTF. Seeing averages in the high 30s isn’t exactly good news. Especially after my car got covered in other people’s shit (sadly, “shit” in the literal sense) on the freeway this morning.
The good news is that Minecraft seems to be doing okay at 17% CPU usage. Can anyone report on how it’s doing from the client side? Since I can’t account for the sudden spike in system load, I’ll assume it’s all going to IOWAIT which doesn’t show up in CPU usage stats. It’s very possible that the backup is hosed and saturating the IO channels. I’ll have to wait and see.
#37 by Joshua on November 4, 2010 - 8:52 AM
My hypothesis was correct. The external disk was acting as a teergrube for the backup and the system was pegged in IOWAIT. It’s fixed now. I may need a new external drive though.
#38 by Chadwick on November 4, 2010 - 2:53 PM
Everything looks/acts fine from my end, for what it’s worth.
#39 by Joshua on November 5, 2010 - 7:49 AM
Yayyy. And I’ve brokered a deal to exchange one cobalt of Sprecher Vampire Blood (Flanders Red Ale w/hint of raspberry) for a 160GB external USB hard drive for the backups since my 5-year old Seagate FreeAgent Desktop is acting stupid and locking up the I/O bus.