Had an interesting problem today with one of our servers at work.
First thing I noticed was yesterday that an upgrade of Apache2 didn’t
complete properly because /etc/init.d/apache2 stop didn’t
return. Killing it and starting apache allowed the upgrade to
finish. I noticed there was a zombie process but didn’t think too much
off it.
Then this morning got an email from the MD saying that various
internal services websites were down (webmail, wiki etc). My manager
noticed that it was due to logrotate hanging, again because restarting
apache had hung. Looking at the server I noticed a few more zombie
processes. One thing I’d noticed was that all these processes had
reparented themselves under init and a quick web search later confirmed
that init(1) should be reaping these processes. I thought maybe
restarting init would clear the zombies. I tried running
telinit q to reread the config file, but that returned an error
message about timing out on /dev/initctl named pipe. I checked that file
existed and everything looked fine. The next thing I checked was the
other end of the named pipe by running lsof -p 1. This
showed that init had /dev/console rather than
/dev/initctl as fd 0. I tried running kill -SIGHUP
1, but that didn’t do anything. Then I tried kill
-SIGUSR1 1, but that didn’t do anything either. I checked the
console, but there wasn’t enough scrollback to see the system booting
and decided to schedule a reboot for this evening.
Rebooting the server presented me with an interesting challenge.
Normally the shutdown command signals to init to change
to runlevel 0 or 6 to shutdown or reboot using /dev/initctl. Of
course init wasn’t listening on this file, so that was out. Sending it
an SIGINT signal (the same signal init gets on ctrl-alt-delete) had no
response. Obviously telinit 0 wasn’t going to work
either. I decided to start shutting services down manually with the help
of Brett Parker. The idea
was to stop all non-essential services, unexporting nfs exports,
remounting disks read-only and then either using sysrq or a hardware
reset. Unfortunately someone accidentally ran /etc/init.d/halt
stop, hanging the server, but he is suffering from a bad cold today so I forgive
him. The server restarted without a hitch (thank god for ext3) and
running lsof -p 1 showed init having
/dev/initctl open. I don’t know what happened to init the last
reboot on Monday, but a reboot seemed to fix it. Odd bug, but thankfully
it was a nice simple fix. I could have spent the evening debugged init.
🙂