Server Upgrade

Time and fate forced my hand. As someone pointed out, five years of pretty much continuous operation really isn't too bad from what really isn't even a server motherboard. On the other hand, the board was sufficient for my purposes. Unfortunately, the one absolute requirement of a server is that it be dependable. It appears that several of the capacitors on my server's old motherboard (ABIT KG7-RAID) were bulged and leaking (the brown schmoo on top of the capacitors near the CPU):

All I know is that the system became more and more unstable. Things like backups were failing and I was getting a kernel panic every once in a while. It's also possible that the PSU was part of the problem (bulged but not leaking capacitors). Thanks to the folks at BadCaps for helping to diagnose the problem.

Since my desktop was also getting a little long in the tooth, I decided the best way to handle the situation was to swap my desktop's motherboard into the server and get a new motherboard and CPU for my desktop. The server (fraud.davenjudy.org) is now powered by a Tyan Tiger MPX motherboard with dual Athlon 2400+ CPUs and 2 GB of RAM. My desktop is now a MSI K9N Neo V3 with an Athlon 5600+ 64X2 and 2GB of RAM. My original goal was to leave the hard drives in each case and just swap motherboards on the assumption that this would allow me to just keep going. Alas, that didn't happen and thus this blog posting.

The first thing that went wrong was I couldn't get the server to boot. As an example, I labelled both ends of the IDE cables so I would know which cable went where and attached to which drives. The cable that was plugged into the first Highpoint RAID IDE port of the KG7-RAID got labeled IDE3 for the motherboard end and each drive connector had enough information to know which hard drive attached to each connector. I got part way through the boot process before there was a kernel panic. I don't know whether moving two pairs of disks each with various Linux software RAID partitions from one box to another was too much to ask or if the second port on the Promise IDE paddle I was using to hang all of these drives off of the system was the problem (I had some trouble getting a drive recognized by it later). Either way, I was stuck with a non-working system and a bunch of disks.

Prior to starting the upgrade/swap, I had considered removing the two smallest drives from the system (two Seagate ST34311A drives of 4.3 GB each) but decided that I would rather get back up and running quicker. Since that didn't happen, I pulled the two Seagate drives and just used the two Western Digital 60GB drives that I had originally used for /home for the system install. This meant that /home would get zapped but I had backed it up before starting this fiasco. At least that much went right.

The next little gotcha was I only had the first installation CD for CentOS 5. For other systems, I had used the first CD to get the install going and had then pointed the installer at a network share for the remaining CD images. The problem was that network share was in the box I was trying to install to. Luckily the Red Hat/CentOS install process sets up and activates the network as part of the initial install process. I was able to get a host-to-host connection working between what was to be my server and my laptop and scp the remaining CD images to the laptop and then burn them on the CD burner on the laptop. Obviously, that worked since I'm posting this but then the fun of restoring the server to its prior capabilities began.

The remainder of this post is separated by topic and describes what I had to do to get each of the various server functions working again. As with /home, I had backed up /etc and /var so the first step was to unpack these archives. This at least made my old CentOS 4 configuration available which provided a good starting point for getting each function working under CentOS 5. Since this was a clean install of CentOS 5, the basic system "just worked" once the install completed. I will add any additional details to the following as I remember what I did and/or have time.

DHCP - The weird thing about getting DHCP working was that everything worked fine until I tried to bring up my new desktop. The weirdness was that it kept getting assigned the IP address of the server's internal NIC. I looked at everything I could think of including the settings for named, the lease files, etc. but nothing I did fixed it. As I kept eliminating possibilities, I became more and more convined that the address was being assigned by the server.

It finally dawned on me that the NIC in my desktop had previously been in the old server (several things got swapped around besides the motherboards). I finally tracked down the problem to /etc/dhcp.conf. This file is where you configure your dhcp server with things like the address pool, and "hard" assignments, things like DNS server and network gateway, etc. The one oddity is the DNS server setup takes the MAC address of the the server's NIC to make sure it always gets the same assignment.

Sure enough, the DNS still had the MAC address of the NIC that was now in my desktop. This wouldn't have been a problem if there were support for the NIC built into the K9N but there isn't. I had stuck the old server NIC into one of the K8N's PCI slots when I couldn't find a driver for the built-in NIC.

the only other issue with DHCP was getting dhcpd to stop complaining that it didn't have any configuration information for the public IP NIC on the server. This is squirrelled away in /etc/sysconfig/dhcpd and is passed in as a command line argument. So the file looks like:

cat /etc/sysconfig/dhcpd
# Command line options here
DHCPDARGS=eth1

with DHCPDARGS getting appended to the dhcpd command line and telling dhcpd to only pay attention to eth1.

web - The worst part of getting the web site back up was just going through the Apache 2.2 configuration file and figuring out how to bring forward the old (Apache 2.0) settings. I ended up commenting out the virtual host settings for the internal host that handles the dSPAM web interface. I'll return to what it took to get dSPAM working at some point below. I also haven't tried to get squirrelmail working yet so that will also be handled separately. WordPress came across with almost no effort at all so I upgraded to 2.3.2 while I was at it (but I remembered to get the php-mysql driver RPM and install it).
I do propose that a particularly hot corner of hades be reserved for those people who change the spelling of a configuration file item because "the new name better describes what it really does."

e-mail (SMTP) - Not much to say here. The sendmail.mc file changed very little in ways that affected me so I could pretty much just bring across the settings specific to my system. I did comment out the settings for dSPAM until I was ready to set up dSPAM.

e-mail (IMAP) - The bad news is that /etc/dovecot.conf changed significantly from CentOS4 to CentOS 6. The good news is that the new defaults seemed to work just fine. All I really had to do was configure the "listen address".

e-mail (dSPAM) - I'm still working on my dSPAM configuration but I have it working with Apache 2.2. Besides the myriad other details of getting a dSPAM installation working, the main problem was how to get the dsspam program to be executable by both the mail receive process and the web user interface (UI). The root of this is that the dspam program needs to be installed setgid (set group ID) so its permissions look like:

[root@fraud etc]# ls -l `which dspam`
-r-x--s--- 1 root mail 376701 Jan 20 14:52 /usr/local/bin/dspam

which means that the dspam program is executable by root and users in the group mail and that when run by users in the group mail, the program runs as if it were run by root. This works great but the problem is that httpd (Apache) runs as group apache and letting it run as mail (what I used to do) probably isn't a good idea.

To make matters worse, the Apache facility for running as a different user or group within a particular virtual host changed from Apache 2.0 to Apache 2.2. Apache 2.2 now uses "SuexecUserGroup" to run as a different user or group and the logic/build options for suexec won't let it run for system users or group IDs. That is, I cannot (I tried, I know) do something like "SuexecUserGroup dspam mail" to run as the dspam user within group mail. I also don't like the idea of setting dspam to be executable by anyone since it runs setgid.

My workaround for the moment is to use two different dspam executables. I use the standard dspam executable in /usr/local/bin with the permissions shown above to work as the mail filter. I use a separate dspam executable with appropriate permissions, owner and group to run from the web interface. This works so far although I haven't tried toreprocess a false positive as a good e-mail. I will be testing that shortly. Currently, this is just the dspam executable in the build directory. This will change depending on what I find with some additional testing (e.g., maybe a second copy of dspam as dspamw in /usr/local/bin but with appropriate permissions and ownership to be functional from the web UI).

Other than the above issues with Apache 2.2, dSPAM seems to be working. The sendmail and web UI entries on the dSPAM wiki were invaluable in getting things this far.

e-mail (squirrelmail) - Sqirrelmail is up and working again. I just had to run the configuration (conf.pl) tool in /usr/share/squirrelmail/config. I prefer to re-run the configuration tool on an upgrade rather than copy in the old configuration file as long as setting the configuration is fairly simple. That way I don't have to worry about some minor syntax or setting change messing things up. Geeting the https redirect to work for apache was even easier. Just reload apache (httpd) once the squirrelmail pieces are in place.

named - Nothing huge here although the default permissions for local.davenjudy.org and 192.168 don't allow dynamic updates. I also moved the zone files from the non-chroot location (.var/named) to the chrooted location (/var/named/chroot/named). MyCentOS 4 configuration had the files in the default location (/var/named) and symbolic links from /var/named/chroot/var/named to the actual files.

NIS - NIS was a work in progress before I even did the server upgrade. Since the majority of the systems here run Linux, I'd really like to have a single set of authentication credentials (passwd, shadow, etc.).This would make life much easier when it comes to making sure permissions and ownership are correct on NFS shared drives. For the most part, getting NIS working was primarily a matter of just bringing across the various configuration files like ypserv.conf, nsswitch.conf, etc. The missing step was to run:

/usr/lib/yp/ypinit -m

on the server.

Also, the web how-to says to start with the client and start ypbind on the client before you configure the server. this may work in other environments but the Red Hat implementation of ypbind hangs for a really long time listening for an NIS server. A better approach is to configure the server and then do the clients.

amanda - Configuring amanda wasn't really a problem since it just entailed bringing across the various files from the old configuration. What made the problem tricky was that amcheck would barf and give a traceback if the "next" tape wasn't in my tape changer.