Greg Hudson’s MIT blog

Frustratingly, I’m having to pull myself away from cobwebs work (in favor of Openfire work) just as I feel like I’m close to having a configuration I can show around and start getting feedback on.  Mostly, I need to set up SSL, write scripts to provision new user accounts, and write a little placeholder front-page content.  If anyone is really interested, I could manually set up some accounts and let them poke around.  Some assorted ramblings below.
LVS was a little confusing to set up, partly due to out-of-date documentation.  Like scripts, cobwebs will use the “direct” (or “gateway”) packet-forwarding mode, which means the web server nodes have an arpless interface alias for the virtual IP address which they share with the arpful interface alias on the LVS node.  Setting up an arpless interface alias in Linux is sort of a black art; the mechanism has changed across kernel releases and it still isn’t all that pretty.  Today you can “ifconfig interfacename -arp” which might or might not scope properly for an interface alias; regardless, the RHEL 5 init infrastructure doesn’t do that if you say ARP=no in an ifcfg file, but instead does “ip link set dev devname arp no” on the parent interface, which is the wrong scope.  The workaround is to stick your arpless interface aliases on the lo device instead of the eth0 device, since it doesn’t matter if the true lo device arps or not.  Also like scripts, cobwebs will use the “source hash” scheduling mechanism, which will tend to route the same requestor IP address to the same server.  That’s probably good for performance over GFS, but it might actually be better to route requests for the same domain (username.cobwebs.mit.edu) to the same server.  That’s much harder to do, though.
Out of curiosity, I looked into how scripts.mit.edu was dealing with sharded /tmp; apparently it isn’t ideal if a node goes down, and having an unshared /tmp might be better even if it breaks the illusion of a single machine.  On cobwebs, I can just bind-mount /home/cobwebs/tmp over /tmp and get proper behavior if a node goes down.  If I’m betting the farm on GFS, I may as well get something out of it.

I asked jis for a relatively exotic X.509 certificate for cobwebs (listing *.cobwebs.mit.edu as well as cobwebs.mit.edu, cobwebs-apache1.mit.edu, and cobwebs-apache2.mit.edu as common names).  Haven’t heard back.  I may fall back to asking for a more commonplace certificate in order to get things going, but it would be cool if https://username.cobwebs.mit.edu/ could work.

For provisioning user accounts, I decided to make web server node #1 the master and require that provisioning take place there (so it won’t be a high-availability service).  This is because the web server nodes have to all agree on the same uid for an account, and unlike scripts, I’m not determining the uid from a higher authority like hesiod or AFS.   So someone has to be the decider.

While requesting keytabs, I ran across an odd corner case: MIT is still handing out srvtabs (yum, krb4 inertia) and if you convert a srvtab to a keytab with stock RHEL krb5 configuration, you get keytabs with the wrong hostname (cobwebs.athena.mit.edu).  Easily fixed, but perplexing until I figured out what was going on.