Greg Hudson’s MIT blog


Athena: What does all this crap do?

Posted in Athena by ghudson on the September 20th, 2007

To plan out the work of the Athena 10 release, I needed to compose a fairly comprehensive list of things people expect from Athena machines beyond the base platform.

Here’s the list.  For every one of those, I get to detail how we handle it in the current Athena release and how I expect to handle it under Ubuntu without building so much third-party crud.  (Or, in a few cases, that we expect to discard that feature.)

XMPP: Client review

Posted in XMPP by ghudson on the September 10th, 2007

I’ve written up a brief review of XMPP clients to help people looking for specific client features for specific platforms.  For instance, if you want to be able to configure chatrooms on OS X, your choices appear to be Spark and Coccinella.

My coworker’s logging bot authorization problem appears to be an Openfire server bug: if the client doesn’t request an authorization name, the server attempts to use username@ATHENA.MIT.EDU as the JID node, which obviously won’t work.  This is fixed in the upstream trunk code base (they’ve added a mapping function) and is due out in the 3.4 release.  This problem is generally easily worked around in the client code when you’re in a position to make such changes.  In addition to my coworker’s Perl code, the problem also affects the Cyrus SASL support in Pidgin.

XMPP: More issues

Posted in XMPP by ghudson on the August 23rd, 2007

* During the debugging of the second cutover, I noticed I had generated invalid XML for vCard entries transferred from the jabberd2 database.  I fixed that before anyone had much chance to notice.

* If you haven’t changed your password since July 2000, you can’t log in.  The same problem happens with win.mit.edu; it’s due to a lack of krb4 support in the relevant code path.  I noticed this problem on the test server (I hadn’t changed my password since then myself), but never had a context to document it in.  At least one user has run into the problem since the cutover.

* People haven’t been able to create new roster entries or queue online messages, due to a bug in my DB conversion script.  I hadn’t set up the “next ID” field for roster items or offline messages, so the server was trying to create new entries with old IDs and running afoul of primary key constraints.  Whoops.  Fixed now.
* A co-worker’s logging bot can’t connect due to a SASL authorization error.   The bot uses XML::Stream but it doesn’t seem to be the same problem as Barnowl had.  I’m still collecting information.

* Our server can’t communicate with gizmoproject due to some kind of dialback issue.  Googling for “gizmoproject openfire” doesn’t immediately give me the answer, so I need to look into that further myself.

Minerva: mod_authz_mitgroup

Posted in minerva by ghudson on the August 22nd, 2007

I’ve written an authz module for authenticating against MIT groups on a standalone Apache 2.x web server.  It uses ldap.mit.edu as a back end currently.

In order to use it, you first need an Apache auth mechanism which produces a username like ghudson@mit.edu or just ghudson.  The simplest way I know of to do that is to use mod_auth_sslcert from the scripts.mit.edu project.  A future option will be to use Shibboleth, which is expected to be piloted soon; I haven’t tried that yet (but I plan to).

So, the details:

1. Setting up mod_auth_sslcert (until Shibboleth becomes an option)
I’ve stashed a copy of the source at:

http://web.mit.edu/minerva-dev/src/mod_auth_sslcert/mod_auth_sslcert.c

or you can grab it from the scripts.mit.edu repository.  Make sure you have the appropriate httpd devel package installed for your OS (or have your path set properly if you built httpd from source) and run:

apxs -c -i -a mod_auth_sslcert.c

which will compile the source, install it in the httpd modules directive, and add a LoadModule directive to your httpd.conf.  You then configure it in some suitably global section of httpd.conf:

AuthSSLCertVar SSL_CLIENT_S_DN_Email

which will produce usernames like ghudson@MIT.EDU.  If for other reasons you’d rather the username look like just ghudson, you can do that with:
AuthSSLCertStripSuffix “@MIT.EDU”

You also need the web server configured to be able to verify MIT client certificates (see http://web.mit.edu/apache-ssl/www/README.certificate for instructions on getting a server certificate; those are written for Apache 1.3, so you’ll probably need to store the certificate elsewhere for your Apache 2.x server), and to have an area of your server configured with:

SSLVerifyClient require

It’s traditional to use a separate port for the portion of the web space which requires client certificates, but with Apache 2.x you can actually just put it in a directive and the server will do an SSL renegotiation once it detects that the requested URL is part of the affected area.

2. Setting up mod_authnz_mitgroup itself

Get the source from:

http://web.mit.edu/minerva-dev/src/mod_authz_mitgroup/mod_authz_mitgroup.c

and install it with:

apxs -c -i -a mod_authz_mitgroup.c

In a .htaccess file or Location directive for a resource you want to control, you would restrict to a specific group with:

AuthType SSLCert
require mitgroup minerva-dev

The first line is specific to using mod_auth_sslcert for authentication; with Shibboleth you’d do something different.
LDAP queries performed by this module will be cached for ten minutes by default.  You can change that with the LDAPCacheTTL directive, e.g. “LDAPCacheTTL 300″ for five-minute caching.

3. Doing the same thing with mod_authnz_ldap
If you’re willing to accept a hackier syntax and a closer tie-in to LDAP as a back end, you can do the same thing with mod_authnz_ldap, which is distributed with httpd 2.2.  You still need mod_auth_sslcert or equivalent to get the username set up.  Your per-resource access restriction directives would look like:

AuthType SSLCert
AuthLDAPUrl ldap://ldap.mit.edu/dc=mit,dc=edu?mail
AuthLDAPGroupAttribute uniquemember
require ldap-group cn=minerva-dev,ou=groups,dc=mit,dc=edu

(If you configured mod_auth_sslcert to strip the “@MIT.EDU” suffix, remove the “?mail” directive at the end of the URL so that mod_authnz_ldap uses the default field instead, which is “uid”.)  You can put AuthLDAPGroupAttribute in a global place, but don’t put AuthLDAPUrl there or every resource will become inaccessible if mod_authnz_ldap can’t determine the user’s DN.

XMPP: Openfire deployment

Posted in XMPP by ghudson on the August 21st, 2007

jabber.mit.edu now runs Openfire instead of jabberd2. This hasn’t been widely publicized because we don’t really have an announcement list or other public infrastructure related to that service.

Other than a somewhat richer set of chatroom features, Openfire doesn’t directly offer much to XMPP users that they didn’t have before. But it will hopefully allow us to offer new features in the future such as presence via the web, click-to-chat for support organizations (the MIT Libraries may be piloting this soon), and possibly some kind of SIP integration.

This changeover has been a big part of what I’ve been working on for the past few months. Some issues related to the deployment:

* It actually went live twice. The first time (last Wednesday night), it had to be backed out because GSSAPI authentication wasn’t working. There’s a property you have to set if the hostname doesn’t match the XMPP domain, which is true on the production server but not the test server. We deployed it again late Thursday night and I fixed the problem in the wee hours of the morning.

* Roster items and queued messages were migrated across systems, but chatroom settings were not. This led to a lot of nonfunctional chatrooms: rooms would get into a state where someone had created the room but not configured it, so it would be “locked,” inaccessible to other people. This probably happened most frequently due to clients auto-joining rooms upon reconnecting, but could also happen when people joined rooms from clients which didn’t support chatroom configuration. I fixed most of the affected rooms by hand.

* The new chatroom configuration dialog uses a multi-value list field for “Roles for which presence is broadcast.” Unfortunately, Gaim/Pidgin has a bug in the handling of multi-value list fields, such that it smashes the list of pre-selected items down to the first item. So every time you configure a chatroom, you have to manually reselect all three options or the room reverts to only displaying presence for moderators. I debugged this problem yesterday and submitted bug reports. We’ll fix the Athena deployment of Gaim, but that probably only accounts for a minority of Gaim usage.

* Barnowl couldn’t connect to the new server. I debugged that problem Friday; it was due to a bug in XML::Stream which would close the stream if the server reused TLS session IDs. Or something. Once I figured out where the code was doing the wrong thing, I was able to google for the answer and find a fix on the web, which I think has now been deployed.

XMPP: Chatroom aggregator idea

Posted in XMPP by ghudson on the June 20th, 2007

The problem: in the Zephyr culture, people join a lot of different chatrooms (or “triplets”), and the clients make this reasonable.  I’m in almost 300 at the moment.  In every other IM culture including XMPP, that’s highly unusual and the clients don’t support it reasonably.  You’d have to have 300 open windows or 300 tabs to be a member of all of those rooms, and keeping up with the incoming traffic would be impossible.

Until recently, my best idea for resolving this issue was to get the clients to change or to make Zephyr clients work for XMPP.  Changing existing clients is slow and failure-prone and it’s not clear how far it can go–the best I can think of is to make it possible to be in a zero-traffic chatroom without a visible tab, and have the tab pop up when traffic comes in (or if you ask the client to join the chatroom when you’re already in it, in case you want to proactively send a message there).  Better than what they do now, but still not great.

Making Zephyr clients work for XMPP is happening, but at best it’s a fractured solution–people using the more traditional GUI clients like Pidgin aren’t going to be able to join in the many-room culture even if people using barnowl can.

At Monday’s meeting, Matt suggested maybe the problem could be solved in the server, somehow.  This led me to think about a server-side chatroom aggregator.  To the client, it would look like a single chatroom, but it would maintain subscriptions to other chatrooms and would display messages from all of them.  It wouldn’t be as flexible as good client-side support for many low-traffic chats, but it would be accessible to any client.

Design questions I came up with yesterday and today (“MUC” means “Multi-User Chat” and is the technical term for a chatroom in XMPP):

  • To the real MUC, does the user appear to have joined from the user’s real JID (ghudson@mit.edu) or from the aggregator’s JID?  The former is more transparent, but probably requires the implementation to be an Openfire server plugin rather than a proper network component.  And it’s more hackish from a protocol perspective.
  • Can a single user have multiple aggregators (e.g. one to aggregate work chats and one to aggregate personal chats) or only one?
  • What do the JIDs of these aggregators look like?  Do different users’ aggregators have to have distinct JIDs?  Do different resources for the same user (ghudson@mit.edu/Home and ghudson@mit.edu/Work) share the same aggregators, or do they each have a separate namespace?
  • Do aggregators have persistent configuration data, such as a list of real MUCs to auto-join, or is it all soft state which disappears when no user clients are joined to the aggregator?
  • Can the aggregator be configured to remain subscribed to the real MUCs while the user is not joined to the aggregator?
  • When the aggregator joins a real MUC, it will receive recent history from that MUC.  Does that need to be handled in any special way, or should it simply be sent through to the user?
  • How does the aggregator convey room presence information to the user?  Does it try to aggregate the room list of all the rooms and present that in a unified room list, or does it convey an empty room list and make the room lists of the real MUCs available through query commands?
  • Will the aggregator have any kind of logging functionality which users could query through commands?
  • What is the display format of messages from the real MUCs?  Will it be configurable?  How pretty can it be made to look in existing XMPP clients?  Will real MUCs have associated nicknames to avoid having to present a long source identifier like help@mit.edu with each message?
  • What is the input format of commands to add or remove real MUCs or send messages to real MUCs?

I can’t run off and implement such a thing because I’m busy with other stuff.  But I thought I would throw the idea and design questions out there while I had them in my head.

XMPP: General update on boring stuff

Posted in XMPP by ghudson on the June 20th, 2007

A little background since this is my first XMPP post: XMPP is the protocol behind the federated chat network commonly known as Jabber (although “Jabber” is a trademark of Jabber Inc. and people don’t always feel right about using that term any more).  Google Talk uses it.  MIT currently runs an XMPP server using the jabberd2 code base, which isn’t really maintained any more.  In a month or so we expect to switch to running Openfire.  Right now our XMPP service isn’t used by as many people as use our older in-house Zephyr messaging service (to the best of my knowledge, anyway), but we’d like Zephyr to die out in favor of XMPP some day.
I haven’t had much to write about lately since I’ve been working on something intensely uninteresting: scripts to convert roster data and queued offline messages and vCard data between jabberd2 and Openfire database formats.  Since I know only a little more than jack about databases, it was slow going.  I think I’m done now.

On Monday I went to a meeting with Matt Tucker, the CTO of Jive Software (the people who make Openfire).  It was a useful meeting, and it prodded me to check out the changes Openfire has made to SASL and GSSAPI authentication.  They made me realize there’s a way I could do account auto-creation for GSSAPI authentication without modifying the core Openfire code, which is neat.  But it looks like they’ve done a huge rework destined for the 3.4 release which will make me redo all my work.  I’m not sure how that affects our deployment timetable.

That’s the summary of the boring stuff in my work life.  During the meeting Matt and I brainstormed an idea related to multi-user chat while is more interesting; I’ll write that up in a separate entry.

Athena: Ubuntu thoughts and experiences

Posted in Athena by ghudson on the June 12th, 2007

I’m working under Ubuntu 7.04 now.  In the past I had difficulty installing it on my 17″ Macbook Pro, but I worked past that.  (Problem: the Live CD tries to use a non-encumbered graphics driver which black-screens and freezes upon starting the X server.  If you specify a resolution at install time, the X server instead crashes on startup, giving you access to a text console from which you can install and configure the encumbered fglrx package.)

It’s good except gnome-terminal appears to be buggy (has some rendering errors from time to time, which go away when you force a redraw) and occasionally text gets pasted or other mysterious things happen when I don’t want them to.  I think the latter may be due to the trackpad support causing mouse events when I do something ham-handed to the keyboard; I have yet to figure out exactly what I’m doing to cause it.  Oh, and Firefox sometimes freezes up after Flash content is displayed.
I was reminded recently that Ubuntu is not a no-brainer choice over Debian, so I thought about that a bit.  I looked over some of Bill Cattey’s Linux survey data in greater detail (it’s still not ready for release) and determined that in fact, Ubuntu does appear to have a larger user community than Debian at MIT, so I’m still comfortable with that direction.  I have some specific concerns about Debian as opposed to Ubuntu:

  • It tends to have a long release cycle, which in turn often encourages people to use less stable variants in order to have access to modern software.
  • Its purist approach would force us to do more work assembling the encumbered components needed to make it work acceptably to users.  (That’s my impression, anyway.)

So, what does an automated RHEL to Ubuntu upgrade look like?  The package manager isn’t going to do the job, so it has to be done at the file level (although the package managers can be used on each end to assemble a list of files that need to be considered).  Athena has past experience with file-level OS upgrades; we did them with BSD Unix, Ultrix, and Solaris.

We had enough problems with Solaris OS upgrades that we eventually started doing them from a miniroot installed in the swap partition rather than trying to do them while the target system is booted and running.  I’m not sure if we want to invent that machinery for Linux; it’s harder to do on PC hardware and it wasn’t a trivial amount of work on Sun hardware.  It might be easier to just work around the problems of doing a live upgrade for this one-time occurrance.

cobwebs: Close to a first draft

Posted in minerva by ghudson on the May 25th, 2007

Frustratingly, I’m having to pull myself away from cobwebs work (in favor of Openfire work) just as I feel like I’m close to having a configuration I can show around and start getting feedback on.  Mostly, I need to set up SSL, write scripts to provision new user accounts, and write a little placeholder front-page content.  If anyone is really interested, I could manually set up some accounts and let them poke around.  Some assorted ramblings below.
LVS was a little confusing to set up, partly due to out-of-date documentation.  Like scripts, cobwebs will use the “direct” (or “gateway”) packet-forwarding mode, which means the web server nodes have an arpless interface alias for the virtual IP address which they share with the arpful interface alias on the LVS node.  Setting up an arpless interface alias in Linux is sort of a black art; the mechanism has changed across kernel releases and it still isn’t all that pretty.  Today you can “ifconfig interfacename -arp” which might or might not scope properly for an interface alias; regardless, the RHEL 5 init infrastructure doesn’t do that if you say ARP=no in an ifcfg file, but instead does “ip link set dev devname arp no” on the parent interface, which is the wrong scope.  The workaround is to stick your arpless interface aliases on the lo device instead of the eth0 device, since it doesn’t matter if the true lo device arps or not.  Also like scripts, cobwebs will use the “source hash” scheduling mechanism, which will tend to route the same requestor IP address to the same server.  That’s probably good for performance over GFS, but it might actually be better to route requests for the same domain (username.cobwebs.mit.edu) to the same server.  That’s much harder to do, though.
Out of curiosity, I looked into how scripts.mit.edu was dealing with sharded /tmp; apparently it isn’t ideal if a node goes down, and having an unshared /tmp might be better even if it breaks the illusion of a single machine.  On cobwebs, I can just bind-mount /home/cobwebs/tmp over /tmp and get proper behavior if a node goes down.  If I’m betting the farm on GFS, I may as well get something out of it.

I asked jis for a relatively exotic X.509 certificate for cobwebs (listing *.cobwebs.mit.edu as well as cobwebs.mit.edu, cobwebs-apache1.mit.edu, and cobwebs-apache2.mit.edu as common names).  Haven’t heard back.  I may fall back to asking for a more commonplace certificate in order to get things going, but it would be cool if https://username.cobwebs.mit.edu/ could work.

For provisioning user accounts, I decided to make web server node #1 the master and require that provisioning take place there (so it won’t be a high-availability service).  This is because the web server nodes have to all agree on the same uid for an account, and unlike scripts, I’m not determining the uid from a higher authority like hesiod or AFS.   So someone has to be the decider.

While requesting keytabs, I ran across an odd corner case: MIT is still handing out srvtabs (yum, krb4 inertia) and if you convert a srvtab to a keytab with stock RHEL krb5 configuration, you get keytabs with the wrong hostname (cobwebs.athena.mit.edu).  Easily fixed, but perplexing until I figured out what was going on.

cobwebs: PHP session directories and related stuff

Posted in minerva by ghudson on the May 15th, 2007

Since I last posted, I’ve been working on the cobwebs db guest image and learning about MySQL.  There’s not much to say about that since it pretty much just works.  I need to write a little bit of machinery to create users and to create databases for users, but it doesn’t seem hard.  For the moment I’m hand-creating databases as I need them.
I’ve also tried deploying a few PHP web apps onto my test apache instance.  The results are pretty encouraging.  WordPress and Drupal worked with no problems.  I do notice that all these web apps assume that the vast majority of web host sites use “localhost” as the MySQL host.  That seems a little odd; I would expect most web hosts to want to separate out the database server pretty quickly.  I wonder if they use redirectors.
When I tried out MediaWiki, I ran into a small hurdle: it wants the PHP session directory to be writable, which it’s not since /etc/php.ini sets save_path to /var/lib/php/session which is writable by group apache but not by random user IDs.  I can configure PHP’s session.save_path in /etc/php.ini to point to a world-writable sticky directory like /tmp.  The planned default umask (072; all users share the same group) would work to protect session data from other users, I think.  But it feels unsafe; I’d rather have a separate session directory for each user.  I don’t think I can do that in a single global php.ini; the documentation and source code don’t reveal any kind of substitution going on in session.save_path which I could use to put in the current user’s home directory or some such.

scripts.mit.edu creates a php.ini alongside its web app auto-deployments which sets session.save_path.  I could do the same, but that only works for web apps I have auto-installers for.  I’d like stuff to work out of the box as much as possible.

On a similar note, how does scripts.mit.edu configure PHP to automatically notice php.ini files dropped in alongside PHP scripts?  I couldn’t immediately figure that out, and it didn’t seem to happen on its own.

Next Page »