Custom Query (196 matches)

(Imported from ​help.mit.edu #431727.)

Now that Nagios doesn't suck, we can actually see the scripts outage caused by the AFS server restart every Sunday morning. This made me realize a few things:

• Our fallback to hodge-podge isn't just an exceptional condition; it happens every week. Thus it's an even worse idea than I thought it was. Viewers will get confused, and search engines may remove pages from their indexes, if they happen to get a 404 error from hodge-podge at the wrong moment.

• Since the heartbeat script is in the scripts locker, the AFS server that serves it (aegisthus) is a single point of failure. Ideally LVS would check multiple heartbeat scripts in lockers on several different AFS servers, and continue routing connections if any of them respond.

(Imported from ​help.mit.edu #432614.)

andersk:

While upgrading packages on scripts4, I received strange errors that I think can be attributed to our shared /tmp directory. We need to find a better solution. (This has made me uncomfortable for a long time, I'm just adding this to our todo list.)

andersk:

This is now one of ghudson's selling points for cobwebs: http://scripts.mit.edu/~ghudson/blog/?p=13 so we should fix it as soon as possible. :-)

Here are some options I see:

1. Keep the NFS solution and try to hack something to solve the failover problem.
2. Unshare /tmp and stop pretending we only have one server.
3. Unshare /tmp, but move PHP sessions and other similar data to some other shared directory (involving one of the other solutions).
4. Put /tmp in AFS somewhere.
5. Experiment with Coda, which I believe is supposed to support what we need.

Thoughts?

I think I'm happiest with either 2 or 3+5. Did we ever find specific examples of popular scripts that depend on a shared /tmp?

jbarnold:

I think that we previously found that some scripts cache data in /tmp, and they expect this data to be either not-there or entirely-up-to-date; they do not expect it to be in an old state.

I think that #2 might hard to get right.

I've considered putting /tmp into AFS, and that option might be the best one.