Opened 12 years ago

Last modified 11 years ago

#209 new enhancement

memcached support

Reported by: geofft Owned by:
Priority: minor Milestone:
Component: web Keywords: service


For certain applications it would be nice to support memcached, a distributed in-memory object caching system.

The primary issue is authentication. Users shouldn't be able to read or write other users' caches, and preferably shouldn't be able to know about them at all. As of IAP 2011, when I last looked at this, there was reasonable authentication support in Membase's fork, in their github repo, but hadn't been rolled into upstream memcached. On the other hand, Membase sucks a little to install if you're not installing everything. I'm not sure if one of the two sides have won this fork battle yet, although I got a fair amount of help from developer Matt Ingenthron on IRC (and then Twitter) when originally approaching this project.

The second issue with support here is consistency in scripts' failover model: if a server goes offline and then comes back, it's better for its caches to expire than for it to report stale data. Similarly, if a user is load-balanced to a different server, and then back to the original one, cached info about the user should not be out-of-date. This all needs to be done in a way that's robust to servers crashing and not causing a single point of failure, and also in a way that doesn't negate memcached's performance gains, e.g., by requiring that data be synced to all the servers.

One possible approach here is to say that we support memcached but only for applications where this is not a problem (e.g., memoizing computations, instead of storing sessions and the like in memory). The drawback is that users (or autoconfiguration scripts, possibly) will assume that we have memcached support and get subtle consistency failures when we have servers behaving abnormally, and never notice anything in the normal case.

Another final concern is memory usage: we don't really have tons of memory on the servers as is, and so we need to be somewhat aggressive about expiring caches, but not so aggressive as to make this pointless.

There should be quite a bit of zlogs from January about this.

Change History (4)

comment:1 Changed 12 years ago by geofft

I think the last place I landed was using moxi, a memcached proxy, to do authn and probably some rewriting to enforce namespaces on keys. I have a VM on XVM with it installed... It also supports SASL. See also

I also swear I took notes on this, but can't find it at the moment.

comment:2 Changed 11 years ago by adehnert

  • Keywords service added

comment:3 Changed 11 years ago by ezyang

  • Milestone The Distant Future deleted

comment:4 Changed 11 years ago by ezyang

  • Priority changed from normal to minor
Note: See TracTickets for help on using tickets.