Notes for repository creation
-----------------------------
-See TODO for full instructions.
+See documentation for full instructions.
- You are not going to be able to doctor a perfect repository
from scratch.
should instantiate a .wizard directory automatically (which should
be ignored) that doesn't need to be created in repositories.
- * wizard.scripts is the location that all Scripts specific code
- will eventually live.
-
- * [HOOK] wizard.sql does a Script's specific call to get SQL
- information if no SQL credentials are explicitly passed.
-
- * In wizard.util
- - [HOOK] get_dir_owner performs a PTS query if normal pwd
- querying fails
- - [HOOK] get_operator_name_from_gssapi requires a patch to SSH
- that we patched support for
- - [HOOK] get_operator_info consults Hesiod for more information
- based on a username.
- - [HOOK] set_author_env guesses emails using scripts.mit.edu
- addresses
-
* [HOOK] wizard.install contains strategies for guessing variables
for an installation that are Scripts specific
to the installation page.
- The need to run egg_info can make things a little fragile when pulling
if you forget. How can we make this less easily forgotten?
+- wizardResolve* files seem to get left in tmp en-mass, and we don't
+ know why.
- Plugin-ify! Hook-ify! In more detail, applications should all be moved
out into plugins, scripts specific behavior should be moved into
to A question is whether or not sending mail actually helps us:
many users will probably have to come back to us for help; many
other users won't care.
+ - Whatever happens here should be used to improve user.email()
[ XXX: metadata.rst ]
--- /dev/null
+:mod:`wizard.plugin`
+====================
+
+.. automodule:: wizard.plugin
+
+Functions
+---------
+
+.. autofunction:: hook
---------
.. autofunction:: quota
+.. autofunction:: passwd
+.. autofunction:: operator
+.. autofunction:: pwnam
+.. autofunction:: email
+
+Classes
+-------
+
+.. autoclass:: Info
.. autofunction:: chdir
.. autofunction:: get_exception_name
.. autofunction:: get_dir_uid
-.. autofunction:: get_dir_owner
.. autofunction:: get_revision
-.. autofunction:: get_operator_info
-.. autofunction:: get_operator_name_from_gssapi
.. autofunction:: get_operator_git
.. autofunction:: set_operator_env
.. autofunction:: set_author_env
plugin authors should take care to "do no harm" if a plugin is unable to
do anything useful.
+
+
Registering plugins without eggs
--------------------------------
to be the plugins that are to be registered, and then run ``python
setup.py egg_info``.
+
+
``wizard.app``
--------------
For more information on how to create an application, check
:doc:`Creating a Repository <create>`.
-.. todo::
- Plugins marked with prototype todo messages are not complete yet.
.. _wizard.strategy:
Used during installation to automatically determine values for
installation parameters the user may have omitted.
+
+
.. _wizard.sql.auth:
``wizard.sql.auth``
returning it is the expected modus operandi of a plugin. The function
that runs this plugin is :func:`wizard.sql.auth`.
+.. note::
+
+ If Wizard is able to determine the login credentials from the
+ application's source files, these plugins will not be run.
+
+
+
.. _wizard.deploy.web:
``wizard.deploy.web``
for an application. The function that runs this plugin is
:func:`wizard.deploy.web`.
+.. note::
+
+ If Wizard is able to determine the web URL from the application's
+ source files or database, these plugins will not run.
+
+
+
.. _wizard.user.email:
``wizard.user.email``
Used to determine a user's email address. By default, we use a
heuristic approach; if your system offers this information in a
-canonical form we recommend taking advantage of it.
+canonical form we recommend taking advantage of it. Plugin should be a
+function that takes a single required argument ``name`` and returns a
+string email address corresponding to that user, or ``None`` if it
+was unable to determine an email address. The function that runs this
+plugin is :func:`wizard.user.email`.
+
+.. note::
+
+ If the :envvar:`EMAIL` environment variable is set and we are
+ requesting the current user's email, these plugins will not run.
+
-.. todo:: Prototype
.. _wizard.user.operator:
------------------------
If Wizard is running as root or as a superuser masquerading as another
-user, it is still useful to record who was in front of the screen. By
-default, we check :envvar:`SUDO_USER`; if your system gives more
+user, it is still useful to record who was in front of the screen. The default
+fallback is checking :envvar:`SUDO_USER`; if your system gives more
information we recommend taking advantage of it.
-.. todo:: Prototype
+
.. _wizard.user.passwd:
If Wizard is running on a system with a network filesystem (such as NFS
or AFS), the standard system ``passwd`` database may not actually
resolve the username from a UID. Use this hook to implement an
-alternative ``passwd`` lookup.
+alternative ``passwd`` lookup. Plugin should be a function that takes a
+directory (to determine filesystem off of) and a user ID (to perform the
+lookup on). The directory is guaranteed to be a real path. The plugin
+should return an instance of :class:`wizard.user.Info` or ``None``, if
+it was unable to perform the lookup. The function that runs this plugin
+is :func:`wizard.user.passwd`.
+
-.. todo:: Prototype
.. _wizard.user.quota:
``wizard.user.quota``
---------------------
-Plugin for :func:`wizard.user.quota`. Wizard has safeguards for
-avoiding exceeding user quotas when an automated upgrade is being
-performed. Unfortunately, methods for enforcing quotas are frequently
-highly system dependent. Use this hook to implement quota usage and
-limit reporting. Plugin should be a function that takes a single
-required argument ``dir``, which is the directory to determine the quota
-for, and return a tuple ``(quota usage, quota limit)`` or ``(0, None)``
-if it could not determine quota. The function that runs this plugin is
-:func:`wizard.user.quota`.
+Wizard has safeguards for avoiding exceeding user quotas when an
+automated upgrade is being performed. Unfortunately, methods for
+enforcing quotas are frequently highly system dependent. Use this hook
+to implement quota usage and limit reporting. Plugin should be a
+function that takes a single required argument ``dir``, which is the
+directory to determine the quota for, and return a tuple ``(quota usage,
+quota limit)`` or ``None`` if it could not determine quota. The
+function that runs this plugin is :func:`wizard.user.quota`.
packages = setuptools.find_packages(),
entry_points = {
'wizard.user.quota': 'scripts = wizard_scripts:user_quota',
+ 'wizard.user.email': 'scripts = wizard_scripts:user_email',
+ 'wizard.user.operator': 'scripts = wizard_scripts:user_operator',
+ 'wizard.user.passwd': 'scripts = wizard_scripts:user_passwd',
'wizard.deploy.web': 'scripts = wizard_scripts:deploy_web',
'wizard.sql.auth': 'scripts = wizard_scripts:sql_auth',
}
import errno
import wizard
-from wizard import shell, util
+from wizard import shell, util, user
def deploy_web(dir):
# try the directory
except shell.CallError:
pass
return None
+
+def user_email(name):
+ # XXX: simplistic strategy which doesn't work most of the time
+ return "%s@scripts.mit.edu" % name
+
+def user_operator():
+ """
+ Returns username of the person operating this script based
+ off of the :envvar:`SSH_GSSAPI_NAME` environment variable.
+
+ .. note::
+
+ :envvar:`SSH_GSSAPI_NAME` is not set by a vanilla OpenSSH
+ distributions. Scripts servers are patched to support this
+ environment variable.
+ """
+ principal = os.getenv("SSH_GSSAPI_NAME")
+ if not principal:
+ return None
+ instance, _, _ = principal.partition("@")
+ if instance.endswith("/root"):
+ username, _, _ = principal.partition("/")
+ else:
+ username = instance
+ return username
+
+def user_passwd(dir, uid):
+ # XXX: simplistic heuristic for detecting AFS. The correct thing to
+ # is either to statfs and match magic number, use one of the
+ # vos tools or check mounted directories.
+ if not dir.startswith("/afs/"):
+ return None
+ try:
+ result = shell.eval("hesinfo %d uid", uid)
+ except shell.CallError:
+ return None
+ name, password, uid, gid, gecos, homedir, shell = result.split(":")
+ realname = gecos.split(",")[0]
+ return user.Info(name, uid, gid, realname, homedir, shell)
# APPVERSION is directly interpolated into bash, so it can represent
# 0 arguments.
if [ "$VERSION" == "head" ]; then
- APPVERSION=""
+ APPVERSION="$APP"
else
APPVERSION="$APP-$VERSION-scripts" # XXX incorrect if a -scripts2 version exists
fi
This protects against malicious mountpoints, and is roughly equivalent
to the suexec checks.
"""
+ # XXX: this is a smidge unfriendly to systems who haven't setup
+ # nswitch.
try:
uid = util.get_dir_uid(location)
real = os.path.realpath(location)
sh.call("git", "checkout", ".scripts")
logging.info("Diffstat:\n" + sh.eval("git", "diff", "--stat"))
# commit user local changes
- message = "Autoinstall migration of %s locker.\n\n%s" % (util.get_dir_owner(), util.get_git_footer())
+ message = "Autoinstall migration.\n\n%s" % util.get_git_footer()
util.set_git_env()
try:
message += "\nMigrated-by: " + util.get_operator_git()
options, args = parse_args(argv, baton)
dir = os.path.abspath(args[0]) if args else os.getcwd()
r = user.quota(dir)
- if r[0] is None or r[1] is None:
+ if r is None:
sys.exit(1)
print "%d %d" % r
USED AVAIL
-In bytes. Returns an exit code of 1 if quota could
+in bytes. Returns an exit code of 1 if quota could
not be determined."""
parser = command.WizardOptionParser(usage)
options, args = parser.parse_all(argv)
sys.stderr.write("Traceback:\n (n/a)\nAlreadyUpgraded\n")
sys.exit(2)
def preflightQuota(self):
- usage, limit = user.quota()
- if limit is not None and (limit - usage) < buffer:
- logging.info("preflightQuota: limit = %d, usage = %d, buffer = %d", limit, usage, buffer)
- raise QuotaTooLow
+ r = user.quota()
+ if r is not None:
+ usage, limit = r
+ if limit is not None and (limit - usage) < buffer:
+ logging.info("preflightQuota: limit = %d, usage = %d, buffer = %d", limit, usage, buffer)
+ raise QuotaTooLow
def merge(self):
if not self.options.dry_run:
for f in added_files:
if os.path.lexists(f): # broken symbolic links count too!
shell.call("git", "add", f)
- message = "Pre-commit of %s locker before autoinstall upgrade.\n\n%s" % (util.get_dir_owner(), util.get_git_footer())
+ message = "Pre-commit before autoinstall upgrade.\n\n%s" % util.get_git_footer()
try:
message += "\nPre-commit-by: " + util.get_operator_git()
except util.NoOperatorInfo:
self.wc.invalidateCache()
self.wc.verifyVersion()
def postflightCommitMessage(self):
- message = "Upgraded autoinstall in %s to %s.\n\n%s" % (util.get_dir_owner(), self.version, util.get_git_footer())
+ message = "Upgraded autoinstall to %s.\n\n%s" % (self.version, util.get_git_footer())
try:
message += "\nUpgraded-by: " + util.get_operator_git()
except util.NoOperatorInfo:
pre_size = int(open(os.path.join(self.temp_wc_dir, ".git/WIZARD_SIZE"), "r").read())
post_size = util.disk_usage(self.temp_wc_dir)
backup = self.prod.backup(self.options)
- usage, limit = user.quota()
- if limit is not None and (limit - usage) - (post_size - pre_size) < buffer:
- shutil.rmtree(os.path.join(".scripts/backups", shell.eval("wizard", "restore").splitlines()[0]))
- raise QuotaTooLow
+ r = user.quota()
+ if r is not None:
+ usage, limit = r
+ if limit is not None and (limit - usage) - (post_size - pre_size) < buffer:
+ shutil.rmtree(os.path.join(".scripts/backups", shell.eval("wizard", "restore").splitlines()[0]))
+ raise QuotaTooLow
return backup
def upgrade(self, backup):
Performs a commit of changes performed during configuration of an install
with an appropriate logfile message.
"""
- message = "Autoinstall configuration of %s locker.\n\n%s" % (util.get_dir_owner(), util.get_git_footer())
+ message = "Autoinstall configuration.\n\n%s" % util.get_git_footer()
util.set_git_env()
try:
message += "\nConfigured-by: " + util.get_operator_git()
--- /dev/null
+"""
+Convenience methods for managing plugins.
+"""
+
+import pkg_resources
+
+def hook(name, args):
+ """
+ Runs plugins named ``name`` for this function. Returns ``None`` if
+ all plugins return ``None``, otherwise returns the result of the
+ first plugin to result that is not ``None``. Assumes that plugins
+ are simple functions that take the arguments ``args``.
+ """
+ for entry in pkg_resources.iter_entry_points(name):
+ func = entry.load()
+ r = func(*args)
+ if r is not None:
+ return r
+ return None
for entry in pkg_resources.iter_entry_points("wizard.sql.auth"):
func = entry.load()
r = func(copy.copy(url))
- print r
if r is not None:
return r
env_dsn = os.getenv("WIZARD_DSN")
"""
-Module for querying information about users. This mostly asks plugins for
-the extra information, and falls back to using a default that should work
-on most systems (but by no means all systems.)
+Module for querying information about users. This mostly asks plugins
+for the extra information, and falls back to using a default that should
+work on most systems (but by no means all systems.)
"""
import pkg_resources
import os
+import socket
+import logging
+import pwd
+
+from wizard import plugin
def quota(dir=None):
"""
- Returns a tuple (quota usage, quota limit). Returns ``(0, None)`` if
+ Returns a tuple (quota usage, quota limit). Returns ``None`` if
the quota usage is unknown. If ``dir`` is omitted, the current
working directory is assumed. Value returned is in bytes.
"""
if dir is None:
dir = os.getcwd()
- unknown = (0, None)
- for entry in pkg_resources.iter_entry_points("wizard.user.quota"):
- func = entry.load()
- r = func(dir)
- if r != unknown:
- return r
- return unknown
+ return plugin.hook("wizard.user.quota", [dir])
+
+def email(name=None):
+ """
+ Converts a username into an email address to that user. If you have
+ a UID, you will have to convert it into a username first. If no
+ canonical source of information is found, an heuristic approach
+ will be used. If ``name`` is ``None``, the current user will be
+ used unless it is root, in which case :func:`operator` is tried
+ first to determine the real current user.
+
+ This function implements a plugin interface named
+ :ref:`wizard.user.email`.
+ """
+ if name is None:
+ logging.info("wizard.user.email: Determining email for current user")
+ env_email = os.getenv("EMAIL")
+ if env_email is not None:
+ logging.info("wizard.user.email: Used environment email %s", env_email)
+ return env_email
+ name = operator()
+ # run plugins
+ r = plugin.hook("wizard.user.email", [name])
+ if r is not None:
+ return r
+ # guess an email
+ try:
+ mailname = open("/etc/mailname").read()
+ except OSError:
+ mailname = socket.getfqdn()
+ return name + "@" + mailname
+
+def operator():
+ """
+ Determines the username of the true person who is running this
+ program. If the process's real uid is nonzero, just do a passwd
+ lookup; otherwise attempt to figure out the user behind the root
+ prompt some other way.
+
+ This function implements a plugin interface named
+ :ref:`wizard.user.operator`.
+ """
+ uid = os.getuid()
+ if uid:
+ pwdentry = pwd.getpwuid(uid)
+ return pwdentry.pw_name
+ # run plugins
+ r = plugin.hook("wizard.user.operator", [])
+ if r is not None:
+ return r
+ # use SUDO_USER
+ sudo_user = os.getenv("SUDO_USER")
+ if not sudo_user:
+ return None
+ pwdentry = pwd.getpwnam(sudo_user)
+ return pwdentry.pw_name
+
+def passwd(path=None, uid=None):
+ """
+ Returns a passwd-like entry (a :class:`Info` object) corresponding
+ to the owner of ``path``. If ``uid`` is specified, ``path`` is used
+ solely to determine the filesystem ``uid`` was determined from. It
+ will fall back to the local passwd database, and return ``None``
+ if no information is available. If ``path`` is omitted, it will
+ fall back to the current working directory.
+
+ This function implements a plugin interface named
+ :ref:`wizard.user.passwd`.
+ """
+ if path is None:
+ path = os.getcwd()
+ path = os.path.realpath(path)
+ if not uid:
+ uid = os.stat(path).st_uid
+ r = plugin.hook("wizard.user.passwd", [path, uid])
+ if r is not None:
+ return r
+ try:
+ return Info.pwentry(pwd.getpwuid(uid))
+ except KeyError:
+ return None
+
+def pwnam(name):
+ """
+ This user converts a username into a :class:`Info` object using
+ *only* the local password database.
+ """
+ return Info.pwentry(pwd.getpwnam(name))
+
+class Info(object):
+ """
+ Object containing information describing a user. It is analogous to
+ passwd, but has dropped the password field and dedicated the
+ ``gecos`` field for real name information.
+
+ .. note::
+
+ If a platform does not support retrieving information about a
+ field, it may have the value ``None``.
+ """
+ #: Login name
+ name = None
+ #: User ID
+ uid = None
+ #: Group ID
+ gid = None
+ #: Real name
+ realname = None
+ #: Home directory
+ homedir = None
+ #: Default command interpreter
+ shell = None
+ @staticmethod
+ def pwentry(pwentry):
+ return Info(pwentry.pw_name, pwentry.pw_uid, pwentry.pw_gid,
+ pwentry.pw_gecos.split(",")[0], pwentry.pw_dir, pwentry.pw_shell)
+ def __init__(self, name, uid, gid, realname, homedir, shell):
+ self.name = name
+ self.uid = uid
+ self.gid = gid
+ self.realname = realname
+ self.homedir = homedir
+ self.shell = shell
+ self._email = None
+ @property
+ def email(self):
+ """The email of this user, calculated on the fly."""
+ if self._email is None:
+ self._email = email(self.name)
+ return self._email
import string
import wizard
+from wizard import user
class ChangeDirectory(object):
"""
"""Finds the uid of the person who owns this directory."""
return os.stat(dir).st_uid
-def get_dir_owner(dir = "."):
- """
- Finds the name of the locker this directory is in.
-
- .. note::
-
- When querying AFS servers, this function only works if
- you're on a Scripts server (which has the correct passwd
- database) or if you're on a Debathena machine.
- """
- uid = get_dir_uid(dir)
- try:
- pwentry = pwd.getpwuid(uid)
- return pwentry.pw_name
- except KeyError:
- # do an pts query to get the name
- return subprocess.Popen(['pts', 'examine', str(uid)], stdout=subprocess.PIPE).communicate()[0].partition(",")[0].partition(": ")[2]
-
def get_revision():
"""Returns the commit ID of the current Wizard install."""
# If you decide to convert this to use wizard.shell, be warned
wizard_git = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), ".git")
return subprocess.Popen(["git", "--git-dir=" + wizard_git, "rev-parse", "HEAD"], stdout=subprocess.PIPE).communicate()[0].rstrip()
-def get_operator_info():
- """
- Returns tuple of ``(realname, email)`` about the person running
- the script. If run from a scripts server, get info from Hesiod.
- Otherwise, use the passwd database (email generated probably won't
- actually accept mail). Useful when generating commit messages.
- """
- username = get_operator_name_from_gssapi()
- if username:
- # scripts approach
- hesinfo = subprocess.Popen(["hesinfo", username, "passwd"],stdout=subprocess.PIPE).communicate()[0]
- fields = hesinfo.partition(",")[0]
- realname = fields.rpartition(":")[2]
- return realname, username + "@mit.edu"
- else:
- # more traditional approach, but the email probably doesn't work
- uid = os.getuid()
- if not uid:
- # since root isn't actually a useful designation, but maybe
- # SUDO_USER contains something helpful
- sudo_user = os.getenv("SUDO_USER")
- if not sudo_user:
- raise NoOperatorInfo
- pwdentry = pwd.getpwnam(sudo_user)
- else:
- pwdentry = pwd.getpwuid(uid)
- # XXX: error checking might be nice
- # We follow the Ubuntu convention of gecos being a comma split field
- # with the person's realname being the first entry.
- return pwdentry.pw_gecos.split(",")[0], pwdentry.pw_name + "@" + socket.gethostname()
-
def get_operator_git():
"""
Returns ``Real Name <username@mit.edu>`` suitable for use in
- Git ``Something-by:`` string.
+ Git ``Something-by:`` string. Throws :exc:`NoOperatorInfo` if
+ no operator information is available.
"""
- return "%s <%s>" % get_operator_info()
-
-def get_operator_name_from_gssapi():
- """
- Returns username of the person operating this script based
- off of the :envvar:`SSH_GSSAPI_NAME` environment variable.
-
- .. note::
-
- :envvar:`SSH_GSSAPI_NAME` is not set by a vanilla OpenSSH
- distributions. Scripts servers are patched to support this
- environment variable.
- """
- principal = os.getenv("SSH_GSSAPI_NAME")
- if not principal:
- return None
- instance, _, _ = principal.partition("@")
- if instance.endswith("/root"):
- username, _, _ = principal.partition("/")
- else:
- username = instance
- return username
+ op = user.operator()
+ if op is None:
+ raise NoOperatorInfo
+ info = user.pwnam(op)
+ return "%s <%s>" % (info.realname, info.email)
def set_operator_env():
"""
Sets :envvar:`GIT_COMMITTER_NAME` and :envvar:`GIT_COMMITTER_EMAIL`
- environment variables if applicable. Does nothing if
- :func:`get_operator_info` throws :exc:`NoOperatorInfo`.
+ environment variables if applicable. Does nothing if no information
+ is available
"""
- try:
- op_realname, op_email = get_operator_info()
- os.putenv("GIT_COMMITTER_NAME", op_realname)
- os.putenv("GIT_COMMITTER_EMAIL", op_email)
- except NoOperatorInfo:
- pass
+ op = user.operator()
+ if op is None:
+ return
+ info = user.pwnam(op)
+ os.putenv("GIT_COMMITTER_NAME", info.realname)
+ os.putenv("GIT_COMMITTER_EMAIL", info.email)
def set_author_env():
"""
- Sets :envvar:`GIT_AUTHOR_NAME` and :envvar:`GIT_AUTHOR_EMAIL` environment
- variables if applicable. Does nothing if :func:`get_dir_owner` fails.
+ Sets :envvar:`GIT_AUTHOR_NAME` and :envvar:`GIT_AUTHOR_EMAIL`
+ environment variables if applicable. Does nothing if
+ :func:`wizard.user.passwd` fails.
"""
- try:
- # XXX: should check if the directory is in AFS, and if not, use
- # a more traditional metric
- lockername = get_dir_owner()
- os.putenv("GIT_AUTHOR_NAME", "%s locker" % lockername)
- os.putenv("GIT_AUTHOR_EMAIL", "%s@scripts.mit.edu" % lockername)
- except KeyError: # XXX: This doesn't actually make sense
- pass
+ info = user.passwd()
+ if info is None:
+ return
+ os.putenv("GIT_AUTHOR_NAME", "%s" % info.realname)
+ os.putenv("GIT_AUTHOR_EMAIL", "%s" % info.email)
def set_git_env():
"""Sets all appropriate environment variables for Git commits."""
def fetch(host, path, subpath, post=None):
try:
- # XXX: Special case if it's https; not sure why this data isn't
- # passed
+ # XXX: Should use urllib instead
h = httplib.HTTPConnection(host)
fullpath = path.rstrip("/") + "/" + subpath.lstrip("/") # to be lenient about input we accept
if post:
scripts.mit.edu / wizard.git / commitdiff
