The Git Autoinstaller TODO NOW: - Test code should auto-nuke the database using `wizard remove` before doing a new install - git diff :1:$file :2:$file to find out what the user did, or is it :3:? - Document how to fix a broken upgrade - php.ini needs to get substituted! - Make wizard install accept appname-head (so that you can do a test with head, and do things without tags). Also make it accept commit hashes. In fact, let it accept any committish. Figure out what to do if we do a test script with x.y.z when we REALLY mean x.y.z-scripts. XXX!!! - Do early validation of inputs for configuration - Let 'wizard configure' be interactive - Parse output HTML for class="error" and give those errors back to the user, then boot them back into configure - Get rid of our custom sizing code and use dialog's built-in sizing (i.e. width=0, height=0). Maybe our sizing code is superior, maybe not. - Replace gaierror with a more descriptive name (this is a DNS error) - Pre-emptively check if daemon/scripts-security-upd is not on scripts-security-upd list (/mit/moira/bin/blanche) - Redo Wordpress conversion, with an eye for automating everything possible (such as downloading the tarball and unpacking) - Web application for installing autoinstalls has a hard problem with credentials (as well as installations that are not conducted on an Athena machine.) Possible solutions include asking the user to SSH into an athena machine and run a bunch of commands, or writing a Java applet (possibly in Clojure or Scala) which gets filesystem permissions and then performs the operations. - Pay back code debt - Genericize callAsUser and drop_priviledges in shell - Summary script should be more machine friendly, and should not output summary charts when I increase specificity - Summary script should do something intelligent when distinguishing between old-style and new-style installs - Report code in wizard/command/__init__.py is ugly as sin. Also, the Report object should operate at a higher level of abstraction so we don't have to manually increment fails. (in fact, that should probably be called something different). The by-percent errors should also be automated. - Move resolutions in mediawiki.py to a text file? (the parsing overhead may not be worth it) - If a process is C-ced, it can result in a upgrade that has an updated filesystem but not updated database. Make this more resilient - PHP end of file allows omitted semicolon, can result in parse error if merge resolutions aren't careful. `php -l` can be a quick stopgap - Other stuff - Figure out why Sphinx sometimes fails to crossref :func: but wil crossref :meth:, even though the dest is very clearly a function. Example: :func:`wizard.app.php.re_var` - The TODO extension for Sphinx doesn't properly force a full-rebuild - Code annotation! - Make single user mass-migrate work when not logged in as root - Don't use the scripts heuristics unless we're on scripts with the AFS patch. Check with `fs sysname` - Make 'wizard summary' generate nice pretty graphs of installs by date (more histograms, will need to check actual .scripts-version files.) - It should be able to handle installs like Django where there's a component that gets installed in web_scripts and another directory that gets installed in Scripts. - ACLs is a starting point for sending mail to users, but it has several failure modes: - Old maintainers who don't care who are still on the ACL - Private AFS groups that aren't mailing lists and that we can't get to A question is whether or not sending mail actually helps us: many users will probably have to come back to us for help; many other users won't care. PULLING OUT CONFIGURATION FILES IN AN AUTOMATED MANNER advancedpoll: Template file to fill out django: Noodles of template files gallery2: Multistage install process joomla: Template file mediawiki: One-step install process phpbb: Multistage install process phpical: Template file trac: NFC turbogears: NFC wordpress: Multistage install process COMMIT MESSAGE FIELDS: Installed-by: username@hostname Pre-commit-by: Real Name Upgraded-by: Real Name Migrated-by: Real Name Wizard-revision: abcdef1234567890 Wizard-args: /wizard/bin/wizard foo bar baz GIT COMMIT FIELDS: Committer: Real Name Author: lockername locker NOTES: - It is not required nor expected for update scripts to exist for all intervening versions that were present pre-migration; only for it to work on the most recent migration. - Currently all repositories are initialized with --shared, which means they have basically ~no space footprint. However, it also means that /mit/scripts/wizard/srv MUST NOT lose revs after deployment. OVERALL PLAN: * Some parts of the infrastructure will not be touched, although I plan on documenting them. Specifically, we will be keeping: - parallel-find.pl, and the resulting /mit/scripts/.htaccess/scripts/sec-tools/store/scriptslist * The new procedure for generating an update is as follows: (check out the mass-migration instructions for something in this spirit, although uglier in some ways; A indicates the step /should/ be automated) 0. ssh into not-backward, temporarily give the daemon.scripts-security-upd bits by blanching it on system:scripts-security-upd, and run parallel-find.pl 1. [ see doc/upgrade.rst ] [ENTER HERE FROM CREATING A NEW REPO] 9. Push all of your changes in a public place, and encourage others to test, using --srv-path and a full path. [ XXX: doc/deploy.rst ] GET APPROVAL BEFORE PROCEEDING ANY FURTHER; THIS IS PUSHING THE CHANGES TO THE PUBLIC NOTE: The following commands are to be run on not-backward.mit.edu. You'll need to add daemon.scripts-security-upd to scripts-security-upd to get bits to do this. Make sure you remove these bits when you're done. 10. Run `wizard research appname` which uses Git commands to check how many working copies apply the change cleanly, and writes out a logfile with the working copies that don't apply cleanly. It also tells us about "corrupt" working copies, i.e. working copies that have over a certain threshold of changes. 11. Run `wizard mass-upgrade appname`, which applies the update to all working copies possible. 12. Run parallel-find.pl to update our inventory [ XXX: doc/upgrade.rst ] * For mass importing into the repository, there are a few extra things: * When mass producing updates, if the patch has changed you will have to do a special procedure for your merge: git checkout pristine # NOTE: Now, the tricky part (this is different from a real update) git symbolic-ref HEAD refs/heads/master # NOTE: Now, we think we're on the master branch, but we have # pristine copy checked out # NOTE: -p0 might need to be twiddled patch -p0 < ../app-1.2.3/app-1.2.3.patch git add . # reconstitute .scripts directory git checkout v1.2.2-scripts -- .scripts git add .scripts # NOTE: Fake the merge git rev-parse pristine > .git/MERGE_HEAD You could also just try your luck with a manual merge using the patch as your guide. [ XXX: doc/layout.rst ] * The repository for a given application will contain the following files: - The actual application's files, as from the official tarball - A .scripts directory, with the intent of holding Scripts specific files if they become necessary.