X-Git-Url: https://scripts.mit.edu/gitweb/wizard.git/blobdiff_plain/90f5c000e30828a10c31cbf50a92f0d06080c076..e5e340c29da2e4ebcf749a5bac75bc00f9eb5058:/TODO diff --git a/TODO b/TODO index 36bcead..e54748b 100644 --- a/TODO +++ b/TODO @@ -2,8 +2,101 @@ The Git Autoinstaller TODO NOW: -* Migrate and upgrade the lone mediawiki-1.5.6 install -* Create the migration script +- Keep my sanity when upgrading 1000 installs + - Replace gaierror with a more descriptive name (this is a DNS error) + +- Make the rest of the world use Wizard + - Make parallel-find.pl use `sudo -u username git describe --tags` + to determine version. Make parallel-find.pl have this have greater + precedence. This also means, however, that we get + full mediawiki-1.2.3-2-abcdef names (Have patch, pending testing and commit) + - Make deployed installer use 'wizard install' /or/ do a migration + after doing a normal install (the latter makes it easier + for mass-rollbacks). + +- Pre-emptively check if daemon/scripts-security-upd + is not on scripts-security-upd list (/mit/moira/bin/blanche) + +- Redo Wordpress conversion, with an eye for automating everything + possible (such as downloading the tarball and unpacking) + +- Pay back code debt + - Genericize callAsUser and drop_priviledges in shell + - Summary script should be more machine friendly, and should not + output summary charts when I increase specificity + - Summary script should do something intelligent when distinguishing + between old-style and new-style installs + - Report code in wizard/command/__init__.py is ugly as sin. Also, + the Report object should operate at a higher level of abstraction + so we don't have to manually increment fails. (in fact, that should + probably be called something different). The by-percent errors should + also be automated. + - Indents in upgrade.py are getting pretty ridiculous; more breaking + into functions is probably a good idea + - Move resolutions in mediawiki.py to a text file? (the parsing overhead + may not be worth it) + - Investigate QuotaParseErrors + - If a process is C-ced, it can result in a upgrade that has + an updated filesystem but not updated database. Make this more + resilient + - PHP end of file allows omitted semicolon, can result in parse error + if merge resolutions aren't careful. + +- Other stuff + - Make single user mass-migrate work when not logged in as root + - Don't use the scripts heuristics unless we're on scripts with the + AFS patch. Check with `fs sysname` + - Make 'wizard summary' generate nice pretty graphs of installs by date + (more histograms, will need to check actual .scripts-version files.) + - It should be able to handle installs like Django where there's a component + that gets installed in web_scripts and another directory that gets installed + in Scripts. + - ACLs is a starting point for sending mail to users, but it has + several failure modes: + - Old maintainers who don't care who are still on the ACL + - Private AFS groups that aren't mailing lists and that we + can't get to + A question is whether or not sending mail actually helps us: + many users will probably have to come back to us for help; many + other users won't care. + +PULLING OUT CONFIGURATION FILES IN AN AUTOMATED MANNER + +advancedpoll: Template file to fill out +django: Noodles of template files +gallery2: Multistage install process +joomla: Template file +mediawiki: One-step install process +phpbb: Multistage install process +phpical: Template file +trac: NFC +turbogears: NFC +wordpress: Multistage install process + +COMMIT MESSAGE FIELDS: + +Installed-by: username@hostname +Pre-commit-by: Real Name +Upgraded-by: Real Name +Migrated-by: Real Name +Wizard-revision: abcdef1234567890 +Wizard-args: /wizard/bin/wizard foo bar baz + +GIT COMMIT FIELDS: + +Committer: Real Name +Author: lockername locker + +NOTES: + +- It is not required nor expected for update scripts to exist for all + intervening versions that were present pre-migration; only for it + to work on the most recent migration. + +- Currently all repositories are initialized with --shared, which + means they have basically ~no space footprint. However, it + also means that /mit/scripts/wizard/srv MUST NOT lose revs after + deployment. OVERALL PLAN: @@ -11,132 +104,107 @@ OVERALL PLAN: on documenting them. Specifically, we will be keeping: - parallel-find.pl, and the resulting -/mit/scripts/sec-tools/store/scriptslist - - - The current install scripts will be kept in place, sans changes - necessary to make them use Git install of copying the script over. - Porting these scripts to Python and making them modular would be - nice, but is priority. For the long term, seeing this scripts - be packaged with rest of our code would be optimal. + /mit/scripts/.htaccess/scripts/sec-tools/store/scriptslist * The new procedure for generating an update is as follows: + (check out the mass-migration instructions for something in this spirit, + although uglier in some ways; A indicates the step /should/ be automated) - 1. Have the Git repository and working copy for the project on hand. - - 2. Download the new tarball - - 3. Extract the tarball over the working copy (`cp -R a/. b` works well) - - 4. Check if there are any special update procedures, and update the - .scripts/update shell script as necessary (this means that any - application specific update logic will be kept with the actual - source code. The language of this update script will vary - depending on context.) - - X. Check for empty directories and add stub files as necessary - (use preserve-empty-dir) - - 5. Commit your changes, and tag as v1.2.3-scripts - - 6. Run the "dry-run script", which uses Git commands to check how many - working copies apply the change cleanly, and writes out a logfile - with the working copies that don't apply cleanly. - - 7. Run the "limited run" script, which applies the update to our - test-bed, and lets us check the basic functionality of the update. - - 8. Run the "deploy" script, which applies the update to all working - copies possible, and sends mail to users to whom the working copy - did not apply cleanly. (It also frobs .scripts/version) + 0. ssh into not-backward, temporarily give the daemon.scripts-security-upd + bits by blanching it on system:scripts-security-upd, and run parallel-find.pl - Note: The last three scripts will need to be implemented, with an - eye towards speed. - -* How to migrate an old autoinstaller to the new autoinstaller - - - Find the oldest tarball/patch set for the application that still - is in use and upgradable. - - - Untar, apply patch, place in a directory (unfurl) and git init - - - Commit this as the "pristine" version (branch "pristine") + 1. Have the Git repository and working copy for the project on hand. - - Apply scripts patches + 2. Checkout the pristine branch - - Create the .scripts directory and populate it with the interesting - information (see below) + 3. Run wizard `prepare-pristine APP-VERSION` - - Commit this as the "scripts" version (branch "master") + 4. Checkout the master branch -* How to update the autoinstaller repository + 5. [FOR EXISTING REPOSITORIES] + Merge the pristine branch in. Resolve any conflicts that our + patches have with new changes. Do NOT let Git auto-commit it + with --no-commit (otherwise, you want to git commit --amend + to keep our history clean - - Checkout pristine branch - - Delete contents of pristine branch (excluding .git, try rm -Rf * and remove stragglers) - - Unfurl tarball - - Commit as new pristine branch - - Checkout scripts branch - - Merge pristine branch - - Fix as necessary + [FOR NEW REPOSITORIES] + Check if any patches are needed to make the application work + on Scripts (ideally, it shouldn't.) Run + `wizard prepare-new` to setup common filesets for our repositories. -* The repository for a given application will contain the following files: + 6. Check if there are any special update procedures, and update + the wizard.app.APPNAME module accordingly (or create it, if + need be). - - The actual application's files, as from the official tarball + 7. Run 'wizard prepare-config' on a scripts server while in a checkout + of this newest version. This will prepare a new version of the + configuration file based on the application's latest installer. + Manually merge back in any custom changes we may have made. + Check if any of the regular expressions need tweaking by inspecting + the configuration files for user-specific gunk, and modify + wizard.app.APPNAME accordingly. - - A .scripts directory, which contains the following information: + 8. Commit your changes, and tag as v1.2.3-scripts (or scripts2, if + you are amending an install without an upstream changes) - * .scripts/update shell script (with the +x bit set appropriately), - which performs the commands necessary to update a script. This can - be in any language. + NOTE: These steps should be run on a scripts server - * .scripts/.htaccess to prevent this directory from being accessed - from the web. + 9. Test the new update procedure using our test scripts. See integration + tests for more information on how to do this. - * .scripts/database (generated) contains the database the - user installed the script to, so scripts-remove can clean it + http://scripts.mit.edu/wizard/testing.html#acceptance-tests - * .scripts/version (generated) which contains the version - last autoinstalled (as distinct from the actual version - the script is) (This is the same as .scripts-version right - now; probably want to keep that for now) + GET APPROVAL BEFORE PROCEEDING ANY FURTHER - - Because there will be no .gitignore file, you *must not* run - `git add .` on an actual running copy of the application. - `git add -u .` will generally be safe, but preferred mode - of operation is to operate on a clean install. + NOTE: The following commands are to be run on not-backward.mit.edu. + You'll need to add daemon.scripts-security-upd to + scripts-security-upd to get bits to do this. Make sure you remove + these bits when you're done. -* The migration process shall be as such: + 10. Run `wizard research appname` + which uses Git commands to check how many + working copies apply the change cleanly, and writes out a logfile + with the working copies that don't apply cleanly. It also tells + us about "corrupt" working copies, i.e. working copies that + have over a certain threshold of changes. - 1. git init + 11. Run `wizard mass-upgrade appname`, which applies the update to all working + copies possible. - 2. git remote add origin /foo + 12. Run parallel-find.pl to update our inventory - 3. git config branch.master.merge refs/heads/master +* For mass importing into the repository, there are a few extra things: - 4. git fetch origin + * Many applications had patches associated with them. Be sure to + apply them, so later merges work better. - 5. git reset v1.2.3-scripts + # the following operation might require -p1 + patch -p0 < ../app-1.2.3/app-1.2.3.patch # [FIDDLY BIT] - 5. git checkout .scripts + * When running updates, if the patch has changed you will have to + do a special procedure for your merge: - 6. Setup .scripts/version (probably pipe the output of real-version) - UNCLEAR if this is a good thing; if it is, make sure we add - a .gitignore to the .scripts directory + git checkout pristine + # NOTE: Now, the tricky part (this is different from a real update) + git symbolic-ref HEAD refs/heads/master + # NOTE: Now, we think we're on the master branch, but we have + # pristine copy checked out + # NOTE: -p0 might need to be twiddled + patch -p0 < ../app-1.2.3/app-1.2.3.patch + git add . + # reconstitute .scripts directory + git checkout v1.2.2-scripts -- .scripts + git add .scripts + # NOTE: Fake the merge + git rev-parse pristine > .git/MERGE_HEAD -* We will not add special code to handle .htaccess; thus the kernel patch - for allowing Apache access to .htaccess sent to scripts-team@mit.edu - must be handled first. + You could also just try your luck with a manual merge using the patch + as your guide. -* The autoupgrade shall be the process of: +* The repository for a given application will contain the following files: - # Make the directory not accessible by the outside world (htaccess, but be careful!) - git add -u . - git commit -m 'automatically generated backup' - git pull origin master - if [ $? ne 0 ]; then git reset --hard; echo 'conflicts during upgrade'; fi - ./.scripts/update - # Make it accessible + - The actual application's files, as from the official tarball - (with some more robust error checking) + - A .scripts directory, with the intent of holding Scripts specific files + if they become necessary. -* Make install-statistics generate nice pretty graphs of installs by date - (more histograms, will need to check actual .scripts-version files.)