X-Git-Url: https://scripts.mit.edu/gitweb/wizard.git/blobdiff_plain/3f3c8a7ec822d73f7af28b6b7ef94f3476863527..6418262061baafb4db8ddbe2f22cd096585330ae:/TODO diff --git a/TODO b/TODO index fbcb2cb..2f44e55 100644 --- a/TODO +++ b/TODO @@ -2,18 +2,138 @@ The Git Autoinstaller TODO NOW: -- Fix mediawiki repository (has lots of stale files and a weird - history. This also means that the ichuang wiki will need to be - remigrated; frob .scripts-version to make it acceptable, but - otherwise should be fairly trivial). When I say "fix", I mean - "redo". Thorough documentation would be good too. Some bits - can be automated and/or have tools to assist it -- Whiteboard the flow for performing an upgrade on a single - install. How assisted does it need to be? -- Conduct migration tool testing -- Create mass-migration tool (should be able to limit on mediawiki) -- Run parallel-find.pl -- Migrate all mediawikis +- Remove "already migrated" cruft that will accumulate if we do small + --limit and then increase. +- Allow to migrate just one user (user filtering of installs, also + has userland capabilities, although it means we need some way of + selectively publishing the versions directory) +- Make migrate script rollback if it's interrupted (especially if + by signal) + +- Make parallel-find.pl use `sudo -u username git describe --tags` + to determine version. Make parallel-find.pl have this have greater + precedence. This also means, however, that we get + full mediawiki-1.2.3-2-abcdef names (Have patch, pending testing and commit) +- Make the installer use 'wizard install' /or/ do a migration + after doing a normal install (the latter makes it easier + for mass-rollbacks). +- Have the upgrader do locking (.scripts/lock, probably) + +- Relax MediaWiki regexes to terminate on semicolon, and not + require its own line. + +- Better error message if daemon/scripts-security-upd + is not on scripts-security-upd list + +- MediaWiki upgrade script does not give proper exit code; + if it fails, so be sure to check for "Done" in the last 10 characters. +- Custom merge algo: absolute php.ini symlinks to relative symlinks +- Custom merge algo: re-constitute AdminSettings.php if missing. It looks + like this is the case for most 1.5.8 installs (check what the merges + do in both directions). All 1.11.0 installs except four have + the other (check diff -u with all in /root) + +- Make upgrade and install take version as a parameter + +- Redo Wordpress conversion, with an eye for automating everything + possible (such as downloading the tarball and unpacking) + +- Genericize callAsUser and drop_priviledges in shell +- Summary script should be more machine friendly, and should not + output summary charts when I increase specificity + +Some other stuff to do in your copious free time: +- Summary script should do something intelligent when distinguishing + between old-style and new-style installs +- Check how many autoinstalls are missing w bits for + daemon.scripts (this would need pyafs) +- Make scripts AFS patch advertise its existence so we can check for it. + (This might be otherwise possible using `fs sysname`) +- Make 'wizard summary' generate nice pretty graphs of installs by date + (more histograms, will need to check actual .scripts-version files.) +- It should be able to handle installs like Django where there's a component + that gets installed in web_scripts and another directory that gets installed + in Scripts. + +PULLING OUT CONFIGURATION FILES IN AN AUTOMATED MANNER + +advancedpoll: Template file to fill out +django: Noodles of template files +gallery2: Multistage install process +joomla: Template file +mediawiki: One-step install process +phpbb: Multistage install process +phpical: Template file +trac: NFC +turbogears: NFC +wordpress: Multistage install process + +PHILOSOPHY ABOUT LOGGING + +Logging is most useful when performing a mass run. This +includes things such as mass-migration as well as when running +summary reports. An interesting property about mass-migration +or mass-upgrade, however, is that if they fail, they are +idempotent, so an individual case can be debugged simply running +the single-install equivalent with --debug on. (This, indeed, +may be easier to do than sifting through a logfile). + +It is a different story when you are running a summary report: +you are primarily bound by your AFS cache and how quickly you can +iterate through all of the autoinstalls. Checking if a file +exists on a cold AFS cache may +take several minutes to perform; on a hot cache the same report +may take a mere 3 seconds. When you get to more computationally +expensive calculations, however, even having a hot AFS cache +is not enough to cut down your runtime. + +There are certain calculations that someone may want to be +able to perform on manipulated data. As such, this data should +be cached on disk, if the process for extracting this data takes +a long time. Also, for usability sake, Wizard should generate +the common case reports. + +Ensuring that machine parseable reports are made, and then making +the machinery to reframe this data, increases complexity. Therefore, +the recommendation is to assume that if you need to run iteratively, +you'll have a hot AFS cache at your fingerprints, and if that's not +fast enough, then cache the data. + +COMMIT MESSAGE FIELDS: + +Installed-by: username@hostname +Pre-commit-by: Real Name +Upgraded-by: Real Name +Migrated-by: Real Name +Wizard-revision: abcdef1234567890 +Wizard-args: /wizard/bin/wizard foo bar baz + +GIT COMMIT FIELDS: + +Committer: Real Name +Author: lockername locker + +NOTES: + +- It is not expected or required for update scripts to exist for all + intervening versions that were present pre-migration; only for it + to work on the most recent migration. + +- Currently all repositories are initialized with --shared, which + means they have basically ~no space footprint. However, it + also means that /mit/scripts/wizard/srv MUST NOT lose revs after + deployment. + +- Full fledged logging options. Namely: + x all loggers (delay implementing this until we actually have debug stmts) + - default is WARNING + - debug => loglevel = DEBUG + x stdout logger + - default is WARNING (see below for exception) + - verbose => loglevel = INFO + x file logger (creates a dir and lots of little logfiles) + - default is OFF + - log-file => loglevel = INFO OVERALL PLAN: @@ -21,31 +141,33 @@ OVERALL PLAN: on documenting them. Specifically, we will be keeping: - parallel-find.pl, and the resulting -/mit/scripts/sec-tools/store/scriptslist + /mit/scripts/.htaccess/scripts/sec-tools/store/scriptslist - - The current install scripts will be kept in place, sans changes - necessary to make them use Git install of copying the script over. - Porting these scripts to Python and making them modular would be - nice, but is priority. For the long term, seeing this scripts - be packaged with rest of our code would be optimal. +* The new procedure for generating an update is as follows: + (check out the mass-migration instructions for something in this spirit, + although uglier in some ways; A indicates the step /should/ be automated) -* The new procedure for generating an update is as follows (this is - also similar to procedure for creating these repositories): + 0. ssh into not-backward, temporarily give the daemon.scripts-security-upd + bits by blanching it on system:scripts-security-upd, and run parallel-find.pl 1. Have the Git repository and working copy for the project on hand. - 2. Checkout the pristine branch +/- wizard prepare-pristine -- + +A 2. Checkout the pristine branch - 3. Remove all files from the working copy (rm -Rf *, and then delete - any dot stragglers. A script to do this would be handy) +A 3. Remove all files from the working copy. Use `wipe-working-dir` - 4. Download the new tarball +A 4. Download the new tarball - 5. Extract the tarball over the working copy (`cp -R a/. b` works well, - remember that the working copy is empty) +A 5. Extract the tarball over the working copy (`cp -R a/. b` works well, + remember that the working copy is empty; this needs some intelligent + input) - 6. Check for empty directories and add stub files as necessary - (use preserve-empty-dir) +A 6. Check for empty directories and add stub files as necessary. + Use `preserve-empty-dir` + +\--- 7. Git add it all, and then commit as a new pristine version (v1.2.3) @@ -57,83 +179,96 @@ OVERALL PLAN: with --no-commit (otherwise, you want to git commit --amend to keep our history clean - [FOR THE FIRST TIME] - Apply the scripts patch that was used for that version here - (usually patch -p1 < patch) + [FOR NEW REPOSITORIES] + Check if any patches are needed to make the application work + on Scripts (ideally, it shouldn't. - 10. Check if there are any special update procedures, and update the - .scripts/update shell script as necessary (this means that any - application specific update logic will be kept with the actual - source code. The language of this update script will vary - depending on context.) +/- wizard prepare-new -- - 11. Commit your changes, and tag as v1.2.3-scripts + Currently not used for anything besides parallel-find.pl, but + we reserve the right to place files in here in the future. - If you're setting up a repository from scratch, stop here, and - repeat as necessary +A mkdir .scripts +A echo "Deny from all" > .scripts/.htaccess - XXX: Should we force people to push to the real repository at - this point, or just make the repository that the script pulls - stuff out of configurable? (Twiddling origin can get you a - devel setup with no code changes) +\--- - 12. Run the "dry-run script", which uses Git commands to check how many - working copies apply the change cleanly, and writes out a logfile - with the working copies that don't apply cleanly. + 10. Check if there are any special update procedures, and update + the wizard.app.APPNAME module accordingly (or create it, if + need be). - 13. Run the "limited run" script, which applies the update to our - test-bed, and lets us check the basic functionality of the update. - This can include a script that lets us update a single directory - with verbose output. + 11. Run 'wizard prepare-config' on a scripts server while in a checkout + of this newest version. This will prepare a new version of the + configuration file based on the application's latest installer. + Manually merge back in any custom changes we may have made. + Check if any of the regular expressions need tweaking by inspecting + the configuration files for user-specific gunk, and modify + wizard.app.APPNAME accordingly. - 14. Run the "deploy" script, which applies the update to all working - copies possible, and sends mail to users to whom the working copy - did not apply cleanly. It also frobs .scripts/version for successful - upgrades. + 12. Commit your changes, and tag as v1.2.3-scripts (or scripts2, if + you are amending an install without an upstream changes) - 15. Run parallel-find.pl + NOTE: These steps should be run on a scripts server -* The repository for a given application will contain the following files: + 13. Test the new update procedure using our test scripts. See integration + tests for more information on how to do this. - - The actual application's files, as from the official tarball + http://scripts.mit.edu/wizard/testing.html#acceptance-tests + + GET APPROVAL BEFORE PROCEEDING ANY FURTHER - - A .scripts directory, which contains the following information: + NOTE: The following commands are to be run on not-backward.mit.edu. + You'll need to add daemon.scripts-security-upd to + scripts-security-upd to get bits to do this. Make sure you remove + these bits when you're done. - * .scripts/update shell script (with the +x bit set appropriately), - which performs the commands necessary to update a script. This can - be in any language. +A 14. Run `wizard research appname` + which uses Git commands to check how many + working copies apply the change cleanly, and writes out a logfile + with the working copies that don't apply cleanly. It also tells + us about "corrupt" working copies, i.e. working copies that + have over a certain threshold of changes. - * .scripts/.htaccess to prevent this directory from being accessed - from the web. +A 15. Run `wizard mass-upgrade appname`, which applies the update to all working + copies possible, and sends mail to users to whom the working copy + did not apply cleanly. - * .scripts/database (generated) contains the database the - user installed the script to, so scripts-remove can clean it + 16. Run parallel-find.pl to update our inventory - * .scripts/version (generated) which contains the version - last autoinstalled (as distinct from the actual version - the script is) (This is the same as .scripts-version right - now; probably want to keep that for now) +* For mass importing into the repository, there are a few extra things: - XXX: It's unclear if we want to move to this wholesale, or - delay this indefinitely. + * Many applications had patches associated with them. Be sure to + apply them, so later merges work better. -* The migration process has been implemented, see 'wizard migrate'. + # the following operation might require -p1 + patch -p0 < ../app-1.2.3/app-1.2.3.patch # [FIDDLY BIT] - XXX: We have not decided what migration should do to .scripts-version; - if it does move it to .scripts, repositories should have a .gitignore - in those directories + * When running updates, if the patch has changed you will have to + do a special procedure for your merge: -* The autoupgrade shall be the process of: + git checkout pristine + # NOTE: Now, the tricky part (this is different from a real update) + git symbolic-ref HEAD refs/heads/master + # NOTE: Now, we think we're on the master branch, but we have + # pristine copy checked out + # NOTE: -p0 might need to be twiddled + patch -p0 < ../app-1.2.3/app-1.2.3.patch + git add . + # reconstitute .scripts directory + git checkout v1.2.2-scripts -- .scripts + git add .scripts + # NOTE: Fake the merge + git rev-parse pristine > .git/MERGE_HEAD - # Make the directory not accessible by the outside world (htaccess, but be careful!) - git add -u . - git commit -m 'automatically generated backup' - git pull origin master - if [ $? ne 0 ]; then git reset --hard; echo 'conflicts during upgrade'; fi - ./.scripts/update - # Make it accessible + You could also just try your luck with a manual merge using the patch + as your guide. - (with some more robust error checking) +* The repository for a given application will contain the following files: + + - The actual application's files, as from the official tarball + + - A .scripts directory, with the intent of holding Scripts specific files + if they become necessary. + + * .scripts/lock (generated) which locks an autoinstall during upgrade -* Make 'wizard summary' generate nice pretty graphs of installs by date - (more histograms, will need to check actual .scripts-version files.)