X-Git-Url: https://scripts.mit.edu/gitweb/wizard.git/blobdiff_plain/ff0505d4d123bed81223d9f16c8d5e76873b78f2..f44a2198d46f2262bd24af3491ad2b0d73e2d518:/TODO diff --git a/TODO b/TODO index 1baa77a..962ef75 100644 --- a/TODO +++ b/TODO @@ -2,28 +2,79 @@ The Git Autoinstaller TODO NOW: -- Add repository flag to migrate so that we can specify an - arbitrary repository to migrate to -- Move repositories - -- The great initial deploy: - - Turn on mediawiki new autoinstaller - - Migrate all mediawik installs - -- Implement proper deploy log parsing; this basically means we - need to be able to introspect Git Log. Consider using git-python - for this. There's also missing functionality, bad error handling - and hacks in the prototype implementation of upgrade - -- Wordpress needs to have .scripts dir in all -scripts versions - (remember --no-walk!) (also make .scripts/.htaccess) -- Wordpress needs to have a .scripts/update script written for - its latest version (do this after its migration) - -- Summary script should be more machine friendly, and should not - output summary charts when I increase specificity -- Check how many autoinstalls are missing w bits for - daemon.scripts (this would need pyafs) +- Make wizard install accept appname-head (so that you can do a test with + head, and do things without tags). Also make it accept commit hashes. + In fact, let it accept any committish. Figure out what to do if we + do a test script with x.y.z when we REALLY mean x.y.z-scripts. XXX!!! +- Do early validation of inputs for configuration +- Let 'wizard configure' be interactive +- Parse output HTML for class="error" and give those errors back to the user, + then boot them back into configure + +- Replace gaierror with a more descriptive name (this is a DNS error) + +- Pre-emptively check if daemon/scripts-security-upd + is not on scripts-security-upd list (/mit/moira/bin/blanche) + +- Redo Wordpress conversion, with an eye for automating everything + possible (such as downloading the tarball and unpacking) + +- Web application for installing autoinstalls has a hard problem + with credentials (as well as installations that are not conducted + on an Athena machine.) Possible solutions include asking the user + to SSH into an athena machine and run a bunch of commands, or writing + a Java applet (possibly in Clojure or Scala) which gets filesystem + permissions and then performs the operations. + +- Pay back code debt + - Genericize callAsUser and drop_priviledges in shell + - Summary script should be more machine friendly, and should not + output summary charts when I increase specificity + - Summary script should do something intelligent when distinguishing + between old-style and new-style installs + - Report code in wizard/command/__init__.py is ugly as sin. Also, + the Report object should operate at a higher level of abstraction + so we don't have to manually increment fails. (in fact, that should + probably be called something different). The by-percent errors should + also be automated. + - Move resolutions in mediawiki.py to a text file? (the parsing overhead + may not be worth it) + - If a process is C-ced, it can result in a upgrade that has + an updated filesystem but not updated database. Make this more + resilient + - PHP end of file allows omitted semicolon, can result in parse error + if merge resolutions aren't careful. `php -l` can be a quick stopgap + +- Other stuff + - Make single user mass-migrate work when not logged in as root + - Don't use the scripts heuristics unless we're on scripts with the + AFS patch. Check with `fs sysname` + - Make 'wizard summary' generate nice pretty graphs of installs by date + (more histograms, will need to check actual .scripts-version files.) + - It should be able to handle installs like Django where there's a component + that gets installed in web_scripts and another directory that gets installed + in Scripts. + - ACLs is a starting point for sending mail to users, but it has + several failure modes: + - Old maintainers who don't care who are still on the ACL + - Private AFS groups that aren't mailing lists and that we + can't get to + A question is whether or not sending mail actually helps us: + many users will probably have to come back to us for help; many + other users won't care. + +PULLING OUT CONFIGURATION FILES IN AN AUTOMATED MANNER + +advancedpoll: Template file to fill out +django: Noodles of template files +gallery2: Multistage install process +joomla: Template file +mediawiki: One-step install process +phpbb: Multistage install process +phpical: Template file +trac: NFC +turbogears: NFC +wordpress: Multistage install process COMMIT MESSAGE FIELDS: @@ -41,40 +92,14 @@ Author: lockername locker NOTES: -- A perfectly formed autoinstall with upgrade paths for all of - the intervening versions is not really feasible to implement. - As such, we want to migrate everything to -scripts, and then - generate a -scripts2 with the correct .scripts directory. - We will then nop update some installs, but this will prevent - us from having to migrate and update concurrently. Treat - a scripts2 upgrade from migration the same way you would treat - a botched scripts upgrade. +- It is not required nor expected for update scripts to exist for all + intervening versions that were present pre-migration; only for it + to work on the most recent migration. - Currently all repositories are initialized with --shared, which means they have basically ~no space footprint. However, it - also means that /mit/scripts/wizard/srv MUST NOT lose revs. - -- Full fledged logging options. Namely: - x all loggers (delay implementing this until we actually have debug stmts) - - default is WARNING - - debug => loglevel = DEBUG - x stdout logger - - default is WARNING (see below for exception) - - verbose => loglevel = INFO - x file logger (only allowed for serial processing) - - default is OFF - - log-file => loglevel = INFO - x database logger (necessary for parallel processing, not implemented) - - default is OFF - - log-db => loglevel = INFO - -- More on the database logger: it will be very simple with one - table named `logs` in SQLite, with columns: `job`, `level`, - `message`. Job identifies the subprocess/thread that emitted - the log, so things can be correlated together. We will then - have `wizard dump` which takes a database like this and dumps - it into a file logger type file. The database may also store - a queue like structure which can be used to coordinate jobs. + also means that /mit/scripts/wizard/srv MUST NOT lose revs after + deployment. OVERALL PLAN: @@ -82,19 +107,11 @@ OVERALL PLAN: on documenting them. Specifically, we will be keeping: - parallel-find.pl, and the resulting - /mitalso make .scripts/.htaccess/scripts/sec-tools/store/scriptslist - This script might need to be adapted if we decide to nuke - .scripts-version files. - - - The current install scripts will be kept in place, sans changes - necessary to make them use Git install of copying the script over. - Porting these scripts to Python and making them modular would be - nice, but is priority. For the long term, seeing this scripts - be packaged with rest of our code would be optimal. + /mit/scripts/.htaccess/scripts/sec-tools/store/scriptslist * The new procedure for generating an update is as follows: (check out the mass-migration instructions for something in this spirit, - although uglier in some ways) + although uglier in some ways; A indicates the step /should/ be automated) 0. ssh into not-backward, temporarily give the daemon.scripts-security-upd bits by blanching it on system:scripts-security-upd, and run parallel-find.pl @@ -103,177 +120,195 @@ OVERALL PLAN: 2. Checkout the pristine branch - 3. Remove all files from the working copy. Use `wipe-working-dir` - - 4. Download the new tarball - - 5. Extract the tarball over the working copy (`cp -R a/. b` works well, - remember that the working copy is empty) - - 6. Check for empty directories and add stub files as necessary. - Use `preserve-empty-dir` + 3. Run wizard `prepare-pristine APP-VERSION` - 7. Git add it all, and then commit as a new pristine version (v1.2.3) + X. Commit, with name "Appname x.y.z" - 8. Checkout the master branch + 4. Checkout the master branch - 9. [FOR EXISTING REPOSITORIES] - Merge the pristine branch in. Resolve any conflicts that our + 5. Merge the pristine branch in. Resolve any conflicts that our patches have with new changes. Do NOT let Git auto-commit it with --no-commit (otherwise, you want to git commit --amend to keep our history clean - [FOR NEW REPOSITORIES] - See if any patches are needed to make this run smoothly on - scripts. + X. Commit, with name "Appname x.y.z-scripts". This is going to be + amended. - [FOR NEW REPOSITORIES] - mkdir .scripts - echo "Deny from all" > .scripts/.htaccess - touch .scripts/update - chmod a+x .scripts/update + 6. Run 'wizard prepare-config' on a scripts server while in a checkout + of this newest version. This will prepare a new version of the + configuration file based on the application's latest installer. + Manually merge back in any custom changes we may have made. + Check if any of the regular expressions need tweaking by inspecting + the configuration files for user-specific gunk, and modify + wizard.app.APPNAME accordingly. Commit with --amend, and + propagate back to your local copy (git reset --hard HEAD~; git pull afs). - 10. Check if there are any special update procedures, and update/create the - .scripts/update shell script as necessary (this means that any - application specific update logic will be kept with the actual - source code. The language of this update script will vary - depending on context.) + [ENTER HERE FROM CREATING A NEW REPO] - 11. Commit your changes, and tag as v1.2.3-scripts (or scripts2, if - you are amending an install without an upstream changes) + 7. Check if there are any special update procedures, and update + the wizard.app.APPNAME module accordingly. If this is the first + time you are performing an upgrade, implement upgrade() in your + Application class. (XXX: extended instructions here). Test + the new update procedure using our test scripts (preferably + on a scripts server). Check this page for more info on our + integration tests: - 12. Test the new update procedure using - `wizard upgrade --with=/path/to/repo /your/autoinstall` (this will - read out master as your "latest" version). - Use git commit --amend to fix any bugs (alternatively, squash them - together later). + http://scripts.mit.edu/wizard/testing.html#acceptance-tests - 13. You can also do a "mass" version of this using: - `wizard -d testbed.txt massupgrade --with=/path/to/repo app` - You'll need perms for any testbed stuff you want. + 8. If you have any further changes, git commit --amend, and finally + tag as v1.2.3-scripts (or scripts2, if you are amending an install + without an upstream changes) - GET APPROVAL BEFORE PROCEEDING ANY FURTHER + 9. Push all of your changes in a public place, and encourage others + to test, using --srv-path and a full path. + + GET APPROVAL BEFORE PROCEEDING ANY FURTHER; + THIS IS PUSHING THE CHANGES TO THE PUBLIC NOTE: The following commands are to be run on not-backward.mit.edu. You'll need to add daemon.scripts-security-upd to scripts-security-upd to get bits to do this. Make sure you remove these bits when you're done. - 14. Run `wizard research appname` + 10. Run `wizard research appname` which uses Git commands to check how many working copies apply the change cleanly, and writes out a logfile with the working copies that don't apply cleanly. It also tells - us about "corrupt" working copies. - - 15. Run `wizard massupgrade appname`, which applies the update to all working - copies possible, and sends mail to users to whom the working copy - did not apply cleanly. It also frobs .scripts-version for successful - upgrades (maybe not, depending on our plans). - - 16. Run parallel-find.pl to update our inventory - -* For mass importing into the repository, the steps are: - (this probably won't ever be automated, becuase there are fiddly bits) - -[TO SET IT UP] -# let app-1.2.3 be the scripts folder originally in deploydev -# let this folder be srv/ -# you can also do a git clone - mkdir app - cd app - git init - cd .. -unfurl app-1.2.3 app # [FIDDLY BIT] -# NOTE: contents of application are now in app directory -cd app -git add . -git commit -s -m "App 1.2.3" -git tag v1.2.3 -git branch pristine -# NOTE: you're still on master branch -# WARNING: the following operation might require -p1 -patch -p0 < ../app-1.2.3/app-1.2.3.patch # [FIDDLY BIT] -# NOTE: please sanity check the patch! -git add . -# NOTE: -a flag is to handle if the patch deleted something -git commit -as -m "App 1.2.3-scripts" -git tag v1.2.3-scripts - -[TO ADD AN UPDATE] -# let this folder be srv/app.git -git checkout pristine -# NOTE: this preserves your .git folder, but removes everything -wipe-working-dir . -cd .. -unfurl app-1.2.3 app # [FIDDLY BIT] -cd app -# NOTE: please sanity check app directory -git add . -# NOTE: -a is to take care of deletions -git commit -as -m "App 1.2.3" -git tag v1.2.3 -[FIDDLE AROUND. FIDDLE AROUND] -[IF THE PATCH HAS CHANGED] - # You are on the pristine branch - # NOTE: Now, the tricky part (this is different from a real update) - git symbolic-ref HEAD refs/heads/master - # NOTE: Now, we think we're on the master branch, but we have - # pristine copy checked out - # NOTE: -p0 might need to be twiddled - patch -p0 < ../app-1.2.3/app-1.2.3.patch - git add . - # COMMENT: used to git checkout .scripts here - # then check if the directory needs an updated update script - # NOTE: Fake the merge - git rev-parse pristine > .git/MERGE_HEAD -[IF THE PATCH HASN'T CHANGED] - git checkout master - git merge --no-commit pristine -git commit -as -m "App 1.2.3-scripts" -git tag v1.2.3-scripts - -* The repository for a given application will contain the following files: + us about "corrupt" working copies, i.e. working copies that + have over a certain threshold of changes. - - The actual application's files, as from the official tarball + 11. Run `wizard mass-upgrade appname`, which applies the update to all working + copies possible. - - A .scripts directory, which contains the following information: + 12. Run parallel-find.pl to update our inventory - * .scripts/update shell script (with the +x bit set appropriately), - which performs the commands necessary to update a script. This can - be in any language. +* For mass importing into the repository, there are a few extra things: - * .scripts/.htaccess to prevent this directory from being accessed - from the web. + * Many applications had patches associated with them. Be sure to + apply them, so later merges work better. - * .scripts/database (generated) contains the database the - user installed the script to, so scripts-remove can clean it + # the following operation might require -p1 + patch -p0 < ../app-1.2.3/app-1.2.3.patch # [FIDDLY BIT] - XXX: Could cause problems if a user copies the autoinstall, - fiddles with the DB credentials, and then scripts-remove's - the autoinstall. Possible fix is to add the original - directory as a sanity check. Additionally, we could have - the application read out of this file. + * When running updates, if the patch has changed you will have to + do a special procedure for your merge: - * .scripts/old-version (optional) the old value of .scripts-versoin + git checkout pristine + # NOTE: Now, the tricky part (this is different from a real update) + git symbolic-ref HEAD refs/heads/master + # NOTE: Now, we think we're on the master branch, but we have + # pristine copy checked out + # NOTE: -p0 might need to be twiddled + patch -p0 < ../app-1.2.3/app-1.2.3.patch + git add . + # reconstitute .scripts directory + git checkout v1.2.2-scripts -- .scripts + git add .scripts + # NOTE: Fake the merge + git rev-parse pristine > .git/MERGE_HEAD - * .scripts/install (eventually) interactively installs the - applicatoin from command line. + You could also just try your luck with a manual merge using the patch + as your guide. -* The autoupgrade shall be the process of: - - # Make the directory not accessible by the outside world (htaccess, but be careful!) - git add -u . - git commit -m 'automatically generated backup' - git pull origin master - if [ $? ne 0 ]; then git reset --hard; echo 'conflicts during upgrade'; fi - ./.scripts/update - # Make it accessible +* The repository for a given application will contain the following files: - (with some more robust error checking, a proper dry run mechanism to, and - lots of su'ing) + - The actual application's files, as from the official tarball -* Make 'wizard summary' generate nice pretty graphs of installs by date - (more histograms, will need to check actual .scripts-version files.) + - A .scripts directory, with the intent of holding Scripts specific files + if they become necessary. + +* Making the module files for a new application + + 1. Create a wizard/app/APPNAME.py file. Create an object Application + inheriting from wizard.app.Application (check existing modules for + the boilerplate code). + + 2. Implement download(). "wizard prepare-pristine" will use this in order + to download the next version of an application. + + 3. Create a git repository with `git init` + + 4. Use `wizard prepare-pristine APP-VERSION` to download the tarball and + extract it into the directory. If download() doesn't work and you don't + want to special case it (for example, you need a /really old version/ + for record-keeping purposes), replace APP-VERSION with PATH, where PATH + is the tarball to extract. + + 5. `git commit -asm "APP VERSION"` + + 6. Check if any patches are needed to make the application work + on Scripts (ideally, it shouldn't.) Pre-existing patches + live in /mit/scripts/deploy/APP-VERSION/ directories. + + 7. Run `wizard prepare-new` to setup common filesets for our repositories. + + 8. If you are running a PHP script, there is usually a php.ini file + that we package. You can see previous instances of this patch + at /mit/scripts/deploy/php.ini/ as well as in the repositories + of any already migrated scripts. We hope to make these changes + unnecessary once PHP 5.3 arrives. + + 9. Do an initial commit (we're gonna be amending the hell of this) + using `git commit -asm "APP VERSION-scripts" + + 10. Implement install(). Test using `wizard install APP`; you won't + be able to do a version-specific install with `wizard install APP-VERSION` + until you generate a tag (which will become out of date once you + amend the commit.) Now might be a good time to create a + tests/test-install-APP.sh file (use the other tests as reference) so + you don't have to constantly enter the parameters when you're doing + an install. + + 11. Push your changes to a directory accessible in the production environment. + In the case of scripts, this is equivalent to your AFS homedir, and + the production environment is a scripts.mit.edu. We're going to + perform a configuration in the production environment to extract + out the canonical configuration files. + + 12. On the production server, call your wizard to perform an installation; + be sure to use the option --no-commit in order to make propagating changes + back easier. Inspect the generated configuration files (you can use `git + status` to find unversioned files that the installer created), and + implement: + - extractors + - substitutions + These are dictionaries of functions that perform extraction + and substitution of variables from config files. You don't + actually have to hand code them; you can app.make_extractors + and app.make_substitutions on a common dictionary. Check + out wizard/app/__init__.py for more information on this + format, as well as other files for samples. + (XXX: extended instructions here) + - parametrized_files + These are any files that contain WIZARD_* variables + - checkConfig() + This is a simple, fs based check on whether or not the application + was configured. Usually checking if some generated config file + is present is sufficient + - detectVersion() + You might be able to reuse machinery from extractors (namely, whatever + function you were using to generate regular expressions), or you might + need to code a custom regular expression to parse this out. + - deprecated_keys? + Usually you won't need this; use it if there's a configuration variable + that needs to get parametrized, but isn't actually necessary and + gets obsoleted in a later version. You probably won't know if that's + the case until later. + + 13. With these implemented, `wizard prepare-config` should now work if you run + it on the installed copy. The configuration file should now contain only + generic WIZARD_* variables, and no user-specific config. If it is, your + script was buggy; try again. + + 14. The current changes in the working copy should be merged in. Add any new + files, and then `git commit --amend`. `git push --force` to stick these + changes back in the "public" repository. + + 15. In your local copy, you can pull the changes by doing `git reset --hard HEAD~` + and then a `git pull` from the relevant source. Otherwise, Git will complain + about a non-fast-forward. + + 16. Congratulations! You've implemented the installation code for a new install. + Now goto "ENTER HERE FROM CREATING A NEW REPO" and finish the rest of the + instructions. -* Update AFS patch to advertise its existence, so we can check for it - here.