X-Git-Url: https://scripts.mit.edu/gitweb/wizard.git/blobdiff_plain/4e0e34b4c14edb4db73378bb56c05ebf1538eec4..dbfdb74c8a6ba673b11153407c1cc042828182ab:/TODO diff --git a/TODO b/TODO index 40ce07e..6cf6b14 100644 --- a/TODO +++ b/TODO @@ -2,58 +2,146 @@ The Git Autoinstaller TODO NOW: -- Migration script needs to delete .scripts-version/move - it somwhere else, since it's deprecated -- Migration script needs to make an empty commit with the - user's changes that says "Initial migration", and then - has wizard info -- We need to figure out how much info Wizard will barf into - the commit message. Candidates: - o Wizard revision - o Wizard command line args - o Person running the command (quentin claims there's an - SSH patch that puts the Kerberos principal in REMOTE_USER) - -- Check how many autoinstalls are missing w bits for - daemon.scripts (this would need pyafs) -- Add repository flag to migrate so that we can specify an - arbitrary repository to migrate to -- Run parallel-find.pl -- Migrate all mediawikis -- Wordpress needs to have a .scripts/update script written for - its latest version - -- Need to upgrade the installer scripts to work out of the - repository -- Need an upgrade script OR -- Need survey script +- Keep my sanity when upgrading 1000 installs + - Custom merge algo: absolute php.ini symlinks to relative symlinks (this + does not seem to have been a problem in practice) + - Custom merge algo: check if it's got extra \r's in the file, + and dos2unix it if it does, before performing the merge + - Prune -7 call errors and automatically reprocess them (with a + strike out counter of 3)--this requires better error parsing. + - IOError should be aggregated, right now contains custom string + that makes this not possible. Partition on a colon. + - Replace gaierror with a more descriptive name (this is a DNS error) + - Stronger skips means that backup failures should also be avoided + - Distinguish between types of backup failures + - Ignore empty blacklists; they should all have reasons + - wizard upgrade should have different exit codes for merge failure + and blacklist error + +- Distinguish from logging and reporting (so we can easily send mail + to users) + - Figure out a way of collecting blacklist data from .scripts/blacklisted + and aggregate it together + +- Let users use Wizard when ssh'ed into Scripts + - Make single user mass-migrate work when not logged in as root + +- Make the rest of the world use Wizard + - Make parallel-find.pl use `sudo -u username git describe --tags` + to determine version. Make parallel-find.pl have this have greater + precedence. This also means, however, that we get + full mediawiki-1.2.3-2-abcdef names (Have patch, pending testing and commit) + - Make deployed installer use 'wizard install' /or/ do a migration + after doing a normal install (the latter makes it easier + for mass-rollbacks). + +- Pre-emptively check if daemon/scripts-security-upd + is not on scripts-security-upd list (/mit/moira/bin/blanche) + +- Redo Wordpress conversion, with an eye for automating everything + possible (such as downloading the tarball and unpacking) + +- Pay back code debt + - Genericize callAsUser and drop_priviledges in shell + - Summary script should be more machine friendly, and should not + output summary charts when I increase specificity + - Summary script should do something intelligent when distinguishing + between old-style and new-style installs + - Report code in wizard/command/__init__.py is ugly as sin. Also, + the Report object should operate at a higher level of abstraction + so we don't have to manually increment fails. (in fact, that should + probably be called something different). The by-percent errors should + also be automated. + - Indents in upgrade.py are getting pretty ridiculous; more breaking + into functions is probably a good idea + - Move resolutions in mediawiki.py to a text file + - Investigate QuotaParseErrors + +- Other stuff + - Don't use the scripts heuristics unless we're on scripts with the + AFS patch. Check with `fs sysname` + - Make 'wizard summary' generate nice pretty graphs of installs by date + (more histograms, will need to check actual .scripts-version files.) + - It should be able to handle installs like Django where there's a component + that gets installed in web_scripts and another directory that gets installed + in Scripts. + - ACLs is a starting point for sending mail to users, but it has + several failure modes: + - Old maintainers who don't care who are still on the ACL + - Private AFS groups that aren't mailing lists and that we + can't get to + A question is whether or not sending mail actually helps us: + many users will probably have to come back to us for help; many + other users won't care. + +PULLING OUT CONFIGURATION FILES IN AN AUTOMATED MANNER + +advancedpoll: Template file to fill out +django: Noodles of template files +gallery2: Multistage install process +joomla: Template file +mediawiki: One-step install process +phpbb: Multistage install process +phpical: Template file +trac: NFC +turbogears: NFC +wordpress: Multistage install process + +PHILOSOPHY ABOUT LOGGING + +Logging is most useful when performing a mass run. This +includes things such as mass-migration as well as when running +summary reports. An interesting property about mass-migration +or mass-upgrade, however, is that if they fail, they are +idempotent, so an individual case can be debugged simply running +the single-install equivalent with --debug on. (This, indeed, +may be easier to do than sifting through a logfile). + +It is a different story when you are running a summary report: +you are primarily bound by your AFS cache and how quickly you can +iterate through all of the autoinstalls. Checking if a file +exists on a cold AFS cache may +take several minutes to perform; on a hot cache the same report +may take a mere 3 seconds. When you get to more computationally +expensive calculations, however, even having a hot AFS cache +is not enough to cut down your runtime. + +There are certain calculations that someone may want to be +able to perform on manipulated data. As such, this data should +be cached on disk, if the process for extracting this data takes +a long time. Also, for usability sake, Wizard should generate +the common case reports. + +Ensuring that machine parseable reports are made, and then making +the machinery to reframe this data, increases complexity. Therefore, +the recommendation is to assume that if you need to run iteratively, +you'll have a hot AFS cache at your fingerprints, and if that's not +fast enough, then cache the data. COMMIT MESSAGE FIELDS: -Installed-by: username@hostname # person who ran autoinstaller -Updated-by: username@hostname # person who upgraded an autoinstall -Migrated-by: username@hostname # person who migrated an autoinstall -Wizard-revision: abcdef1234567890 # revision of wizard used -Wizard-args: wizard foo bar baz # arguments passed to Wizard +Installed-by: username@hostname +Pre-commit-by: Real Name +Upgraded-by: Real Name +Migrated-by: Real Name +Wizard-revision: abcdef1234567890 +Wizard-args: /wizard/bin/wizard foo bar baz -NOTES: +GIT COMMIT FIELDS: + +Committer: Real Name +Author: lockername locker -- A perfectly formed autoinstall with upgrade paths for all of - the intervening versions is not really feasible to implement. - As such, we want to migrate everything to -scripts, and then - generate a -scripts2 with the correct .scripts directory. - We will then nop update some installs, but this will prevent - us from having to migrate and update concurrently. Treat - a scripts2 upgrade from migration the same way you would treat - a botched scripts upgrade. +NOTES: -- summary and info are still not using loggers. Maybe they should, - maybe they shouldn't. Using loggers means we lose interactivity - with the Git output +- It is not expected or required for update scripts to exist for all + intervening versions that were present pre-migration; only for it + to work on the most recent migration. - Currently all repositories are initialized with --shared, which means they have basically ~no space footprint. However, it - also means that /mit/scripts/wizard/srv MUST NOT lose revs. + also means that /mit/scripts/wizard/srv MUST NOT lose revs after + deployment. - Full fledged logging options. Namely: x all loggers (delay implementing this until we actually have debug stmts) @@ -62,20 +150,9 @@ NOTES: x stdout logger - default is WARNING (see below for exception) - verbose => loglevel = INFO - x file logger (only allowed for serial processing) + x file logger (creates a dir and lots of little logfiles) - default is OFF - log-file => loglevel = INFO - x database logger (necessary for parallel processing, not implemented) - - default is OFF - - log-db => loglevel = INFO - -- More on the database logger: it will be very simple with one - table named `logs` in SQLite, with columns: `job`, `level`, - `message`. Job identifies the subprocess/thread that emitted - the log, so things can be correlated together. We will then - have `wizard dump` which takes a database like this and dumps - it into a file logger type file. The database may also store - a queue like structure which can be used to coordinate jobs. OVERALL PLAN: @@ -83,37 +160,34 @@ OVERALL PLAN: on documenting them. Specifically, we will be keeping: - parallel-find.pl, and the resulting - /mit/scripts/sec-tools/store/scriptslist - This script might need to be adapted if we decide to nuke - .scripts-version files. - - - The current install scripts will be kept in place, sans changes - necessary to make them use Git install of copying the script over. - Porting these scripts to Python and making them modular would be - nice, but is priority. For the long term, seeing this scripts - be packaged with rest of our code would be optimal. + /mit/scripts/.htaccess/scripts/sec-tools/store/scriptslist * The new procedure for generating an update is as follows: (check out the mass-migration instructions for something in this spirit, - although uglier in some ways) + although uglier in some ways; A indicates the step /should/ be automated) 0. ssh into not-backward, temporarily give the daemon.scripts-security-upd bits by blanching it on system:scripts-security-upd, and run parallel-find.pl 1. Have the Git repository and working copy for the project on hand. - 2. Checkout the pristine branch +/- wizard prepare-pristine -- + +A 2. Checkout the pristine branch - 3. Remove all files from the working copy. Use `wipe-working-dir` +A 3. Remove all files from the working copy. Use `wipe-working-dir` - 4. Download the new tarball +A 4. Download the new tarball - 5. Extract the tarball over the working copy (`cp -R a/. b` works well, - remember that the working copy is empty) +A 5. Extract the tarball over the working copy (`cp -R a/. b` works well, + remember that the working copy is empty; this needs some intelligent + input) - 6. Check for empty directories and add stub files as necessary. +A 6. Check for empty directories and add stub files as necessary. Use `preserve-empty-dir` +\--- + 7. Git add it all, and then commit as a new pristine version (v1.2.3) 8. Checkout the master branch @@ -125,33 +199,40 @@ OVERALL PLAN: to keep our history clean [FOR NEW REPOSITORIES] - See if any patches are needed to make this run smoothly on - scripts. - - [FOR NEW REPOSITORIES] - mkdir .scripts - echo "Deny from all" > .scripts/.htaccess - touch .scripts/update - chmod a+x .scripts/update - - 10. Check if there are any special update procedures, and update/create the - .scripts/update shell script as necessary (this means that any - application specific update logic will be kept with the actual - source code. The language of this update script will vary - depending on context.) - - 11. Commit your changes, and tag as v1.2.3-scripts (or scripts2, if + Check if any patches are needed to make the application work + on Scripts (ideally, it shouldn't. + +/- wizard prepare-new -- + + Currently not used for anything besides parallel-find.pl, but + we reserve the right to place files in here in the future. + +A mkdir .scripts +A echo "Deny from all" > .scripts/.htaccess + +\--- + + 10. Check if there are any special update procedures, and update + the wizard.app.APPNAME module accordingly (or create it, if + need be). + + 11. Run 'wizard prepare-config' on a scripts server while in a checkout + of this newest version. This will prepare a new version of the + configuration file based on the application's latest installer. + Manually merge back in any custom changes we may have made. + Check if any of the regular expressions need tweaking by inspecting + the configuration files for user-specific gunk, and modify + wizard.app.APPNAME accordingly. + + 12. Commit your changes, and tag as v1.2.3-scripts (or scripts2, if you are amending an install without an upstream changes) - 12. Test the new update procedure using - `wizard upgrade --with=/path/to/repo /your/autoinstall` (this will - read out master as your "latest" version). - Use git commit --amend to fix any bugs (alternatively, squash them - together later). + NOTE: These steps should be run on a scripts server + + 13. Test the new update procedure using our test scripts. See integration + tests for more information on how to do this. - 13. You can also do a "mass" version of this using: - `wizard -d testbed.txt massupgrade --with=/path/to/repo app` - You'll need perms for any testbed stuff you want. + http://scripts.mit.edu/wizard/testing.html#acceptance-tests GET APPROVAL BEFORE PROCEEDING ANY FURTHER @@ -160,139 +241,53 @@ OVERALL PLAN: scripts-security-upd to get bits to do this. Make sure you remove these bits when you're done. - 14. Run `wizard research appname` +A 14. Run `wizard research appname` which uses Git commands to check how many working copies apply the change cleanly, and writes out a logfile with the working copies that don't apply cleanly. It also tells - us about "corrupt" working copies. + us about "corrupt" working copies, i.e. working copies that + have over a certain threshold of changes. - 15. Run `wizard massupgrade appname`, which applies the update to all working +A 15. Run `wizard mass-upgrade appname`, which applies the update to all working copies possible, and sends mail to users to whom the working copy - did not apply cleanly. It also frobs .scripts-version for successful - upgrades (maybe not, depending on our plans). + did not apply cleanly. 16. Run parallel-find.pl to update our inventory -* For mass importing into the repository, the steps are: - (this probably won't ever be automated, becuase there are fiddly bits) - -[TO SET IT UP] -# let app-1.2.3 be the scripts folder originally in deploydev -# let this folder be srv/ -# you can also do a git clone - mkdir app - cd app - git init - cd .. -unfurl app-1.2.3 app # [FIDDLY BIT] -# NOTE: contents of application are now in app directory -cd app -git add . -git commit -s -m "App 1.2.3" -git tag v1.2.3 -git branch pristine -# NOTE: you're still on master branch -# WARNING: the following operation might require -p1 -patch -p0 < ../app-1.2.3/app-1.2.3.patch # [FIDDLY BIT] -# NOTE: please sanity check the patch! -git add . -# NOTE: -a flag is to handle if the patch deleted something -git commit -as -m "App 1.2.3-scripts" -git tag v1.2.3-scripts - -[TO ADD AN UPDATE] -# let this folder be srv/app.git -git checkout pristine -# NOTE: this preserves your .git folder, but removes everything -wipe-working-dir . -cd .. -unfurl app-1.2.3 app # [FIDDLY BIT] -cd app -# NOTE: please sanity check app directory -git add . -# NOTE: -a is to take care of deletions -git commit -as -m "App 1.2.3" -git tag v1.2.3 -[FIDDLE AROUND. FIDDLE AROUND] -[IF THE PATCH HAS CHANGED] - # You are on the pristine branch - # NOTE: Now, the tricky part (this is different from a real update) - git symbolic-ref HEAD refs/heads/master - # NOTE: Now, we think we're on the master branch, but we have - # pristine copy checked out - # NOTE: -p0 might need to be twiddled - patch -p0 < ../app-1.2.3/app-1.2.3.patch - git add . - # COMMENT: used to git checkout .scripts here - # then check if the directory needs an updated update script - # NOTE: Fake the merge - git rev-parse pristine > .git/MERGE_HEAD -[IF THE PATCH HASN'T CHANGED] - git checkout master - git merge --no-commit pristine -git commit -as -m "App 1.2.3-scripts" -git tag v1.2.3-scripts - - -* The repository for a given application will contain the following files: - - - The actual application's files, as from the official tarball - - - A .scripts directory, which contains the following information: - - * .scripts/update shell script (with the +x bit set appropriately), - which performs the commands necessary to update a script. This can - be in any language. - - * .scripts/.htaccess to prevent this directory from being accessed - from the web. - - * .scripts/database (generated) contains the database the - user installed the script to, so scripts-remove can clean it +* For mass importing into the repository, there are a few extra things: - XXX: Could cause problems if a user copies the autoinstall, - fiddles with the DB credentials, and then scripts-remove's - the autoinstall. Possible fix is to add the original - directory as a sanity check. Additionally, we could have - the application read out of this file. + * Many applications had patches associated with them. Be sure to + apply them, so later merges work better. - * .scripts/version (generated) which contains the version - last autoinstalled (as distinct from the actual version - the script is) (This is the same as .scripts-version right - now; probably want to keep that for now) + # the following operation might require -p1 + patch -p0 < ../app-1.2.3/app-1.2.3.patch # [FIDDLY BIT] - XXX: It's unclear if we want to move to this wholesale, or - delay this indefinitely. quentin thinks that the Git - repository itself is a sufficient record. + * When running updates, if the patch has changed you will have to + do a special procedure for your merge: -* The migration process has been implemented, see 'wizard migrate'. + git checkout pristine + # NOTE: Now, the tricky part (this is different from a real update) + git symbolic-ref HEAD refs/heads/master + # NOTE: Now, we think we're on the master branch, but we have + # pristine copy checked out + # NOTE: -p0 might need to be twiddled + patch -p0 < ../app-1.2.3/app-1.2.3.patch + git add . + # reconstitute .scripts directory + git checkout v1.2.2-scripts -- .scripts + git add .scripts + # NOTE: Fake the merge + git rev-parse pristine > .git/MERGE_HEAD - XXX: We have not decided what migration should do to .scripts-version; - if it does move it to .scripts, repositories should have a .gitignore - in those directories + You could also just try your luck with a manual merge using the patch + as your guide. -* The autoupgrade shall be the process of: - - # Make the directory not accessible by the outside world (htaccess, but be careful!) - git add -u . - git commit -m 'automatically generated backup' - git pull origin master - if [ $? ne 0 ]; then git reset --hard; echo 'conflicts during upgrade'; fi - ./.scripts/update - # Make it accessible +* The repository for a given application will contain the following files: - (with some more robust error checking, a proper dry run mechanism to, and - lots of su'ing) + - The actual application's files, as from the official tarball -* All code that operates on an untrusted Git repository, or runs - executable code, should be done on NOT-BACKWARD.mit.edu. Pending - accounts confirmation, it will also get a principal - daemon.scripts-security-upd, which is what we'll actually put - in the scripts-security-upd group. parallel-find.pl should also - be run on not-backward, by virtue of its fat pipe to the AFS servers. + - A .scripts directory, with the intent of holding Scripts specific files + if they become necessary. -* Make 'wizard summary' generate nice pretty graphs of installs by date - (more histograms, will need to check actual .scripts-version files.) + * .scripts/lock (generated) which locks an autoinstall during upgrade -* Update AFS patch to advertise its existence, so we can check for it - here.