X-Git-Url: https://scripts.mit.edu/gitweb/wizard.git/blobdiff_plain/49978bdf288eb15aa5fa62f687540a21a0b7aab2..3f18b7fff1c5d5cd3d91897a85a425704ba4295e:/TODO diff --git a/TODO b/TODO index c467861..4288d07 100644 --- a/TODO +++ b/TODO @@ -2,29 +2,132 @@ The Git Autoinstaller TODO NOW: -- For the long running programs, I definitely want to use logging. -- Whiteboard the flow for performing an upgrade on a single - install. How assisted does it need to be? -- Conduct migration tool testing -- Create mass-migration tool (should be able to limit on mediawiki) -- Run parallel-find.pl -- Migrate all mediawikis -- Wordpress needs to have a .scripts/update script written for - its latest version +- --retry option for install, so it won't complain about a directory already + being there. +- The calling web code invocations are a mess, with stubs living + in the install, deploy modules and the real deal living in util. Furthermore, + we use the scripts-specific heuristic to determine where the app + lives, and the only reason my test scripts work is because they + get manually fed the domain and path by my environment variables. + + We will record the URL used for the initial installation, and save it in + .scripts/url. If autodetection in either direction is + available, we verify this value against the actual file path the installation + lives in (for the scripts case, we can do a file-level comparison because we + know the web root of any given file). If they mismatch, we error out + and have someone manually resolve the problem. If autodetection is not + available, we use the saved .scripts/url for operations. + +- wizard install wordpress should ask for password +- Test code should auto-nuke the database using `wizard remove` before doing a new install +- git diff :1:$file :2:$file to find out what the user did, or is it :3:? +- Document how to fix a broken upgrade +- php.ini needs to get substituted! +- Make wizard install accept appname-head (so that you can do a test with + head, and do things without tags). Also make it accept commit hashes. + In fact, let it accept any committish. Figure out what to do if we + do a test script with x.y.z when we REALLY mean x.y.z-scripts. XXX!!! +- Do early validation of inputs for configuration +- Let 'wizard configure' be interactive +- Parse output HTML for class="error" and give those errors back to the user, + then boot them back into configure +- Get rid of our custom sizing code and use dialog's built-in sizing (i.e. width=0, height=0). + Maybe our sizing code is superior, maybe not. + +- Replace gaierror with a more descriptive name (this is a DNS error) + +- Pre-emptively check if daemon/scripts-security-upd + is not on scripts-security-upd list (/mit/moira/bin/blanche) + +- Redo Wordpress conversion, with an eye for automating everything + possible (such as downloading the tarball and unpacking) + +- Web application for installing autoinstalls has a hard problem + with credentials (as well as installations that are not conducted + on an Athena machine.) Possible solutions include asking the user + to SSH into an athena machine and run a bunch of commands, or writing + a Java applet (possibly in Clojure or Scala) which gets filesystem + permissions and then performs the operations. + +- Pay back code debt + - Genericize callAsUser and drop_priviledges in shell + - Summary script should be more machine friendly, and should not + output summary charts when I increase specificity + - Summary script should do something intelligent when distinguishing + between old-style and new-style installs + - Report code in wizard/command/__init__.py is ugly as sin. Also, + the Report object should operate at a higher level of abstraction + so we don't have to manually increment fails. (in fact, that should + probably be called something different). The by-percent errors should + also be automated. + - Move resolutions in mediawiki.py to a text file? (the parsing overhead + may not be worth it) + - If a process is C-ced, it can result in a upgrade that has + an updated filesystem but not updated database. Make this more + resilient + - PHP end of file allows omitted semicolon, can result in parse error + if merge resolutions aren't careful. `php -l` can be a quick stopgap + +- Other stuff + - Figure out why Sphinx sometimes fails to crossref :func: but wil + crossref :meth:, even though the dest is very clearly a function. + Example: :func:`wizard.app.php.re_var` + - The TODO extension for Sphinx doesn't properly force a full-rebuild + - Code annotation! + - Make single user mass-migrate work when not logged in as root + - Don't use the scripts heuristics unless we're on scripts with the + AFS patch. Check with `fs sysname` + - Make 'wizard summary' generate nice pretty graphs of installs by date + (more histograms, will need to check actual .scripts-version files.) + - It should be able to handle installs like Django where there's a component + that gets installed in web_scripts and another directory that gets installed + in Scripts. + - ACLs is a starting point for sending mail to users, but it has + several failure modes: + - Old maintainers who don't care who are still on the ACL + - Private AFS groups that aren't mailing lists and that we + can't get to + A question is whether or not sending mail actually helps us: + many users will probably have to come back to us for help; many + other users won't care. + +PULLING OUT CONFIGURATION FILES IN AN AUTOMATED MANNER + +advancedpoll: Template file to fill out +django: Noodles of template files +gallery2: Multistage install process +joomla: Template file +mediawiki: One-step install process +phpbb: Multistage install process +phpical: Template file +trac: NFC +turbogears: NFC +wordpress: Multistage install process + +COMMIT MESSAGE FIELDS: + +Installed-by: username@hostname +Pre-commit-by: Real Name +Upgraded-by: Real Name +Migrated-by: Real Name +Wizard-revision: abcdef1234567890 +Wizard-args: /wizard/bin/wizard foo bar baz + +GIT COMMIT FIELDS: + +Committer: Real Name +Author: lockername locker NOTES: -- A perfectly formed autoinstall with upgrade paths for all of - the intervening versions is not really feasible to implement. - As such, we want to migrate everything to -scripts, and then - generate a -scripts2 with the correct .scripts directory. - We will then nop update some installs, but this will prevent - us from having to migrate and update concurrently. +- It is not required nor expected for update scripts to exist for all + intervening versions that were present pre-migration; only for it + to work on the most recent migration. -- summary and info are still not using loggers. Maybe they should, - maybe they shouldn't - -- We need another patched AFS server to deploy updates off of. +- Currently all repositories are initialized with --shared, which + means they have basically ~no space footprint. However, it + also means that /mit/scripts/wizard/srv MUST NOT lose revs after + deployment. OVERALL PLAN: @@ -32,191 +135,71 @@ OVERALL PLAN: on documenting them. Specifically, we will be keeping: - parallel-find.pl, and the resulting -/mit/scripts/sec-tools/store/scriptslist - - - The current install scripts will be kept in place, sans changes - necessary to make them use Git install of copying the script over. - Porting these scripts to Python and making them modular would be - nice, but is priority. For the long term, seeing this scripts - be packaged with rest of our code would be optimal. - -* The new procedure for generating an update is as follows (this is - also similar to procedure for creating these repositories): - - 1. Have the Git repository and working copy for the project on hand. - - 2. Checkout the pristine branch - - 3. Remove all files from the working copy (rm -Rf *, and then delete - any dot stragglers. A script to do this would be handy) + /mit/scripts/.htaccess/scripts/sec-tools/store/scriptslist - 4. Download the new tarball +* The new procedure for generating an update is as follows: + (check out the mass-migration instructions for something in this spirit, + although uglier in some ways; A indicates the step /should/ be automated) - 5. Extract the tarball over the working copy (`cp -R a/. b` works well, - remember that the working copy is empty) + 0. ssh into not-backward, temporarily give the daemon.scripts-security-upd + bits by blanching it on system:scripts-security-upd, and run parallel-find.pl - 6. Check for empty directories and add stub files as necessary - (use preserve-empty-dir) + 1. [ see doc/upgrade.rst ] - 7. Git add it all, and then commit as a new pristine version (v1.2.3) + [ENTER HERE FROM CREATING A NEW REPO] - 8. Checkout the master branch + 9. Push all of your changes in a public place, and encourage others + to test, using --srv-path and a full path. - 9. [FOR EXISTING REPOSITORIES] - Merge the pristine branch in. Resolve any conflicts that our - patches have with new changes. Do NOT let Git auto-commit it - with --no-commit (otherwise, you want to git commit --amend - to keep our history clean +[ XXX: doc/deploy.rst ] + GET APPROVAL BEFORE PROCEEDING ANY FURTHER; + THIS IS PUSHING THE CHANGES TO THE PUBLIC - [FOR THE FIRST TIME] - Apply the scripts patch that was used for that version here - (usually patch -p1 < patch) + NOTE: The following commands are to be run on not-backward.mit.edu. + You'll need to add daemon.scripts-security-upd to + scripts-security-upd to get bits to do this. Make sure you remove + these bits when you're done. - 10. Check if there are any special update procedures, and update the - .scripts/update shell script as necessary (this means that any - application specific update logic will be kept with the actual - source code. The language of this update script will vary - depending on context.) - - 11. Commit your changes, and tag as v1.2.3-scripts - - If you're setting up a repository from scratch, stop here, and - repeat as necessary - - XXX: Should we force people to push to the real repository at - this point, or just make the repository that the script pulls - stuff out of configurable? (Twiddling origin can get you a - devel setup with no code changes) - - 12. Run the "dry-run script", which uses Git commands to check how many + 10. Run `wizard research appname` + which uses Git commands to check how many working copies apply the change cleanly, and writes out a logfile - with the working copies that don't apply cleanly. - - 13. Run the "limited run" script, which applies the update to our - test-bed, and lets us check the basic functionality of the update. - This can include a script that lets us update a single directory - with verbose output. - - 14. Run the "deploy" script, which applies the update to all working - copies possible, and sends mail to users to whom the working copy - did not apply cleanly. It also frobs .scripts/version for successful - upgrades. - - 15. Run parallel-find.pl - -* For mass importing into the repository, the steps are: - -[TO SET IT UP] -# let app-1.2.3 be the scripts folder originally in deploydev -# let this folder be srv/ -# you can also do a git clone - mkdir app - cd app - git init - cd .. -unfurl app-1.2.3 app -# NOTE: contents of application are now in app directory -cd app -git add . -git commit -s -m "App 1.2.3" -git tag v1.2.3 -git branch pristine -# NOTE: you're still on master branch -# WARNING: the following operation might require -p1 -patch -p0 < ../app-1.2.3/app-1.2.3.patch -# NOTE: please sanity check the patch! -git add . -# NOTE: -a flag is to handle if the patch deleted something -git commit -as -m "App 1.2.3-scripts" -git tag v1.2.3-scripts - -[TO ADD AN UPDATE] -# let this folder be srv/app.git -git checkout pristine -# NOTE: this preserves your .git folder, but removes everything -wipe-working-dir . -cd .. -unfurl app-1.2.3 app -cd app -# NOTE: please sanity check app directory -git add . -# NOTE: -a is to take care of deletions -git commit -as -m "App 1.2.3" -git tag v1.2.3 -[IF THE PATCH HAS CHANGED] - # You are on the pristine branch - # NOTE: Now, the tricky part (this is different from a real update) - git symbolic-ref HEAD refs/heads/master - # NOTE: Now, we think we're on the master branch, but we have - # pristine copy checked out - # NOTE: -p0 might need to be twiddled - patch -p0 < ../app-1.2.3/app-1.2.3.patch - git add . - # COMMENT: used to git checkout .scripts here - # then check if the directory needs an updated update script - # NOTE: Fake the merge - git rev-parse pristine > .git/MERGE_HEAD -[IF THE PATCH HASN'T CHANGED] - git checkout master - git merge --no-commit pristine -git commit -as -m "App 1.2.3-scripts" -git tag v1.2.3-scripts - - + with the working copies that don't apply cleanly. It also tells + us about "corrupt" working copies, i.e. working copies that + have over a certain threshold of changes. + + 11. Run `wizard mass-upgrade appname`, which applies the update to all working + copies possible. + + 12. Run parallel-find.pl to update our inventory + +[ XXX: doc/upgrade.rst ] +* For mass importing into the repository, there are a few extra things: + + * When mass producing updates, if the patch has changed you will have to + do a special procedure for your merge: + + git checkout pristine + # NOTE: Now, the tricky part (this is different from a real update) + git symbolic-ref HEAD refs/heads/master + # NOTE: Now, we think we're on the master branch, but we have + # pristine copy checked out + # NOTE: -p0 might need to be twiddled + patch -p0 < ../app-1.2.3/app-1.2.3.patch + git add . + # reconstitute .scripts directory + git checkout v1.2.2-scripts -- .scripts + git add .scripts + # NOTE: Fake the merge + git rev-parse pristine > .git/MERGE_HEAD + + You could also just try your luck with a manual merge using the patch + as your guide. + +[ XXX: doc/layout.rst ] * The repository for a given application will contain the following files: - The actual application's files, as from the official tarball - - A .scripts directory, which contains the following information: - - [IF THIS IS THE FIRST UPDATE] - mkdir .scripts - echo "Deny from all" > .scripts/.htaccess - touch .scripts/update - chmod a+x .scripts/update - # OPERATION: create the update script - - * .scripts/update shell script (with the +x bit set appropriately), - which performs the commands necessary to update a script. This can - be in any language. - - * .scripts/.htaccess to prevent this directory from being accessed - from the web. - - * .scripts/database (generated) contains the database the - user installed the script to, so scripts-remove can clean it - - XXX: Could cause problems if a user copies the autoinstall, - fiddles with the DB credentials, and then scripts-remove's - the autoinstall. Possible fix is to add the original - directory as a sanity check. Additionally, we could have - the application read out of this file. - - * .scripts/version (generated) which contains the version - last autoinstalled (as distinct from the actual version - the script is) (This is the same as .scripts-version right - now; probably want to keep that for now) - - XXX: It's unclear if we want to move to this wholesale, or - delay this indefinitely. - -* The migration process has been implemented, see 'wizard migrate'. - - XXX: We have not decided what migration should do to .scripts-version; - if it does move it to .scripts, repositories should have a .gitignore - in those directories - -* The autoupgrade shall be the process of: - - # Make the directory not accessible by the outside world (htaccess, but be careful!) - git add -u . - git commit -m 'automatically generated backup' - git pull origin master - if [ $? ne 0 ]; then git reset --hard; echo 'conflicts during upgrade'; fi - ./.scripts/update - # Make it accessible - - (with some more robust error checking) + - A .scripts directory, with the intent of holding Scripts specific files + if they become necessary. -* Make 'wizard summary' generate nice pretty graphs of installs by date - (more histograms, will need to check actual .scripts-version files.)