X-Git-Url: https://scripts.mit.edu/gitweb/wizard.git/blobdiff_plain/f44a2198d46f2262bd24af3491ad2b0d73e2d518..6617b37f3e19b1d97292f6a28b39c5b8b696f427:/TODO diff --git a/TODO b/TODO index 962ef75..9cb904a 100644 --- a/TODO +++ b/TODO @@ -2,32 +2,36 @@ The Git Autoinstaller TODO NOW: -- Make wizard install accept appname-head (so that you can do a test with - head, and do things without tags). Also make it accept commit hashes. - In fact, let it accept any committish. Figure out what to do if we - do a test script with x.y.z when we REALLY mean x.y.z-scripts. XXX!!! -- Do early validation of inputs for configuration -- Let 'wizard configure' be interactive -- Parse output HTML for class="error" and give those errors back to the user, - then boot them back into configure +- XXX: Upgrades don't pull updated tags, breaking git describe --tags! + Fix this for the future, and figure out how to make everyone else happy! +- XXX: Some installs are throwing spurious errors; investigate +- XXX: Prolly would be nice to have some information about how many installs actually succeeded +- If you try to do an install on scripts w/o sql, it will sign you up but fail to write + the sql.cnf file. This sucks. + +- wizard install wordpress should ask for password. One problem with this is that + Wordpress will still send mail with the wrong username and password, so Wordpress + will need to be patched to not do that. Alternatively we can initally set the admin + email to a null address and then fix it manually. +- --raw parameter for install which means an arbitrary commit can be installed +- Parse output HTML for class="error" and give those errors back to the user (done), + then boot them back into configure so they can enter in something different +- Get rid of our custom sizing code and use dialog's built-in sizing (i.e. width=0, height=0). + Maybe our sizing code is superior, maybe not. - Replace gaierror with a more descriptive name (this is a DNS error) - Pre-emptively check if daemon/scripts-security-upd is not on scripts-security-upd list (/mit/moira/bin/blanche) -- Redo Wordpress conversion, with an eye for automating everything - possible (such as downloading the tarball and unpacking) - - Web application for installing autoinstalls has a hard problem with credentials (as well as installations that are not conducted - on an Athena machine.) Possible solutions include asking the user - to SSH into an athena machine and run a bunch of commands, or writing - a Java applet (possibly in Clojure or Scala) which gets filesystem - permissions and then performs the operations. + on an Athena machine.) We have some crazy ideas involving a signed + Java applet that uses jsch to SSH into athena.dialup and perform + operations. - Pay back code debt - - Genericize callAsUser and drop_priviledges in shell + - Tidy up common code in callAsUser and drop_priviledges in shell - Summary script should be more machine friendly, and should not output summary charts when I increase specificity - Summary script should do something intelligent when distinguishing @@ -46,7 +50,15 @@ TODO NOW: if merge resolutions aren't careful. `php -l` can be a quick stopgap - Other stuff - - Make single user mass-migrate work when not logged in as root + - Figure out why Sphinx sometimes fails to crossref :func: but wil + crossref :meth:, even though the dest is very clearly a function. + Example: :func:`wizard.app.php.re_var` + - The TODO extension for Sphinx doesn't properly force a full-rebuild + - Code annotation! + - Make single user mass-migrate work when not logged in as root. The + primary difficulty is making the parallel-find information easily + accessible to individual users: perhaps we can do a single-user + parallel-find on the fly. - Don't use the scripts heuristics unless we're on scripts with the AFS patch. Check with `fs sysname` - Make 'wizard summary' generate nice pretty graphs of installs by date @@ -116,52 +128,14 @@ OVERALL PLAN: 0. ssh into not-backward, temporarily give the daemon.scripts-security-upd bits by blanching it on system:scripts-security-upd, and run parallel-find.pl - 1. Have the Git repository and working copy for the project on hand. - - 2. Checkout the pristine branch - - 3. Run wizard `prepare-pristine APP-VERSION` - - X. Commit, with name "Appname x.y.z" - - 4. Checkout the master branch - - 5. Merge the pristine branch in. Resolve any conflicts that our - patches have with new changes. Do NOT let Git auto-commit it - with --no-commit (otherwise, you want to git commit --amend - to keep our history clean - - X. Commit, with name "Appname x.y.z-scripts". This is going to be - amended. - - 6. Run 'wizard prepare-config' on a scripts server while in a checkout - of this newest version. This will prepare a new version of the - configuration file based on the application's latest installer. - Manually merge back in any custom changes we may have made. - Check if any of the regular expressions need tweaking by inspecting - the configuration files for user-specific gunk, and modify - wizard.app.APPNAME accordingly. Commit with --amend, and - propagate back to your local copy (git reset --hard HEAD~; git pull afs). + 1. [ see doc/upgrade.rst ] [ENTER HERE FROM CREATING A NEW REPO] - 7. Check if there are any special update procedures, and update - the wizard.app.APPNAME module accordingly. If this is the first - time you are performing an upgrade, implement upgrade() in your - Application class. (XXX: extended instructions here). Test - the new update procedure using our test scripts (preferably - on a scripts server). Check this page for more info on our - integration tests: - - http://scripts.mit.edu/wizard/testing.html#acceptance-tests - - 8. If you have any further changes, git commit --amend, and finally - tag as v1.2.3-scripts (or scripts2, if you are amending an install - without an upstream changes) - 9. Push all of your changes in a public place, and encourage others to test, using --srv-path and a full path. +[ XXX: doc/deploy.rst ] GET APPROVAL BEFORE PROCEEDING ANY FURTHER; THIS IS PUSHING THE CHANGES TO THE PUBLIC @@ -182,15 +156,10 @@ OVERALL PLAN: 12. Run parallel-find.pl to update our inventory +[ XXX: doc/upgrade.rst ] * For mass importing into the repository, there are a few extra things: - * Many applications had patches associated with them. Be sure to - apply them, so later merges work better. - - # the following operation might require -p1 - patch -p0 < ../app-1.2.3/app-1.2.3.patch # [FIDDLY BIT] - - * When running updates, if the patch has changed you will have to + * When mass producing updates, if the patch has changed you will have to do a special procedure for your merge: git checkout pristine @@ -210,6 +179,7 @@ OVERALL PLAN: You could also just try your luck with a manual merge using the patch as your guide. +[ XXX: doc/layout.rst ] * The repository for a given application will contain the following files: - The actual application's files, as from the official tarball @@ -217,98 +187,5 @@ OVERALL PLAN: - A .scripts directory, with the intent of holding Scripts specific files if they become necessary. -* Making the module files for a new application - - 1. Create a wizard/app/APPNAME.py file. Create an object Application - inheriting from wizard.app.Application (check existing modules for - the boilerplate code). - - 2. Implement download(). "wizard prepare-pristine" will use this in order - to download the next version of an application. - - 3. Create a git repository with `git init` - - 4. Use `wizard prepare-pristine APP-VERSION` to download the tarball and - extract it into the directory. If download() doesn't work and you don't - want to special case it (for example, you need a /really old version/ - for record-keeping purposes), replace APP-VERSION with PATH, where PATH - is the tarball to extract. - - 5. `git commit -asm "APP VERSION"` - - 6. Check if any patches are needed to make the application work - on Scripts (ideally, it shouldn't.) Pre-existing patches - live in /mit/scripts/deploy/APP-VERSION/ directories. - - 7. Run `wizard prepare-new` to setup common filesets for our repositories. - - 8. If you are running a PHP script, there is usually a php.ini file - that we package. You can see previous instances of this patch - at /mit/scripts/deploy/php.ini/ as well as in the repositories - of any already migrated scripts. We hope to make these changes - unnecessary once PHP 5.3 arrives. - - 9. Do an initial commit (we're gonna be amending the hell of this) - using `git commit -asm "APP VERSION-scripts" - - 10. Implement install(). Test using `wizard install APP`; you won't - be able to do a version-specific install with `wizard install APP-VERSION` - until you generate a tag (which will become out of date once you - amend the commit.) Now might be a good time to create a - tests/test-install-APP.sh file (use the other tests as reference) so - you don't have to constantly enter the parameters when you're doing - an install. - - 11. Push your changes to a directory accessible in the production environment. - In the case of scripts, this is equivalent to your AFS homedir, and - the production environment is a scripts.mit.edu. We're going to - perform a configuration in the production environment to extract - out the canonical configuration files. - - 12. On the production server, call your wizard to perform an installation; - be sure to use the option --no-commit in order to make propagating changes - back easier. Inspect the generated configuration files (you can use `git - status` to find unversioned files that the installer created), and - implement: - - extractors - - substitutions - These are dictionaries of functions that perform extraction - and substitution of variables from config files. You don't - actually have to hand code them; you can app.make_extractors - and app.make_substitutions on a common dictionary. Check - out wizard/app/__init__.py for more information on this - format, as well as other files for samples. - (XXX: extended instructions here) - - parametrized_files - These are any files that contain WIZARD_* variables - - checkConfig() - This is a simple, fs based check on whether or not the application - was configured. Usually checking if some generated config file - is present is sufficient - - detectVersion() - You might be able to reuse machinery from extractors (namely, whatever - function you were using to generate regular expressions), or you might - need to code a custom regular expression to parse this out. - - deprecated_keys? - Usually you won't need this; use it if there's a configuration variable - that needs to get parametrized, but isn't actually necessary and - gets obsoleted in a later version. You probably won't know if that's - the case until later. - - 13. With these implemented, `wizard prepare-config` should now work if you run - it on the installed copy. The configuration file should now contain only - generic WIZARD_* variables, and no user-specific config. If it is, your - script was buggy; try again. - - 14. The current changes in the working copy should be merged in. Add any new - files, and then `git commit --amend`. `git push --force` to stick these - changes back in the "public" repository. - - 15. In your local copy, you can pull the changes by doing `git reset --hard HEAD~` - and then a `git pull` from the relevant source. Otherwise, Git will complain - about a non-fast-forward. - - 16. Congratulations! You've implemented the installation code for a new install. - Now goto "ENTER HERE FROM CREATING A NEW REPO" and finish the rest of the - instructions. + - .scripts/dsn, overriding database source name