Update README and TODO.

Signed-off-by: Edward Z. Yang <ezyang@mit.edu>

diff --git a/README b/README
index 11ab9a40c43e2633b0c11b3dd0eada1f98377bdb..d411e9d86353002d534573227c4b7011796bf9db 100644 (file)
--- a/README
+++ b/README
@@ -1,29 +1,16 @@
 Wizard is the new autoinstaller management system for
 Wizard is the new autoinstaller management system for

-scripts.mit.edu.  It resides on not-backward.mit.edu at:
+scripts.mit.edu.  This is a developer oriented introduction,
+and will get placed in doc/ soon.

 
 

-    /wizard

 
 

-Developers, read on:
-
-== Compatibility ==
+Compatibility
+-------------

 
 This is Python 2.6 only!
 
 
 This is Python 2.6 only!
 

-== Module overview ==
-
-- deploy: object model for autoinstall deployments, applications
-    and versions
-
-- shell: subprocess wrapper that does logging, error handling
-    and asynchronous processing
-
-- sset: "serialized set", use in a mass* command for tracking
-    what installs have already been seen and successfully processed
-
-- util: miscellaneous utility functions, right now mostly one-off
-    parsing and convenience wrappers

 
 

-== 'command' package ==
+The command package
+-------------------

 
 To add a new command to wizard, you need to touch the following
 locations:
 
 To add a new command to wizard, you need to touch the following
 locations:


@@ -31,24 +18,43 @@ locations:
 - bin/wizard
     Add a line describing your command in the helpstring
 
 - bin/wizard
     Add a line describing your command in the helpstring
 

-- wizard/command/commandname.py
+- wizard/command/command_name.py

     Implement your command there as main(). Copy the function
     signature and scaffolding from another file; it's non-trivial
     Implement your command there as main(). Copy the function
     signature and scaffolding from another file; it's non-trivial

+    Underscores in the module-name are referenced using dashes.

 
 As a command file, you're expected to have a main() and parse_args()
 function.  Try to keep main() one to two page function that can
 be browsed for a descriptive overview of what the command does
 
 As a command file, you're expected to have a main() and parse_args()
 function.  Try to keep main() one to two page function that can
 be browsed for a descriptive overview of what the command does

-internally.  Use helper functions liberally.  Place all error classes
-at the bottom.
+internally.  Use helper functions to create self-documenting
+code.  Don't make helper functions for one-liners.  Place
+all error classes at the bottom.

 
 

-== 'app' package ==

 
 

-There is a certain amount of application specific code that
-needs to be written for each autoinstaller.  These go here.
+Notes for repository creation
+-----------------------------

 
 

-== Notes when fiddling with application repositories ==
+See TODO for full instructions.

 
 

-- 'git rebase -p -i' is your friend.  Use it to rewrite history!
+- You are not going to be able to doctor a perfect repository
+  from scratch.
+
+    * If you have a simple change to a commit message, or
+      something that you don't really need interactive control
+      over (such as methodically rm'ing a file), git
+      filter-branch is very superior.
+
+    * 'git rebase -p -i' can be used to rewrite history interactively.
+      Doing this will nuke any manual merge resolution you
+      may have done (which may be a lot), thus, I highly
+      recommend that you enable rerere (rerere.enabled = true)
+      so that Git can automatically merge things for you.
+      If the merge fails, you'll still have to reconstitute
+      things manually.
+
+    * You may want to consider using the Git sequencer, for
+      which you can find a patch on the interwebs (I've never
+      tried it before)

 
 - The obvious scripts specific change that a new repository
   will have are the patches, but you should also check the
 
 - The obvious scripts specific change that a new repository
   will have are the patches, but you should also check the


@@ -60,9 +66,11 @@ needs to be written for each autoinstaller.  These go here.
       and WIZARD_TMPDIR
 
     - You will probably need to generate relevant configuration
       and WIZARD_TMPDIR
 
     - You will probably need to generate relevant configuration

-      files for each version
+      files for each version using 'wizard prepare-config'
+

 
 

-== Design decisions ==
+Design decisions
+----------------

 
 - A mass tool will plow ahead even if the subcommand throws
   Wizard errors.  Normal errors will cause us to halt
 
 - A mass tool will plow ahead even if the subcommand throws
   Wizard errors.  Normal errors will cause us to halt

diff --git a/TODO b/TODO
index ac022ca3eaaf78bcfd77286104386bf49e38d236..53e33c42c2b21fea6855086940afe4151d49dfc1 100644 (file)
--- a/TODO
+++ b/TODO
@@ -4,60 +4,47 @@ TODO NOW:
 
 - Make sure massmigrate cleanly ignores already migrated
   installs
 
 - Make sure massmigrate cleanly ignores already migrated
   installs

-- Make sure MediaWiki repository is as close to perfect
-  as possible:
-    - Do an install, migrate and then `git status`
-    - Check out possible missing php.ini's
-    - Remove "Merge comments" from lines
-    - Fix Signed-off-by lines
-- Add some safeguard code to make sure you don't run migrate
-  or upgrade as root
-
-- We have safe, non-braindead
-  version detection with `git describe --tags`.  Switch
-  everything to use it.  (I think the only thing left is
-  parallel-find.pl)
-- wizard.util is pretty braindead at this point.  Fix up
-  the wildly varying conventions in it.
+- Push su code to migrate, not mass-migrate (only do it
+  as root, this serves as a safe guard)
+- Rename massmigrate to mass-migrate

 - Move migration code into Wizard, since we already deal
 - Move migration code into Wizard, since we already deal

-  with installation there anyway.
+  with installation there anyway (this TODO has been updated
+  accordingly)
+
+- Make parallel-find.pl use `sudo -u username git describe --tags`
+  to determine version
+- Remove .scripts/version generation in install script
+- Make the installer use 'wizard install'

 
 - Better error message if daemon/scripts-security-upd
   is not on scripts-security-upd list
 
 - Fix retarded logging mechanism
 
 - Better error message if daemon/scripts-security-upd
   is not on scripts-security-upd list
 
 - Fix retarded logging mechanism

+    - This is interesting because the non-retarded way would be
+      to have each subprocess send data through stdout, but
+      this runs the risk of filling up the tubes.  Maybe just
+      create a log directory and give each process different
+      files.
+- Remove log functionality; we don't care about it besides
+  for parsing old style installs

 
 - The great initial deploy:
     - Turn on mediawiki new autoinstaller
     - Migrate all mediawiki installs
 
 
 - The great initial deploy:
     - Turn on mediawiki new autoinstaller
     - Migrate all mediawiki installs
 

-Doing Wordpress:
-- Build automation for generating config files; this automation
-  will be shared with the migrate script and the installer script
-  (migrate script needs to be able to pull out values from config
-  file, so will we; installer script needs to be able to run
-  the installer to generate config files, so will this)
-- This should all be automated:
-    - Wordpress needs to have .scripts dir in all -scripts versions
-      (also make .scripts/.htaccess)
-    - Wordpress needs to have a .scripts/update script written for
-      its latest version (do this after its migration)
-    - Wordpress needs to check for php.ini files (which it almost
-      certianly has) and commit messages
-    - Wordpress needs user config and php.ini links made
+- Redo Wordpress conversion, with an eye for automating everything
+  possible (such as downloading the tarball and unpacking)

 
 - Summary script should be more machine friendly, and should not
   output summary charts when I increase specificity
 
 - Summary script should be more machine friendly, and should not
   output summary charts when I increase specificity

-- Summary script needs to be updated for new format

 
 Some other stuff to do in your copious free time:
 
 Some other stuff to do in your copious free time:

+- Summary script should do something intelligent when distinguishing
+  between old-style and new-style installs

 - Check how many autoinstalls are missing w bits for
   daemon.scripts (this would need pyafs)
 - Make scripts AFS patch advertise its existence so we can check for it.
 - Check how many autoinstalls are missing w bits for
   daemon.scripts (this would need pyafs)
 - Make scripts AFS patch advertise its existence so we can check for it.

-  (This might be otherwise possible using `fs sysname`
-- Implement proper deploy log parsing; this basically means we
-  need to be able to introspect Git Log. Consider using git-python
-  for this.
+  (This might be otherwise possible using `fs sysname`)

 - Make 'wizard summary' generate nice pretty graphs of installs by date
   (more histograms, will need to check actual .scripts-version files.)
 - It should be able to handle installs like Django where there's a component
 - Make 'wizard summary' generate nice pretty graphs of installs by date
   (more histograms, will need to check actual .scripts-version files.)
 - It should be able to handle installs like Django where there's a component


@@ -130,7 +117,8 @@ NOTES:
 
 - Currently all repositories are initialized with --shared, which
   means they have basically ~no space footprint.  However, it
 
 - Currently all repositories are initialized with --shared, which
   means they have basically ~no space footprint.  However, it

-  also means that /mit/scripts/wizard/srv MUST NOT lose revs.
+  also means that /mit/scripts/wizard/srv MUST NOT lose revs after
+  deployment.

 
 - Full fledged logging options. Namely:
   x all loggers (delay implementing this until we actually have debug stmts)
 
 - Full fledged logging options. Namely:
   x all loggers (delay implementing this until we actually have debug stmts)


@@ -139,20 +127,9 @@ NOTES:
   x stdout logger
     - default is WARNING (see below for exception)
     - verbose   => loglevel = INFO
   x stdout logger
     - default is WARNING (see below for exception)
     - verbose   => loglevel = INFO

-  x file logger (only allowed for serial processing)
+  x file logger (creates a dir and lots of little logfiles)

     - default is OFF
     - log-file   => loglevel = INFO
     - default is OFF
     - log-file   => loglevel = INFO

-  x database logger (necessary for parallel processing, not implemented)
-    - default is OFF
-    - log-db    => loglevel = INFO
-
-- More on the database logger: it will be very simple with one
-  table named `logs` in SQLite, with columns: `job`, `level`,
-  `message`.  Job identifies the subprocess/thread that emitted
-  the log, so things can be correlated together.  We will then
-  have `wizard dump` which takes a database like this and dumps
-  it into a file logger type file.  The database may also store
-  a queue like structure which can be used to coordinate jobs.

 
 OVERALL PLAN:
 
 
 OVERALL PLAN:
 


@@ -164,12 +141,14 @@ OVERALL PLAN:
 
 * The new procedure for generating an update is as follows:
   (check out the mass-migration instructions for something in this spirit,
 
 * The new procedure for generating an update is as follows:
   (check out the mass-migration instructions for something in this spirit,

-  although uglier in some ways; A indicates the step /should/ be automated.)
+  although uglier in some ways; A indicates the step /should/ be automated)

 
     0. ssh into not-backward, temporarily give the daemon.scripts-security-upd
        bits by blanching it on system:scripts-security-upd, and run parallel-find.pl
 
 
     0. ssh into not-backward, temporarily give the daemon.scripts-security-upd
        bits by blanching it on system:scripts-security-upd, and run parallel-find.pl
 

-A   1. Have the Git repository and working copy for the project on hand.
+    1. Have the Git repository and working copy for the project on hand.
+
+/- wizard prepare-pristine --

 
 A   2. Checkout the pristine branch
 
 
 A   2. Checkout the pristine branch
 


@@ -184,9 +163,11 @@ A   5. Extract the tarball over the working copy (`cp -R a/. b` works well,
 A   6. Check for empty directories and add stub files as necessary.
        Use `preserve-empty-dir`
 
 A   6. Check for empty directories and add stub files as necessary.
        Use `preserve-empty-dir`
 

-A   7. Git add it all, and then commit as a new pristine version (v1.2.3)
+\---

 
 

-A   8. Checkout the master branch
+    7. Git add it all, and then commit as a new pristine version (v1.2.3)
+
+    8. Checkout the master branch

 
     9. [FOR EXISTING REPOSITORIES]
        Merge the pristine branch in. Resolve any conflicts that our
 
     9. [FOR EXISTING REPOSITORIES]
        Merge the pristine branch in. Resolve any conflicts that our


@@ -195,35 +176,37 @@ A   8. Checkout the master branch
        to keep our history clean
 
        [FOR NEW REPOSITORIES]
        to keep our history clean
 
        [FOR NEW REPOSITORIES]

-       See if any patches are needed to make this run smoothly on
-       scripts.
+       Check if any patches are needed to make the application work
+       on Scripts (ideally, it shouldn't.
+
+/- wizard prepare-new --

 
 

-    [FOR NEW REPOSITORIES]

 A       mkdir .scripts
 A       echo "Deny from all" > .scripts/.htaccess
 A       mkdir .scripts
 A       echo "Deny from all" > .scripts/.htaccess

-        touch .scripts/update
-        chmod a+x .scripts/update

 
 

-   10. Check if there are any special update procedures, and update/create the
-       .scripts/update shell script as necessary (this means that any
-       application specific update logic will be kept with the actual
-       source code.  The language of this update script will vary
-       depending on context.)
+\---
+
+   10. Check if there are any special update procedures, and update
+       the wizard.app.APPNAME module accordingly (or create it, if
+       need be).
+
+   11. Run 'wizard prepare-config' on a scripts server while in a checkout
+       of this newest version.  This will prepare a new version of the
+       configuration file based on the application's latest installer.
+       Manually merge back in any custom changes we may have made.
+       Check if any of the regular expressions need tweaking by inspecting
+       the configuration files for user-specific gunk, and modify
+       wizard.app.APPNAME accordingly.

 
 

-   11. Commit your changes, and tag as v1.2.3-scripts (or scripts2, if
+   12. Commit your changes, and tag as v1.2.3-scripts (or scripts2, if

        you are amending an install without an upstream changes)
 
       NOTE: These steps should be run on a scripts server
 
        you are amending an install without an upstream changes)
 
       NOTE: These steps should be run on a scripts server
 

-   12. Test the new update procedure using
-       `wizard upgrade --with=/path/to/repo /your/autoinstall` (this will
-       read out master as your "latest" version).
-       Use git commit --amend to fix any bugs (alternatively, squash them
-       together later).
+   13. Test the new update procedure using our test scripts.  See integration
+       tests for more information on how to do this.

 
 

-   13. You can also do a "mass" version of this using:
-       `wizard -d testbed.txt massupgrade --with=/path/to/repo app`
-       You'll need perms for any testbed stuff you want. (not implemented)
+        http://scripts.mit.edu/wizard/testing.html#acceptance-tests

 
       GET APPROVAL BEFORE PROCEEDING ANY FURTHER
 
 
       GET APPROVAL BEFORE PROCEEDING ANY FURTHER
 


@@ -232,78 +215,46 @@ A       echo "Deny from all" > .scripts/.htaccess
       scripts-security-upd to get bits to do this.  Make sure you remove
       these bits when you're done.
 
       scripts-security-upd to get bits to do this.  Make sure you remove
       these bits when you're done.
 

-   14. Run `wizard research appname`
+A  14. Run `wizard research appname`

        which uses Git commands to check how many
        working copies apply the change cleanly, and writes out a logfile
        with the working copies that don't apply cleanly.  It also tells
        us about "corrupt" working copies, i.e. working copies that
        have over a certain threshold of changes.
 
        which uses Git commands to check how many
        working copies apply the change cleanly, and writes out a logfile
        with the working copies that don't apply cleanly.  It also tells
        us about "corrupt" working copies, i.e. working copies that
        have over a certain threshold of changes.
 

-   15. Run `wizard massupgrade appname`, which applies the update to all working
+A  15. Run `wizard mass-upgrade appname`, which applies the update to all working

        copies possible, and sends mail to users to whom the working copy
        did not apply cleanly.
 
    16. Run parallel-find.pl to update our inventory
 
        copies possible, and sends mail to users to whom the working copy
        did not apply cleanly.
 
    16. Run parallel-find.pl to update our inventory
 

-* For mass importing into the repository, the steps are:
-  (this probably won't ever be automated, becuase there are fiddly bits)
-
-[TO SET IT UP]
-# let app-1.2.3 be the scripts folder originally in deploydev
-# let this folder be srv/
-# you can also do a git clone
-    mkdir app
-    cd app
-    git init
-    cd ..
-unfurl app-1.2.3 app  # [FIDDLY BIT]
-# NOTE: contents of application are now in app directory
-cd app
-git add .
-git commit -s -m "App 1.2.3"
-git tag v1.2.3
-git branch pristine
-# NOTE: you're still on master branch
-# WARNING: the following operation might require -p1
-patch -p0 < ../app-1.2.3/app-1.2.3.patch  # [FIDDLY BIT]
-# NOTE: please sanity check the patch!
-git add .
-# NOTE: -a flag is to handle if the patch deleted something
-git commit -as -m "App 1.2.3-scripts"
-git tag v1.2.3-scripts
-
-[TO ADD AN UPDATE]
-# let this folder be srv/app.git
-git checkout pristine
-# NOTE: this preserves your .git folder, but removes everything
-wipe-working-dir .
-cd ..
-unfurl app-1.2.3 app  # [FIDDLY BIT]
-cd app
-# NOTE: please sanity check app directory
-git add .
-# NOTE: -a is to take care of deletions
-git commit -as -m "App 1.2.3"
-git tag v1.2.3
-[FIDDLE AROUND. FIDDLE AROUND]
-[IF THE PATCH HAS CHANGED]
-    # You are on the pristine branch
-    # NOTE: Now, the tricky part (this is different from a real update)
-    git symbolic-ref HEAD refs/heads/master
-    # NOTE: Now, we think we're on the master branch, but we have
-    # pristine copy checked out
-    # NOTE: -p0 might need to be twiddled
-    patch -p0 < ../app-1.2.3/app-1.2.3.patch
-    git add .
-    # COMMENT: used to git checkout .scripts here
-    # then check if the directory needs an updated update script
-    # NOTE: Fake the merge
-    git rev-parse pristine > .git/MERGE_HEAD
-[IF THE PATCH HASN'T CHANGED]
-    git checkout master
-    git merge --no-commit pristine
-git commit -as -m "App 1.2.3-scripts"
-git tag v1.2.3-scripts
+* For mass importing into the repository, there are a few extra things:
+
+    * Many applications had patches associated with them.  Be sure to
+      apply them, so later merges work better.
+
+        # the following operation might require -p1
+        patch -p0 < ../app-1.2.3/app-1.2.3.patch  # [FIDDLY BIT]
+
+    * When running updates, if the patch has changed you will have to
+      do a special procedure for your merge:
+
+        git checkout pristine
+        # NOTE: Now, the tricky part (this is different from a real update)
+        git symbolic-ref HEAD refs/heads/master
+        # NOTE: Now, we think we're on the master branch, but we have
+        # pristine copy checked out
+        # NOTE: -p0 might need to be twiddled
+        patch -p0 < ../app-1.2.3/app-1.2.3.patch
+        git add .
+        # reconstitute .scripts directory
+        git checkout v1.2.2-scripts -- .scripts
+        git add .scripts
+        # NOTE: Fake the merge
+        git rev-parse pristine > .git/MERGE_HEAD
+
+      You could also just try your luck with a manual merge using the patch
+      as your guide.

 
 * The repository for a given application will contain the following files:
 
 
 * The repository for a given application will contain the following files:
 


@@ -311,10 +262,6 @@ git tag v1.2.3-scripts
 
     - A .scripts directory, which contains the following information:
 
 
     - A .scripts directory, which contains the following information:
 

-        * .scripts/update shell script (with the +x bit set appropriately),
-          which performs the commands necessary to update a script.  This can
-          be in any language. (XXX: This is going to get removed soon)
-

         * .scripts/.htaccess to prevent this directory from being accessed
           from the web.
 
         * .scripts/.htaccess to prevent this directory from being accessed
           from the web.
 


@@ -323,4 +270,6 @@ git tag v1.2.3-scripts
 
         * .scripts/lock (generated) which locks the autoinstall during an upgrade
 
 
         * .scripts/lock (generated) which locks the autoinstall during an upgrade
 

+        * .scripts/version (deprecated)
+