Implement --log-file-chmod

[wizard.git] / TODO

diff --git a/TODO b/TODO
index 36bcead79c3c58407f716d5227120db875d2353c..6769736c87e9960f24a65b03a74dcc3e20329c1c 100644 (file)
--- a/TODO
+++ b/TODO
@@ -2,8 +2,53 @@ The Git Autoinstaller
 
 TODO NOW:
 
 
 TODO NOW:
 

-* Migrate and upgrade the lone mediawiki-1.5.6 install
-* Create the migration script
+- Check how many autoinstalls are missing w bits for
+  daemon.scripts (this would need pyafs)
+- Run parallel-find.pl
+- Migrate all mediawikis
+- Wordpress needs to have a .scripts/update script written for
+  its latest version
+
+NOTES:
+
+- A perfectly formed autoinstall with upgrade paths for all of
+  the intervening versions is not really feasible to implement.
+  As such, we want to migrate everything to -scripts, and then
+  generate a -scripts2 with the correct .scripts directory.
+  We will then nop update some installs, but this will prevent
+  us from having to migrate and update concurrently.  Treat
+  a scripts2 upgrade from migration the same way you would treat
+  a botched scripts upgrade.
+
+- summary and info are still not using loggers. Maybe they should,
+  maybe they shouldn't.  Using loggers means we lose interactivity
+  with the Git output
+
+- Currently all repositories are initialized with --shared, which
+  means they have basically ~no space footprint.  However, it
+  also means that /mit/scripts/wizard/srv MUST NOT lose revs.
+
+- Full fledged logging options. Namely:
+  x all loggers (delay implementing this until we actually have debug stmts)
+    - default is WARNING
+    - debug     => loglevel = DEBUG
+  x stdout logger
+    - default is WARNING (see below for exception)
+    - verbose   => loglevel = INFO
+  x file logger (only allowed for serial processing)
+    - default is OFF
+    - log-file   => loglevel = INFO
+  x database logger (necessary for parallel processing, not implemented)
+    - default is OFF
+    - log-db    => loglevel = INFO
+
+- More on the database logger: it will be very simple with one
+  table named `logs` in SQLite, with columns: `job`, `level`,
+  `message`.  Job identifies the subprocess/thread that emitted
+  the log, so things can be correlated together.  We will then
+  have `wizard dump` which takes a database like this and dumps
+  it into a file logger type file.  The database may also store
+  a queue like structure which can be used to coordinate jobs.

 
 OVERALL PLAN:
 
 
 OVERALL PLAN:
 


@@ -11,7 +56,9 @@ OVERALL PLAN:
   on documenting them.  Specifically, we will be keeping:
 
     - parallel-find.pl, and the resulting
   on documenting them.  Specifically, we will be keeping:
 
     - parallel-find.pl, and the resulting

-/mit/scripts/sec-tools/store/scriptslist
+      /mit/scripts/sec-tools/store/scriptslist
+      This script might need to be adapted if we decide to nuke
+      .scripts-version files.

 
     - The current install scripts will be kept in place, sans changes
       necessary to make them use Git install of copying the script over.
 
     - The current install scripts will be kept in place, sans changes
       necessary to make them use Git install of copying the script over.


@@ -20,63 +67,145 @@ OVERALL PLAN:
       be packaged with rest of our code would be optimal.
 
 * The new procedure for generating an update is as follows:
       be packaged with rest of our code would be optimal.
 
 * The new procedure for generating an update is as follows:

+  (check out the mass-migration instructions for something in this spirit,
+  although uglier in some ways)
+
+    0. ssh into not-backward, temporarily give the daemon.scripts-security-upd
+       bits by blanching it on system:scripts-security-upd, and run parallel-find.pl

 
     1. Have the Git repository and working copy for the project on hand.
 
 
     1. Have the Git repository and working copy for the project on hand.
 

-    2. Download the new tarball
+    2. Checkout the pristine branch

 
 

-    3. Extract the tarball over the working copy (`cp -R a/. b` works well)
+    3. Remove all files from the working copy.  Use `wipe-working-dir`

 
 

-    4. Check if there are any special update procedures, and update the
-       .scripts/update shell script as necessary (this means that any
-       application specific update logic will be kept with the actual
-       source code.  The language of this update script will vary
-       depending on context.)
+    4. Download the new tarball

 
 

-    X. Check for empty directories and add stub files as necessary
-       (use preserve-empty-dir)
+    5. Extract the tarball over the working copy (`cp -R a/. b` works well,
+       remember that the working copy is empty)

 
 

-    5. Commit your changes, and tag as v1.2.3-scripts
+    6. Check for empty directories and add stub files as necessary.
+       Use `preserve-empty-dir`

 
 

-    6. Run the "dry-run script", which uses Git commands to check how many
-       working copies apply the change cleanly, and writes out a logfile
-       with the working copies that don't apply cleanly.
+    7. Git add it all, and then commit as a new pristine version (v1.2.3)

 
 

-    7. Run the "limited run" script, which applies the update to our
-       test-bed, and lets us check the basic functionality of the update.
+    8. Checkout the master branch

 
 

-    8. Run the "deploy" script, which applies the update to all working
-       copies possible, and sends mail to users to whom the working copy
-       did not apply cleanly. (It also frobs .scripts/version)
+    9. [FOR EXISTING REPOSITORIES]
+       Merge the pristine branch in. Resolve any conflicts that our
+       patches have with new changes. Do NOT let Git auto-commit it
+       with --no-commit (otherwise, you want to git commit --amend
+       to keep our history clean

 
 

-    Note: The last three scripts will need to be implemented, with an
-          eye towards speed.
+       [FOR NEW REPOSITORIES]
+       See if any patches are needed to make this run smoothly on
+       scripts.

 
 

-* How to migrate an old autoinstaller to the new autoinstaller
+    [FOR NEW REPOSITORIES]
+        mkdir .scripts
+        echo "Deny from all" > .scripts/.htaccess
+        touch .scripts/update
+        chmod a+x .scripts/update

 
 

-    - Find the oldest tarball/patch set for the application that still
-      is in use and upgradable.
+   10. Check if there are any special update procedures, and update/create the
+       .scripts/update shell script as necessary (this means that any
+       application specific update logic will be kept with the actual
+       source code.  The language of this update script will vary
+       depending on context.)

 
 

-    - Untar, apply patch, place in a directory (unfurl) and git init
+   11. Commit your changes, and tag as v1.2.3-scripts (or scripts2, if
+       you are amending an install without an upstream changes)

 
 

-    - Commit this as the "pristine" version (branch "pristine")
+   12. Test the new update procedure using
+       `wizard upgrade --with=/path/to/repo /your/autoinstall` (this will
+       read out master as your "latest" version).
+       Use git commit --amend to fix any bugs (alternatively, squash them
+       together later).

 
 

-    - Apply scripts patches
+   13. You can also do a "mass" version of this using:
+       `wizard -d testbed.txt massupgrade --with=/path/to/repo app`
+       You'll need perms for any testbed stuff you want.

 
 

-    - Create the .scripts directory and populate it with the interesting
-      information (see below)
+      GET APPROVAL BEFORE PROCEEDING ANY FURTHER

 
 

-    - Commit this as the "scripts" version (branch "master")
+      NOTE: The following commands are to be run on not-backward.mit.edu.
+      You'll need to add daemon.scripts-security-upd to
+      scripts-security-upd to get bits to do this.  Make sure you remove
+      these bits when you're done.

 
 

-* How to update the autoinstaller repository
+   14. Run `wizard research appname`
+       which uses Git commands to check how many
+       working copies apply the change cleanly, and writes out a logfile
+       with the working copies that don't apply cleanly.  It also tells
+       us about "corrupt" working copies.
+
+   15. Run `wizard massupgrade appname`, which applies the update to all working
+       copies possible, and sends mail to users to whom the working copy
+       did not apply cleanly. It also frobs .scripts-version for successful
+       upgrades (maybe not, depending on our plans).
+
+   16. Run parallel-find.pl to update our inventory
+
+* For mass importing into the repository, the steps are:
+  (this probably won't ever be automated, becuase there are fiddly bits)
+
+[TO SET IT UP]
+# let app-1.2.3 be the scripts folder originally in deploydev
+# let this folder be srv/
+# you can also do a git clone
+    mkdir app
+    cd app
+    git init
+    cd ..
+unfurl app-1.2.3 app  # [FIDDLY BIT]
+# NOTE: contents of application are now in app directory
+cd app
+git add .
+git commit -s -m "App 1.2.3"
+git tag v1.2.3
+git branch pristine
+# NOTE: you're still on master branch
+# WARNING: the following operation might require -p1
+patch -p0 < ../app-1.2.3/app-1.2.3.patch  # [FIDDLY BIT]
+# NOTE: please sanity check the patch!
+git add .
+# NOTE: -a flag is to handle if the patch deleted something
+git commit -as -m "App 1.2.3-scripts"
+git tag v1.2.3-scripts
+
+[TO ADD AN UPDATE]
+# let this folder be srv/app.git
+git checkout pristine
+# NOTE: this preserves your .git folder, but removes everything
+wipe-working-dir .
+cd ..
+unfurl app-1.2.3 app  # [FIDDLY BIT]
+cd app
+# NOTE: please sanity check app directory
+git add .
+# NOTE: -a is to take care of deletions
+git commit -as -m "App 1.2.3"
+git tag v1.2.3
+[FIDDLE AROUND. FIDDLE AROUND]
+[IF THE PATCH HAS CHANGED]
+    # You are on the pristine branch
+    # NOTE: Now, the tricky part (this is different from a real update)
+    git symbolic-ref HEAD refs/heads/master
+    # NOTE: Now, we think we're on the master branch, but we have
+    # pristine copy checked out
+    # NOTE: -p0 might need to be twiddled
+    patch -p0 < ../app-1.2.3/app-1.2.3.patch
+    git add .
+    # COMMENT: used to git checkout .scripts here
+    # then check if the directory needs an updated update script
+    # NOTE: Fake the merge
+    git rev-parse pristine > .git/MERGE_HEAD
+[IF THE PATCH HASN'T CHANGED]
+    git checkout master
+    git merge --no-commit pristine
+git commit -as -m "App 1.2.3-scripts"
+git tag v1.2.3-scripts

 
 

-    - Checkout pristine branch
-    - Delete contents of pristine branch (excluding .git, try rm -Rf * and remove stragglers)
-    - Unfurl tarball
-    - Commit as new pristine branch
-    - Checkout scripts branch
-    - Merge pristine branch
-    - Fix as necessary

 
 * The repository for a given application will contain the following files:
 
 
 * The repository for a given application will contain the following files:
 


@@ -94,37 +223,26 @@ OVERALL PLAN:
         * .scripts/database (generated) contains the database the
           user installed the script to, so scripts-remove can clean it
 
         * .scripts/database (generated) contains the database the
           user installed the script to, so scripts-remove can clean it
 

+            XXX: Could cause problems if a user copies the autoinstall,
+            fiddles with the DB credentials, and then scripts-remove's
+            the autoinstall.  Possible fix is to add the original
+            directory as a sanity check.  Additionally, we could have
+            the application read out of this file.
+

         * .scripts/version (generated) which contains the version
           last autoinstalled (as distinct from the actual version
           the script is) (This is the same as .scripts-version right
           now; probably want to keep that for now)
 
         * .scripts/version (generated) which contains the version
           last autoinstalled (as distinct from the actual version
           the script is) (This is the same as .scripts-version right
           now; probably want to keep that for now)
 

-    - Because there will be no .gitignore file, you *must not* run
-      `git add .` on an actual running copy of the application.
-      `git add -u .` will generally be safe, but preferred mode
-      of operation is to operate on a clean install.
-
-* The migration process shall be as such:
-
-    1. git init
-
-    2. git remote add origin /foo
+            XXX: It's unclear if we want to move to this wholesale, or
+            delay this indefinitely.  quentin thinks that the Git
+            repository itself is a sufficient record.

 
 

-    3. git config branch.master.merge refs/heads/master
+* The migration process has been implemented, see 'wizard migrate'.

 
 

-    4. git fetch origin
-
-    5. git reset v1.2.3-scripts
-
-    5. git checkout .scripts
-
-    6. Setup .scripts/version (probably pipe the output of real-version)
-        UNCLEAR if this is a good thing; if it is, make sure we add
-        a .gitignore to the .scripts directory
-
-* We will not add special code to handle .htaccess; thus the kernel patch
-  for allowing Apache access to .htaccess sent to scripts-team@mit.edu
-  must be handled first.
+    XXX: We have not decided what migration should do to .scripts-version;
+    if it does move it to .scripts, repositories should have a .gitignore
+    in those directories

 
 * The autoupgrade shall be the process of:
 
 
 * The autoupgrade shall be the process of:
 


@@ -136,7 +254,15 @@ OVERALL PLAN:
     ./.scripts/update
     # Make it accessible
 
     ./.scripts/update
     # Make it accessible
 

-  (with some more robust error checking)
+  (with some more robust error checking, a proper dry run mechanism to, and
+  lots of su'ing)
+
+* All code that operates on an untrusted Git repository, or runs
+  executable code, should be done on NOT-BACKWARD.mit.edu.  Pending
+  accounts confirmation, it will also get a principal
+  daemon.scripts-security-upd, which is what we'll actually put
+  in the scripts-security-upd group.  parallel-find.pl should also
+  be run on not-backward, by virtue of its fat pipe to the AFS servers.

 
 

-* Make install-statistics generate nice pretty graphs of installs by date
+* Make 'wizard summary' generate nice pretty graphs of installs by date

   (more histograms, will need to check actual .scripts-version files.)
   (more histograms, will need to check actual .scripts-version files.)