The Git Autoinstaller

TODO NOW:

- Fix mediawiki repository (has lots of stale files and a weird
  history. This also means that the ichuang wiki will need to be
  remigrated; frob .scripts-version to make it acceptable, but
  otherwise should be fairly trivial).  When I say "fix", I mean
  "redo". Thorough documentation would be good too.  Some bits
  can be automated and/or have tools to assist it
- Whiteboard the flow for performing an upgrade on a single
  install. How assisted does it need to be?
- Conduct migration tool testing
- Create mass-migration tool (should be able to limit on mediawiki)
- Run parallel-find.pl
- Migrate all mediawikis

OVERALL PLAN:

* Some parts of the infrastructure will not be touched, although I plan
  on documenting them.  Specifically, we will be keeping:

    - parallel-find.pl, and the resulting
/mit/scripts/sec-tools/store/scriptslist

    - The current install scripts will be kept in place, sans changes
      necessary to make them use Git install of copying the script over.
      Porting these scripts to Python and making them modular would be
      nice, but is priority.  For the long term, seeing this scripts
      be packaged with rest of our code would be optimal.

* The new procedure for generating an update is as follows (this is
  also similar to procedure for creating these repositories):

    1. Have the Git repository and working copy for the project on hand.

    2. Checkout the pristine branch

    3. Remove all files from the working copy (rm -Rf *, and then delete
       any dot stragglers.  A script to do this would be handy)

    4. Download the new tarball

    5. Extract the tarball over the working copy (`cp -R a/. b` works well,
       remember that the working copy is empty)

    6. Check for empty directories and add stub files as necessary
       (use preserve-empty-dir)

    7. Git add it all, and then commit as a new pristine version (v1.2.3)

    8. Checkout the master branch

    9. [FOR EXISTING REPOSITORIES]
       Merge the pristine branch in. Resolve any conflicts that our
       patches have with new changes. Do NOT let Git auto-commit it
       with --no-commit (otherwise, you want to git commit --amend
       to keep our history clean

       [FOR THE FIRST TIME]
       Apply the scripts patch that was used for that version here
       (usually patch -p1 < patch)

   10. Check if there are any special update procedures, and update the
       .scripts/update shell script as necessary (this means that any
       application specific update logic will be kept with the actual
       source code.  The language of this update script will vary
       depending on context.)

   11. Commit your changes, and tag as v1.2.3-scripts

   If you're setting up a repository from scratch, stop here, and
   repeat as necessary

       XXX: Should we force people to push to the real repository at
       this point, or just make the repository that the script pulls
       stuff out of configurable? (Twiddling origin can get you a
       devel setup with no code changes)

   12. Run the "dry-run script", which uses Git commands to check how many
       working copies apply the change cleanly, and writes out a logfile
       with the working copies that don't apply cleanly.

   13. Run the "limited run" script, which applies the update to our
       test-bed, and lets us check the basic functionality of the update.
       This can include a script that lets us update a single directory
       with verbose output.

   14. Run the "deploy" script, which applies the update to all working
       copies possible, and sends mail to users to whom the working copy
       did not apply cleanly. It also frobs .scripts/version for successful
       upgrades.

   15. Run parallel-find.pl

* For mass importing into the repository, the steps are:

[TO SET IT UP]
# let app-1.2.3 be the scripts folder originally in deploydev
# let this folder be srv/
mkdir app
cd app
git init
cd ..
unfurl app-1.2.3 app
# NOTE: contents of application are now in app directory
cd app
git add .
git commit -s -m "App 1.2.3"
git tag v1.2.3
git branch pristine
# NOTE: you're still on master branch
mkdir .scripts
echo "Deny from all" > .scripts/.htaccess
touch .scripts/update
chmod a+x .scripts/update
# OPERATION: create the update script
# WARNING: the following operation might require -p1
patch -p0 < ../app-1.2.3/app-1.2.3.patch
# NOTE: please sanity check the patch!
git add .
# NOTE: -a flag is to handle if the patch deleted something
git commit -as -m "App 1.2.3-scripts"
git tag v1.2.3-scripts

[TO ADD AN UPDATE]
# let this folder be srv/app.git
git checkout pristine
# NOTE: this preserves your .git folder, but removes everything
wipe-working-dir .
cd ..
unfurl app-1.2.3 app
cd app
# NOTE: please sanity check app directory
git add .
# NOTE: -a is to take care of deletions
git commit -as -m "App 1.2.3"
git tag v1.2.3
[IF THE PATCH HAS CHANGED]
    # NOTE: Now, the tricky part (this is different from a real update)
    git symbolic-ref HEAD refs/heads/master
    # NOTE: Now, we think we're on the master branch, but we have
    # pristine copy checked out
    # NOTE: -p0 might need to be twiddled
    patch -p0 < ../app-1.2.3/app-1.2.3.patch
    git add .
    # NOTE: DON'T FORGET THIS STEP!!! Otherwise fixing this is going
    # to be painful later down the line
    git checkout .scripts
    # OPERATION: Check if the directory needs an updated update script
    # NOTE: Fake the merge
    git rev-parse pristine > .git/MERGE_HEAD
[IF THE PATCH HASN'T CHANGED]
    git checkout master
    git merge --no-commit pristine
git commit -as -m "App 1.2.3-scripts"
git tag v1.2.3-scripts


* The repository for a given application will contain the following files:

    - The actual application's files, as from the official tarball

    - A .scripts directory, which contains the following information:

        * .scripts/update shell script (with the +x bit set appropriately),
          which performs the commands necessary to update a script.  This can
          be in any language.

        * .scripts/.htaccess to prevent this directory from being accessed
          from the web.

        * .scripts/database (generated) contains the database the
          user installed the script to, so scripts-remove can clean it

            XXX: Could cause problems if a user copies the autoinstall,
            fiddles with the DB credentials, and then scripts-remove's
            the autoinstall.  Possible fix is to add the original
            directory as a sanity check.  Additionally, we could have
            the application read out of this file.

        * .scripts/version (generated) which contains the version
          last autoinstalled (as distinct from the actual version
          the script is) (This is the same as .scripts-version right
          now; probably want to keep that for now)

            XXX: It's unclear if we want to move to this wholesale, or
            delay this indefinitely.

* The migration process has been implemented, see 'wizard migrate'.

    XXX: We have not decided what migration should do to .scripts-version;
    if it does move it to .scripts, repositories should have a .gitignore
    in those directories

* The autoupgrade shall be the process of:

    # Make the directory not accessible by the outside world (htaccess, but be careful!)
    git add -u .
    git commit -m 'automatically generated backup'
    git pull origin master
    if [ $? ne 0 ]; then git reset --hard; echo 'conflicts during upgrade'; fi
    ./.scripts/update
    # Make it accessible

  (with some more robust error checking)

* Make 'wizard summary' generate nice pretty graphs of installs by date
  (more histograms, will need to check actual .scripts-version files.)