TODO NOW:
-- Something needs to be done if disk quota is exceeded:
- - Catch the OSError and throw a domain-specific error
- so massmigrate can deal gracefully
- - Perform an added memory calculation, check this against
- remaining quotai, and bail out if it's within some
- percentage of their remaining quota
- - Checks should also be performed against the partition
- X with the new --shared flag this may not be necessary
- as repos weighs less than 200K
-- Should write to a "processed" file to make resuming with
- unexpected failure faster (possibly should be database if
- parallelized).
-- Figure out how to make su'ing work in Python. Probably os.setuid
- and some forks. IPC is a bitch.
-- Make version directory also work with a location (with no
- colon) format; this will make making fake "testbed" directories
- easier to do. Also let it accept a file, instead of a directory.
-- Check how many autoinstalls are missing w bits for
- daemon.scripts
-- Run parallel-find.pl
-- Migrate all mediawikis
+- Better error message if daemon/scripts-security-upd
+ is not on scripts-security-upd list
+
+- Add repository flag to migrate so that we can specify an
+ arbitrary repository to migrate to
+
+- Build automation for generating config files; this automation
+ will be shared with the migrate script and the installer script
+ (migrate script needs to be able to pull out values from config
+ file, so will we; installer script needs to be able to run
+ the installer to generate config files, so will this)
+
+- The great initial deploy:
+ - Turn on mediawiki new autoinstaller
+ - Migrate all mediawiki installs
+
+- Consider making usermode wizard operation a support mode
+ (mostly for letting users upgrade things themself)
+
+- Implement proper deploy log parsing; this basically means we
+ need to be able to introspect Git Log. Consider using git-python
+ for this. There's also missing functionality, bad error handling
+ and hacks in the prototype implementation of upgrade
+
+- Wordpress needs to have .scripts dir in all -scripts versions
+ (remember --no-walk!) (also make .scripts/.htaccess)
- Wordpress needs to have a .scripts/update script written for
- its latest version
+ its latest version (do this after its migration)
+- Wordpress needs to check for php.ini files (which it almost
+ certianly has) and commit messages
+- Wordpress needs user config and php.ini links made
+
+- Summary script should be more machine friendly, and should not
+ output summary charts when I increase specificity
+- Check how many autoinstalls are missing w bits for
+ daemon.scripts (this would need pyafs)
+- Consider fixing Wizard's commit messages
+
+PULLING OUT CONFIGURATION FILES IN AN AUTOMATED MANNER
+
+advancedpoll: Template file to fill out
+django: Noodles of template files
+gallery2: Multistage install process
+joomla: Template file
+mediawiki: One-step install process
+phpbb: Multistage install process
+phpical: Template file
+trac: NFC
+turbogears: NFC
+wordpress: Multistage install process
+
+PHILOSOPHY ABOUT LOGGING
+
+Logging is most useful when performing a mass run. This
+includes things such as mass-migration as well as when running
+summary reports. An interesting property about mass-migration
+or mass-upgrade, however, is that if they fail, they are
+idempotent, so an individual case can be debugged simply running
+the single-install equivalent with --debug on. (This, indeed,
+may be easier to do than sifting through a logfile).
+
+It is a different story when you are running a summary report:
+you are primarily bound by your AFS cache and how quickly you can
+iterate through all of the autoinstalls. Checking if a file
+exists on a cold AFS cache may
+take several minutes to perform; on a hot cache the same report
+may take a mere 3 seconds. When you get to more computationally
+expensive calculations, however, even having a hot AFS cache
+is not enough to cut down your runtime.
+
+There are certain calculations that someone may want to be
+able to perform on manipulated data. As such, this data should
+be cached on disk, if the process for extracting this data takes
+a long time. Also, for usability sake, Wizard should generate
+the common case reports.
+
+Ensuring that machine parseable reports are made, and then making
+the machinery to reframe this data, increases complexity. Therefore,
+the recommendation is to assume that if you need to run iteratively,
+you'll have a hot AFS cache at your fingerprints, and if that's not
+fast enough, then cache the data.
+
+COMMIT MESSAGE FIELDS:
+
+Installed-by: username@hostname
+Pre-commit-by: Real Name <username@mit.edu>
+Upgraded-by: Real Name <username@mit.edu>
+Migrated-by: Real Name <username@mit.edu>
+Wizard-revision: abcdef1234567890
+Wizard-args: /wizard/bin/wizard foo bar baz
+
+GIT COMMIT FIELDS:
+
+Committer: Real Name <username@mit.edu>
+Author: lockername locker <lockername@scripts.mit.edu>
NOTES:
a scripts2 upgrade from migration the same way you would treat
a botched scripts upgrade.
-- summary and info are still not using loggers. Maybe they should,
- maybe they shouldn't. Using loggers means we lose interactivity
- with the Git output
-
- Currently all repositories are initialized with --shared, which
means they have basically ~no space footprint. However, it
also means that /mit/scripts/wizard/srv MUST NOT lose revs.
+- Full fledged logging options. Namely:
+ x all loggers (delay implementing this until we actually have debug stmts)
+ - default is WARNING
+ - debug => loglevel = DEBUG
+ x stdout logger
+ - default is WARNING (see below for exception)
+ - verbose => loglevel = INFO
+ x file logger (only allowed for serial processing)
+ - default is OFF
+ - log-file => loglevel = INFO
+ x database logger (necessary for parallel processing, not implemented)
+ - default is OFF
+ - log-db => loglevel = INFO
+
+- More on the database logger: it will be very simple with one
+ table named `logs` in SQLite, with columns: `job`, `level`,
+ `message`. Job identifies the subprocess/thread that emitted
+ the log, so things can be correlated together. We will then
+ have `wizard dump` which takes a database like this and dumps
+ it into a file logger type file. The database may also store
+ a queue like structure which can be used to coordinate jobs.
+
OVERALL PLAN:
* Some parts of the infrastructure will not be touched, although I plan
on documenting them. Specifically, we will be keeping:
- parallel-find.pl, and the resulting
- /mit/scripts/sec-tools/store/scriptslist
+ /mitalso make .scripts/.htaccess/scripts/sec-tools/store/scriptslist
This script might need to be adapted if we decide to nuke
.scripts-version files.
git commit -as -m "App 1.2.3-scripts"
git tag v1.2.3-scripts
-
* The repository for a given application will contain the following files:
- The actual application's files, as from the official tarball
directory as a sanity check. Additionally, we could have
the application read out of this file.
- * .scripts/version (generated) which contains the version
- last autoinstalled (as distinct from the actual version
- the script is) (This is the same as .scripts-version right
- now; probably want to keep that for now)
+ * .scripts/old-version (optional) the old value of .scripts-versoin
- XXX: It's unclear if we want to move to this wholesale, or
- delay this indefinitely. quentin thinks that the Git
- repository itself is a sufficient record.
-
-* The migration process has been implemented, see 'wizard migrate'.
-
- XXX: We have not decided what migration should do to .scripts-version;
- if it does move it to .scripts, repositories should have a .gitignore
- in those directories
+ * .scripts/install (eventually) interactively installs the
+ applicatoin from command line.
* The autoupgrade shall be the process of:
(with some more robust error checking, a proper dry run mechanism to, and
lots of su'ing)
-* All code that operates on an untrusted Git repository, or runs
- executable code, should be done on NOT-BACKWARD.mit.edu. Pending
- accounts confirmation, it will also get a principal
- daemon.scripts-security-upd, which is what we'll actually put
- in the scripts-security-upd group. parallel-find.pl should also
- be run on not-backward, by virtue of its fat pipe to the AFS servers.
-
* Make 'wizard summary' generate nice pretty graphs of installs by date
(more histograms, will need to check actual .scripts-version files.)
+
+* Update AFS patch to advertise its existence, so we can check for it
+ here.
scripts.mit.edu / wizard.git / blobdiff