5 - Make sure massmigrate cleanly ignores already migrated
7 - Make sure MediaWiki repository is as close to perfect
9 - Do an install, migrate and then `git status`
10 - Check out possible missing php.ini's
11 - Remove "Merge comments" from lines
12 - Fix Signed-off-by lines
13 - Add some safeguard code to make sure you don't run migrate
16 - We have safe, non-braindead
17 version detection with `git describe --tags`. Switch
18 everything to use it. (I think the only thing left is
20 - wizard.util is pretty braindead at this point. Fix up
21 the wildly varying conventions in it.
22 - Move migration code into Wizard, since we already deal
23 with installation there anyway.
25 - Better error message if daemon/scripts-security-upd
26 is not on scripts-security-upd list
28 - Fix retarded logging mechanism
30 - The great initial deploy:
31 - Turn on mediawiki new autoinstaller
32 - Migrate all mediawiki installs
35 - Build automation for generating config files; this automation
36 will be shared with the migrate script and the installer script
37 (migrate script needs to be able to pull out values from config
38 file, so will we; installer script needs to be able to run
39 the installer to generate config files, so will this)
40 - This should all be automated:
41 - Wordpress needs to have .scripts dir in all -scripts versions
42 (also make .scripts/.htaccess)
43 - Wordpress needs to have a .scripts/update script written for
44 its latest version (do this after its migration)
45 - Wordpress needs to check for php.ini files (which it almost
46 certianly has) and commit messages
47 - Wordpress needs user config and php.ini links made
49 - Summary script should be more machine friendly, and should not
50 output summary charts when I increase specificity
51 - Summary script needs to be updated for new format
53 Some other stuff to do in your copious free time:
54 - Check how many autoinstalls are missing w bits for
55 daemon.scripts (this would need pyafs)
56 - Make scripts AFS patch advertise its existence so we can check for it.
57 (This might be otherwise possible using `fs sysname`
58 - Implement proper deploy log parsing; this basically means we
59 need to be able to introspect Git Log. Consider using git-python
61 - Make 'wizard summary' generate nice pretty graphs of installs by date
62 (more histograms, will need to check actual .scripts-version files.)
63 - It should be able to handle installs like Django where there's a component
64 that gets installed in web_scripts and another directory that gets installed
67 PULLING OUT CONFIGURATION FILES IN AN AUTOMATED MANNER
69 advancedpoll: Template file to fill out
70 django: Noodles of template files
71 gallery2: Multistage install process
73 mediawiki: One-step install process
74 phpbb: Multistage install process
75 phpical: Template file
78 wordpress: Multistage install process
80 PHILOSOPHY ABOUT LOGGING
82 Logging is most useful when performing a mass run. This
83 includes things such as mass-migration as well as when running
84 summary reports. An interesting property about mass-migration
85 or mass-upgrade, however, is that if they fail, they are
86 idempotent, so an individual case can be debugged simply running
87 the single-install equivalent with --debug on. (This, indeed,
88 may be easier to do than sifting through a logfile).
90 It is a different story when you are running a summary report:
91 you are primarily bound by your AFS cache and how quickly you can
92 iterate through all of the autoinstalls. Checking if a file
93 exists on a cold AFS cache may
94 take several minutes to perform; on a hot cache the same report
95 may take a mere 3 seconds. When you get to more computationally
96 expensive calculations, however, even having a hot AFS cache
97 is not enough to cut down your runtime.
99 There are certain calculations that someone may want to be
100 able to perform on manipulated data. As such, this data should
101 be cached on disk, if the process for extracting this data takes
102 a long time. Also, for usability sake, Wizard should generate
103 the common case reports.
105 Ensuring that machine parseable reports are made, and then making
106 the machinery to reframe this data, increases complexity. Therefore,
107 the recommendation is to assume that if you need to run iteratively,
108 you'll have a hot AFS cache at your fingerprints, and if that's not
109 fast enough, then cache the data.
111 COMMIT MESSAGE FIELDS:
113 Installed-by: username@hostname
114 Pre-commit-by: Real Name <username@mit.edu>
115 Upgraded-by: Real Name <username@mit.edu>
116 Migrated-by: Real Name <username@mit.edu>
117 Wizard-revision: abcdef1234567890
118 Wizard-args: /wizard/bin/wizard foo bar baz
122 Committer: Real Name <username@mit.edu>
123 Author: lockername locker <lockername@scripts.mit.edu>
127 - It is not expected or required for update scripts to exist for all
128 intervening versions that were present pre-migration; only for it
129 to work on the most recent migration.
131 - Currently all repositories are initialized with --shared, which
132 means they have basically ~no space footprint. However, it
133 also means that /mit/scripts/wizard/srv MUST NOT lose revs.
135 - Full fledged logging options. Namely:
136 x all loggers (delay implementing this until we actually have debug stmts)
138 - debug => loglevel = DEBUG
140 - default is WARNING (see below for exception)
141 - verbose => loglevel = INFO
142 x file logger (only allowed for serial processing)
144 - log-file => loglevel = INFO
145 x database logger (necessary for parallel processing, not implemented)
147 - log-db => loglevel = INFO
149 - More on the database logger: it will be very simple with one
150 table named `logs` in SQLite, with columns: `job`, `level`,
151 `message`. Job identifies the subprocess/thread that emitted
152 the log, so things can be correlated together. We will then
153 have `wizard dump` which takes a database like this and dumps
154 it into a file logger type file. The database may also store
155 a queue like structure which can be used to coordinate jobs.
159 * Some parts of the infrastructure will not be touched, although I plan
160 on documenting them. Specifically, we will be keeping:
162 - parallel-find.pl, and the resulting
163 /mit/scripts/.htaccess/scripts/sec-tools/store/scriptslist
165 * The new procedure for generating an update is as follows:
166 (check out the mass-migration instructions for something in this spirit,
167 although uglier in some ways; A indicates the step /should/ be automated.)
169 0. ssh into not-backward, temporarily give the daemon.scripts-security-upd
170 bits by blanching it on system:scripts-security-upd, and run parallel-find.pl
172 A 1. Have the Git repository and working copy for the project on hand.
174 A 2. Checkout the pristine branch
176 A 3. Remove all files from the working copy. Use `wipe-working-dir`
178 A 4. Download the new tarball
180 A 5. Extract the tarball over the working copy (`cp -R a/. b` works well,
181 remember that the working copy is empty; this needs some intelligent
184 A 6. Check for empty directories and add stub files as necessary.
185 Use `preserve-empty-dir`
187 A 7. Git add it all, and then commit as a new pristine version (v1.2.3)
189 A 8. Checkout the master branch
191 9. [FOR EXISTING REPOSITORIES]
192 Merge the pristine branch in. Resolve any conflicts that our
193 patches have with new changes. Do NOT let Git auto-commit it
194 with --no-commit (otherwise, you want to git commit --amend
195 to keep our history clean
197 [FOR NEW REPOSITORIES]
198 See if any patches are needed to make this run smoothly on
201 [FOR NEW REPOSITORIES]
203 A echo "Deny from all" > .scripts/.htaccess
204 touch .scripts/update
205 chmod a+x .scripts/update
207 10. Check if there are any special update procedures, and update/create the
208 .scripts/update shell script as necessary (this means that any
209 application specific update logic will be kept with the actual
210 source code. The language of this update script will vary
211 depending on context.)
213 11. Commit your changes, and tag as v1.2.3-scripts (or scripts2, if
214 you are amending an install without an upstream changes)
216 NOTE: These steps should be run on a scripts server
218 12. Test the new update procedure using
219 `wizard upgrade --with=/path/to/repo /your/autoinstall` (this will
220 read out master as your "latest" version).
221 Use git commit --amend to fix any bugs (alternatively, squash them
224 13. You can also do a "mass" version of this using:
225 `wizard -d testbed.txt massupgrade --with=/path/to/repo app`
226 You'll need perms for any testbed stuff you want. (not implemented)
228 GET APPROVAL BEFORE PROCEEDING ANY FURTHER
230 NOTE: The following commands are to be run on not-backward.mit.edu.
231 You'll need to add daemon.scripts-security-upd to
232 scripts-security-upd to get bits to do this. Make sure you remove
233 these bits when you're done.
235 14. Run `wizard research appname`
236 which uses Git commands to check how many
237 working copies apply the change cleanly, and writes out a logfile
238 with the working copies that don't apply cleanly. It also tells
239 us about "corrupt" working copies, i.e. working copies that
240 have over a certain threshold of changes.
242 15. Run `wizard massupgrade appname`, which applies the update to all working
243 copies possible, and sends mail to users to whom the working copy
244 did not apply cleanly.
246 16. Run parallel-find.pl to update our inventory
248 * For mass importing into the repository, the steps are:
249 (this probably won't ever be automated, becuase there are fiddly bits)
252 # let app-1.2.3 be the scripts folder originally in deploydev
253 # let this folder be srv/
254 # you can also do a git clone
259 unfurl app-1.2.3 app # [FIDDLY BIT]
260 # NOTE: contents of application are now in app directory
263 git commit -s -m "App 1.2.3"
266 # NOTE: you're still on master branch
267 # WARNING: the following operation might require -p1
268 patch -p0 < ../app-1.2.3/app-1.2.3.patch # [FIDDLY BIT]
269 # NOTE: please sanity check the patch!
271 # NOTE: -a flag is to handle if the patch deleted something
272 git commit -as -m "App 1.2.3-scripts"
273 git tag v1.2.3-scripts
276 # let this folder be srv/app.git
277 git checkout pristine
278 # NOTE: this preserves your .git folder, but removes everything
281 unfurl app-1.2.3 app # [FIDDLY BIT]
283 # NOTE: please sanity check app directory
285 # NOTE: -a is to take care of deletions
286 git commit -as -m "App 1.2.3"
288 [FIDDLE AROUND. FIDDLE AROUND]
289 [IF THE PATCH HAS CHANGED]
290 # You are on the pristine branch
291 # NOTE: Now, the tricky part (this is different from a real update)
292 git symbolic-ref HEAD refs/heads/master
293 # NOTE: Now, we think we're on the master branch, but we have
294 # pristine copy checked out
295 # NOTE: -p0 might need to be twiddled
296 patch -p0 < ../app-1.2.3/app-1.2.3.patch
298 # COMMENT: used to git checkout .scripts here
299 # then check if the directory needs an updated update script
300 # NOTE: Fake the merge
301 git rev-parse pristine > .git/MERGE_HEAD
302 [IF THE PATCH HASN'T CHANGED]
304 git merge --no-commit pristine
305 git commit -as -m "App 1.2.3-scripts"
306 git tag v1.2.3-scripts
308 * The repository for a given application will contain the following files:
310 - The actual application's files, as from the official tarball
312 - A .scripts directory, which contains the following information:
314 * .scripts/update shell script (with the +x bit set appropriately),
315 which performs the commands necessary to update a script. This can
316 be in any language. (XXX: This is going to get removed soon)
318 * .scripts/.htaccess to prevent this directory from being accessed
321 * .scripts/old-version (optional) the old value of .scripts-version,
322 basically used for reverting an install to pre-migrated state.
324 * .scripts/lock (generated) which locks the autoinstall during an upgrade