6 One of Wizard's goals is to replace the previous autoinstaller infrastructure.
7 Pre-wizard autoinstalls live in :file:`/mit/scripts/deploy` and consist of a
8 tarball from upstream, possibly a scripts patch, and possibly some post-install
9 munging (such as the creation of a :file:`php.ini` file and appropriate
12 Conversion to use Wizard involves placing :term:`pristine` versions of the source
13 code (from the upstream tarballs) and appropriately patched scripts
14 versions into a Git repository, as well as writing a :mod:`wizard.app`
15 module for the application that implements application specific logic, such
16 as how to install, upgrade or backup the installation.
18 Here is a tutorial for performing a conversion, using Wordpress as
19 an example. We will implement only the functions necessary for installing
20 an application--upgrades and backups are not described here.
24 This guide duplicates a lot of what would be covered if you were
25 discussing how to create a new application from scratch. Those
26 bits should be separated out and put in their own document.
31 Probably the easiest way to do development is entirely on :term:`AFS`: all
32 of your source code should live in publically readable (i.e.
33 ``system:anyuser`` as read permissions) directories, so that if you
34 SSH into a scripts server to perform testing, you will be able
35 to invoke your tools and read your development repository. In order
36 to be able to run the test scripts in the tests directory, this
37 is preferably in :file:`web_scripts`. In that
38 case, setup is as simple as::
40 git clone /mit/scripts/git/wizard.git /mit/$USER/web_scripts/wizard
41 athrun consult fsr /mit/$USER/web_scripts/wizard system:anyuser read
42 # for any application you're going to do development on, also:
43 git clone /mit/scripts/git/autoinstalls/$APP.git /mit/$USER/web_scripts/wizard/srv/$APP
45 If you'd like to be able to develop offline, just note that you will
46 have to push your changes to AFS once you start doing testing on
47 scripts servers, but before your changes get incorporated into
48 canonical upstream. Git doesn't exactly make this easy, but you
49 can follow this recipe::
51 git clone /mit/scripts/git/wizard.git ~/wizard
57 git remote add afs /mit/$USER/wizard.git
58 git push -f afs master
59 git clone /mit/$USER/wizard.git /mit/$USER/wizard
61 And then you can perform updates from your local copy with::
67 If :file:`/mit/$USER/wizard.git` has write permissions for
68 ``system:scripts-security-upd``, this is especially useful if you were hacking
69 on a copy living on ``not-backward.mit.edu``, and now need to transfer the
70 changes back to the canonical repository (please don't give ``not-backward.mit.edu``
71 your root tickets!) You can also setup a wizard directory similar to the
72 first set of directions for on-server testing.
74 From this point on, we will assume you are doing development from an AFS directory
75 named ``$WIZARD``; note that application repositories live in ``$WIZARD/srv``.
80 This is a tutorial centered around migrating `Wordpress <http://wordpress.org/>`_
81 into our Git repository. For the sake of demonstration,
82 we shall assume that this repository hasn't been created yet.
83 The repository then doesn't exist, we should create it::
90 We also have to create a module for the application, so we
91 create ``$WIZARD/wizard/app/wordpress.py`` and fill it in with a bare bones template:
93 .. code-block:: python
99 from wizard import app, install, resolve, sql, util
100 from wizard.app import php
102 class Application(app.Application):
105 Now we are ready to put some code in our repository: the first thing we will
106 add is the :term:`pristine` branch, which contains verbatim the code from upstream.
107 If we were starting a new autoinstaller, we'd pop off and use the latest version,
108 but since we're dealing with legacy we want to start our repository history
109 with the **oldest** version still extant on our servers. To find this out run::
111 wizard summary version APP
113 You'll need to be in the ``scripts-team`` list in order to access the data to
116 Try running the following command in :file:`$WIZARD/srv/wordpress`::
118 wizard prepare-pristine wordpress-2.0.2
120 You should get an error complaining about :meth:`wizard.app.Application.download`
121 not being implemented yet. Let's fix that:
123 .. code-block:: python
125 class Application(app.Application):
126 def download(self, version):
127 return "http://wordpress.org/wordpress-%s.tar.gz" % version
129 We determined this by finding `Wordpress's Release Archive <http://wordpress.org/download/release-archive/>`_
130 and inferring the naming scheme by inspecting various links. You should now
131 be able to run the prepare-pristine command successfully: when it is
132 done, you'll now have a bunch of files in your repository, and they
133 will be ready to be committed. Inspect the files and commit (note that the
134 format of the commit message is a plain Appname Version.Number)::
137 git commit -asm "Wordpress 2.0.2"
138 git tag wordpress-2.0.2
142 Sometimes, ``http://wordpress.org/wordpress-2.0.2.tar.gz`` won't
143 actually exist anymore (it didn't exist when we did it). In this case,
144 you'll probably be able to find the original tarball in
145 :file:`/mit/scripts/deploy/wordpress-2.0.2`, and you can feed it
146 manually to prepare pristine with
147 ``wizard prepare-pristine /mit/scripts/deploy/wordpress-2.0.2/wordpress-2.0.2.tar.gz``
149 Some last house-keeping bits: now that you have a commit in a repository, you
150 can also create a pristine branch::
154 From the point of view of the pristine branch pointer, history should be a
155 straight-forward progression of versions. Once you have more versions,
156 it will look something like:
158 .. digraph:: pristine_dag
162 subgraph cluster_pristine {
175 In a perfect world, the pristine version would be equivalent to the scriptsified
176 version that would actually get deployed. However, we have historically needed
177 to apply patches and add extra configuration files to get applications to
178 work correctly. Due to the way Git's merge algorithm works, the closer we are
179 able to reconstruct a version of the application that was actually used, the
180 better off we will be when we try to subsequently upgrade those applications.
182 To give you an idea of what the Git history graph will look like when we
183 are done, here is the graph from before, but augmented with the scripts versions:
185 .. digraph:: master_dag
189 subgraph cluster_pristine {
198 subgraph cluster_master {
200 as [label="1.0-scripts"]
201 bs [label="1.1-scripts"]
202 cs [label="2.0-scripts"]
212 First things first: verify that we are on the master branch::
216 Check for pre-existing patches in the old application directory,
217 :file:`/mit/scripts/deploy/wordpress-2.0.2` in the case of Wordpress,
220 patch -n0 < /mit/scripts/deploy/wordpress-2.0.2/wordpress.patch
222 Then, run the following command to setup a :file:`.scripts` directory::
226 This directory holds Wizard related files, and is also used by
227 :command:`parallel-find.pl` to determine if a directory is an autoinstall.
229 Finally, if you are running a PHP application, you'll need to setup
230 a :file:`php.ini` and symlinks to it in all subdirectories::
232 cp /mit/scripts/deploy/php.ini/wordpress php.ini
233 athrun scripts fix-php-ini
237 As of November 2009, all PHP applications load the same :file:`php.ini` file.
239 Now commit, but don't get too attached to your commit; we're going
240 to be heavily modifying it soon::
242 git commit -asm "Wordpress 2.0.2-scripts"
247 We now need to make it possible for a user to install the application.
248 Most web applications have a number of web scripts for generating a
249 configuration file, so creating the install script involves:
251 1. Determining what input values you will need from the user, such
252 as a title for the new application or database credentials; more
255 2. Determining what POST values need to be sent to what URLs.
256 Since you're converting a repository, this job is even simpler: you just
257 need to port the Perl script that was originally used into Python.
259 There's an in-depth explanation of named input values in
260 :mod:`wizard.install`. The short version is that your application
261 contains a class-wide :data:`~wizard.app.Application.install_schema`
262 attribute that encodes this information. Instantiate it with
263 :class:`wizard.install.ArgSchema` (passing in arguments to get
264 some pre-canned input values), and then add application specific
265 arguments by passing instances of :class:`wizard.install.Arg`
266 to the method :meth:`~wizard.install.ArgSchema.add`. Usually you should
267 be able to get away with pre-canned attributes. You can access
268 these arguments inside :meth:`~wizard.app.Application.install` via
269 the ``options`` value.
271 Some tips and tricks for writing :meth:`wizard.app.Application.install`:
273 * Some configuration file generators will get unhappy if the
274 target directory is not chmod'ed to be writable; dropping
275 in a ``os.chmod(dir, 0777)`` and then undoing the chmod
276 when you're done is a decent workaround.
278 * :func:`wizard.install.fetch` is the standard workhorse for making
279 requests to applications. It accepts three parameters; the first
280 is ``options`` (which was the third argument to ``install`` itself),
281 the second is the page to query, relative to the installation's
282 web root, and ``post`` is a dictionary of keys to values to POST.
284 * You should log any web page output using :func:`logging.debug`.
286 * If you need to manually manipulate the database afterwards, you
287 can use :func:`wizard.sql.mysql_connect` (passing it ``options``)
288 to get a `SQLAlchemy metadata object
289 <http://www.sqlalchemy.org/docs/05/sqlexpression.html>`_, which can
290 consequently be queried. For convenience, we've bound metadata
291 to the connection, you can perform implicit execution.
295 Our installer needs to also parametrize :file:`php.ini`, which we haven't
298 To test if your installation function works, it's probably convenient to
299 create a test script in :file:`tests`; :file:`tests/test-install-wordpress.sh`
300 in the case of Wordpress. It will look something like::
305 TESTNAME="install_wordpress"
308 wizard install "wordpress-$VERSION-scripts" "$TESTDIR" --non-interactive -- --title="My Blog"
310 ``DEFAULT_HEAD=1`` indicates that this script can perform a reasonable
311 operation without any version specified (since we haven't tagged any of our
312 commits yet, we can't use the specific version functionality; not that we'd want
313 to, though). ``TESTNAME`` is simply the name of the file with the leading
314 ``test-`` stripped and dashes converted to underscores. Run the script with
315 verbose debugging information by using::
317 env WIZARD_DEBUG=1 ./test-install-wordpress.sh
322 A design decision that was made early on during Wizard's development was that
323 the scriptsified versions would contain generic copies of the configuration
324 files. You're going to generate this generic copy now and in doing so,
325 overload your previous scripts commit. Because some installers
326 exhibit different behavior depending on server configuration, you should run
327 the installation on a Scripts server::
329 wizard install wordpress --no-commit
331 The installer will interactively prompt you for some values, and then conclude
332 the install. ``--no-commit`` prevents the installer from generating a Git
333 commit after the install, and will make it easier for us to propagate the
334 change back to the parent repository.
336 Look at the changes the installer made::
340 There are probably now a few unversioned files lounging around; these are probably
341 the configuration files that the installer generated.
343 You will now need to implement the following data attributes and methods in your
344 :class:`~wizard.app.Application` class: :attr:`~wizard.app.Application.extractors`,
345 :attr:`~wizard.app.Application.substitutions`, :attr:`~wizard.app.Application.parametrized_files`,
346 :meth:`~wizard.app.Application.checkConfig` and :meth:`~wizard.app.Application.detectVersion`.
347 These are all closely related to the configuration files that the installer generated.
349 :meth:`~wizard.app.Application.checkConfig` is the most straightforward method to
350 write: you will usually only need to test for the existence of the configuration file.
351 Note that this function will always be called with the current working directory being
352 the deployment, so you can simplify your code accordingly:
354 .. code-block:: python
356 class Application(app.Application):
358 def checkConfig(self, deployment):
359 return os.path.isfile("wp-config.php")
361 :meth:`~wizard.app.Application.detectVersion` should detect the version of the application
362 by regexing it out of a source file. We first have to figure out where the version number
363 is stored: a quick grep tells us that it's in :file:`wp-includes/version.php`:
369 // This just holds the version number, in a separate file so we can bump it without cluttering the SVN
371 $wp_version = '2.0.4';
372 $wp_db_version = 3440;
376 We could now grab the :mod:`re` module and start constructing a regex to grab ``2.0.4``, but it
377 turns out this isn't necessary: :meth:`wizard.app.php.re_var` does this for us already!
379 With this function in hand, writing a version detection function is pretty straightforward.
380 There is one gotcha: the value that ``re_var`` returns as the second subpattern is quoted (the reasons for this
381 will become clear shortly), so you will need to trim off the last and first characters or
382 use :mod:`shlex`. In the case of version numbers, there are probably no escape characters
383 in the string, so the former is relatively safe.
385 .. code-block:: python
387 class Application(app.Application):
389 def detectVersion(self, deployment):
390 contents = open("wp-includes/version.php").read()
391 match = php.re_var("wp_version").search(contents)
392 if not match: return None
393 return distutils.version.LooseVersion(match.group(2)[1:-1])
395 :attr:`~wizard.app.Application.parametrized_files` is a simple list of files that the
396 program's installer wrote or touched during the installation process.
398 .. code-block:: python
400 class Application(app.Application):
402 parametrized_files = ['wp-config.php']
404 This is actually is a lie: we also need to include changes to :file:`php.ini` that
407 .. code-block:: python
409 class Application(app.Application):
411 parametrized_files = ['wp-config.php'] + php.parametrized_files
413 And finally, we have :attr:`~wizard.app.Application.extractors` and
414 :attr:`~wizard.app.Application.substitutions`. At the bare metal, these
415 are simply dictionaries of variable names to functions: when you call the
416 function, it performs either an extraction or a substitution. However, we can
417 use higher-level constructs to generate these functions for us.
419 The magic sauce is a data structure we'll refer to as ``seed``. Its form is a
420 dictionary of variable names to a tuple ``(filename, regular expression)``.
421 The regular expression has a slightly special form (which we mentioned
422 earlier): it contains three (not two or four) subpatterns; the second
423 subpattern matches (quotes and all) the value that the regular expression is
424 actually looking for, and the first and third subpatterns match everything to
425 the left and right, respectively.
429 The flanking subpatterns make it easier to use this regular expression
430 to perform a substitution: we are then allowed to use ``\1FOOBAR\3`` as
433 If we manually coded ``seed`` out, it might look like:
435 .. code-block:: python
438 'WIZARD_DBSERVER': ('wp-config.php', re.compile(r'''^(define\('DB_HOST', )(.*)(\))''', re.M)),
439 'WIZARD_DBNAME': ('wp-config.php', re.compile(r'''^(define\('DB_NAME', )(.*)(\))''', re.M)),
442 There's a lot of duplication, though. For one thing, the regular expressions are almost
443 identical, safe for a single substitution within the string. We have a function
444 :meth:`wizard.app.php.re_define` that does this for us:
446 .. code-block:: python
449 'WIZARD_DBSERVER': ('wp-config.php', php.re_define('DB_HOST')),
450 'WIZARD_DBNAME': ('wp-config.php', php.re_define('DB_NAME')),
455 If you find yourself needing to define a custom regular expression generation function,
456 be sure to use :func:`wizard.app.expand_re`, which will escape an incoming variable
457 to be safe for inclusion in a regular expression, and also let you pass a list,
458 and have correct behavior. Check out :mod:`wizard.app.php` for some examples.
460 Additionally, if you are implementing a function for another language, or a general pattern of
461 variables, consider placing it in an appropriate language module instead.
463 We can shorten this even further: in most cases, all of the configuration values live in
464 one file, so let's make ourselves a function that generates the whole tuple:
466 .. code-block:: python
468 def make_filename_regex_define(var):
469 return 'wp-config.php', php.re_define(var)
471 Then we can use :func:`util.dictmap` to apply this:
473 .. code-block:: python
475 seed = util.dictmap(make_filename_regex_define, {
476 'WIZARD_DBSERVER': 'DB_HOST',
477 'WIZARD_DBNAME': 'DB_NAME',
478 'WIZARD_DBUSER': 'DB_USER',
479 'WIZARD_DBPASSWORD': 'DB_PASSWORD',
482 Short and sweet. From there, setting up :attr:`~wizard.app.Application.extractors` and
483 :attr:`~wizard.app.Application.substitutions` is easy:
485 .. code-block:: python
487 class Application(app.Application):
489 extractors = app.make_extractors(seed)
490 extractors.update(php.extractors)
491 substitutions = app.make_substitutions(seed)
492 substitutions.update(php.substitutions)
494 Note how we combine our own dictionaries with the dictionaries of :mod:`wizard.app.php`, much like
495 we did for :attr:`~wizard.app.Application.parametrized_files`.
497 With all of these pieces in place, run the following command::
499 wizard prepare-config
501 If everything is working, when you open up the configuration files, any user-specific
502 variables should have been replaced by ``WIZARD_FOOBAR`` variables. If not, check
503 your regular expressions, and then try running the command again.
505 When you are satisfied with your changes, add your files, amend your previous
506 commit with these changes and force them back into the public repository::
509 git add wp-config.php
510 git commit --amend -a
516 Congratulations! You have just implemented the installation code for a new install.
517 If you have other copies of the application checked out, you can pull the forced
520 git reset --hard HEAD~
523 One last thing to do: after you are sure that your commit is correct, tag the new
524 commit as ``appname-x.y.z-scripts``, or in this specific case::
526 git tag wordpress-2.0.4-scripts
532 You've only implemented a scriptsified version for only a single version; most applications
533 have multiple versions--you will have to do this process again. Fortunately, the most
534 time consuming parts (implementing logic for :class:`wizard.app.Application`) are already,
535 done so you'll only have to tweak these algorithms if the application changes their
540 Ultimately, we should have another condensed page that describes how to craft
541 an update (with emphasis on what tests to perform to make sure things still
542 work), and pages on how to implement upgrades and backups.