4 :Author: Edward Z. Yang <ezyang@mit.edu>
8 Adding Wizard support for a web application requires some glue code
9 and a specially prepared repository. Creating a new repository for an
10 application in Wizard involves placing :term:`pristine` versions of the source
11 code (from the upstream tarballs) and appropriately patched scripts versions
12 into a Git repository, as well as writing a :mod:`wizard.app` module for the
13 application that implements application specific logic, such as how to install,
14 upgrade or backup the installation.
16 Here is a tutorial for creating such a repository, using an old version of
17 Wordpress as an example. We will implement only the functions necessary for
18 installing an application--upgrades and backups are not described here.
20 .. supplement:: Conversions
22 One of Wizard's goals is to replace the previous autoinstaller
23 infrastructure. These boxes will explain extra steps that you must perform
24 in order to carry out a conversion of old-style autoinstalls to a Wizard
25 autoinstall. In brief, pre-wizard autoinstalls live in
26 :file:`/mit/scripts/deploy` and consist of a tarball from upstream,
27 possibly a scripts patch, and possibly some post-install munging (such as
28 the creation of a :file:`php.ini` file and appropriate symlinks).
29 Performing a conversion means that we will recreate these changes in our
30 Wizard autoinstall, and you will start you repository with the *earliest*
31 version of the application extant on our servers.
36 Probably the easiest way to do development is entirely on :term:`AFS`, so that if you
37 SSH into a scripts server to perform testing, you will be able
38 to invoke your tools and read your development repository. In order
39 to be able to run the test scripts in the tests directory, this
40 is preferably in :file:`web_scripts`. In that
41 case, setup is as simple as::
43 git clone /mit/scripts/git/wizard.git /mit/$USER/web_scripts/wizard
44 # for any application you're going to do development on, also:
45 git clone /mit/scripts/git/autoinstalls/$APP.git /mit/$USER/web_scripts/wizard/srv/$APP
47 From this point on, we will assume you are doing development from an AFS directory
48 named ``$WIZARD``; note that application repositories live in ``$WIZARD/srv``.
52 Other users will not be able to access your source code. If you
53 change the access control to allow ``system:anyuser`` to read your
54 source code (in case you want to let other people test your
55 code), do *not* give access to the :file:`tests` directory,
56 which may contain sensitive data.
61 These instructions are for if you'd like to be able to develop offline and are
62 on ``scripts-root``. Wizard will mostly work on your local machine, but you
63 won't be able to do all development offline; some steps in the development
64 process must be performed on scripts servers. Thus, the difficult part is
65 marshalling commits from one repository to another. Git doesn't exactly make
66 this easy, but you can follow this recipe::
68 git clone ssh://scripts@scripts.mit.edu/mit/scripts/git/wizard.git ~/wizard
74 git remote add afs /mit/$USER/wizard.git
75 git push -f afs master
76 git clone /mit/$USER/wizard.git /mit/$USER/wizard
78 We create a bare repository :file:`/mit/$USER/wizard.git` that you can push and
79 pull from, and then setup an alternate remote ``afs`` on your offline copy.
81 And then you can perform updates from your local copy with::
87 If :file:`/mit/$USER/wizard.git` has write permissions for
88 ``system:scripts-security-upd``, this is especially useful if you were hacking
89 on a copy living on ``not-backward.mit.edu``, and now need to transfer the
90 changes back to the canonical repository (please don't give ``not-backward.mit.edu``
91 your root tickets!) You can also setup a wizard directory similar to the
92 first set of directions for on-server testing.
96 These instructions are not well tested. Let me know if you run into
102 This is a tutorial centered around creating a `Wordpress <http://wordpress.org/>`_
103 repository. For the sake of demonstration,
104 we shall assume that this repository hasn't been created yet.
105 The repository then doesn't exist, we should create it::
112 We also have to create a module for the application, so we
113 create :file:`$WIZARD/wizard/app/wordpress.py` and fill it in with a bare bones template:
115 .. code-block:: python
122 from wizard import app, install, resolve, sql, util
123 from wizard.app import php
125 class Application(app.Application):
128 Now we are ready to put some code in our repository: the first thing we will
129 add is the :term:`pristine` branch, which contains verbatim the code from upstream.
131 .. supplement:: Conversions
133 If we were starting a new autoinstaller, we'd pop off and use the latest version,
134 but since we're dealing with legacy we want to start our repository history
135 with the *oldest* version still extant on our servers. To find this out run::
137 wizard summary version APP
139 You'll need to be in the ``scripts-team`` list to have access rights to the
140 folder we store this information in: :file:`/mit/scripts/sec-tools/store/versions`.
142 For the purposes of demonstration, we'll use Wordpress 2.0.2; in reality you
143 should use the latest version. Try running the following command in
144 :file:`$WIZARD/srv/wordpress`::
146 wizard prepare-pristine wordpress-2.0.2
148 You should get an error complaining about :meth:`wizard.app.Application.download`
149 not being implemented yet. Let's fix that:
151 .. code-block:: python
153 class Application(app.Application):
155 def download(self, version):
156 return "http://wordpress.org/wordpress-%s.tar.gz" % version
158 We determined this by finding `Wordpress's Release Archive <http://wordpress.org/download/release-archive/>`_
159 and inferring the naming scheme by inspecting various links. You should now
160 be able to run the prepare-pristine command successfully: when it is
161 done, you'll now have a bunch of files in your repository, and they
162 will be ready to be committed. Inspect the files and commit (note that the
163 format of the commit message is a plain Appname Version.Number)::
166 git commit -asm "Wordpress 2.0.2"
167 git tag wordpress-2.0.2
171 Sometimes, ``http://wordpress.org/wordpress-2.0.2.tar.gz`` won't
172 actually exist anymore (it didn't exist when we did it). In this case,
173 you'll probably be able to find the original tarball in
174 :file:`/mit/scripts/deploy/wordpress-2.0.2`, and you can feed it
175 manually to prepare pristine with
176 ``wizard prepare-pristine /mit/scripts/deploy/wordpress-2.0.2/wordpress-2.0.2.tar.gz``
178 Some last house-keeping bits: now that you have a commit in a repository, you
179 can also create a pristine branch::
185 The following text and graph should be put in a more general overview of
186 the topology of Wizard git repositories.
188 From the point of view of the pristine branch pointer, history should be a
189 straight-forward progression of versions. Once you have more versions,
190 it will look something like:
192 .. digraph:: pristine_dag
196 subgraph cluster_pristine {
209 In a perfect world, the pristine version would be equivalent to the scriptsified
210 version that would actually get deployed. However, we have historically needed
211 to apply patches and add extra configuration files to get applications to
212 work correctly. Due to the way Git's merge algorithm works, the closer we are
213 able to reconstruct a version of the application that was actually used, the
214 better off we will be when we try to subsequently upgrade those applications.
218 The following text and graph should be put in a more general overview of
219 the topology of Wizard git repositories.
221 To give you an idea of what the Git history graph will look like when we
222 are done, here is the graph from before, but augmented with the scripts versions:
224 .. digraph:: master_dag
228 subgraph cluster_pristine {
237 subgraph cluster_master {
239 as [label="1.0-scripts"]
240 bs [label="1.1-scripts"]
241 cs [label="2.0-scripts"]
251 First things first: verify that we are on the master branch::
255 .. supplement:: Conversions
257 Check for pre-existing patches in the old application directory,
258 :file:`/mit/scripts/deploy/wordpress-2.0.2` in the case of Wordpress,
261 patch -n0 < /mit/scripts/deploy/wordpress-2.0.2/wordpress.patch
263 Then, run the following command to setup a :file:`.scripts` directory::
267 This directory holds Wizard related files, and is also used by
268 :command:`parallel-find.pl` to determine if a directory is an autoinstall.
270 Finally, if you are running a PHP application, you'll need to setup
271 a :file:`php.ini` and symlinks to it in all subdirectories.
272 As of November 2009, all PHP applications load the same :file:`php.ini` file;
273 so just grab one from another of the PHP projects. We'll rob our own
276 cp /mit/scripts/deploy/php.ini/wordpress php.ini
277 athrun scripts fix-php-ini
279 Now commit, but don't get too attached to your commit; we're going
280 to be heavily modifying it soon::
282 git commit -asm "Wordpress 2.0.2-scripts"
287 We now need to make it possible for a user to install the application.
288 Most web applications have a number of web scripts for generating a
289 configuration file, so creating the install script involves:
291 1. Determining what input values you will need from the user, such
292 as a title for the new application or database credentials; more
295 2. Determining what POST values need to be sent to what URLs.
296 Since you're converting a repository, this job is even simpler: you just
297 need to port the Perl script that was originally used into Python.
299 There's an in-depth explanation of named input values in
300 :mod:`wizard.install`. The short version is that your application
301 contains a class-wide :data:`~wizard.app.Application.install_schema`
302 attribute that encodes this information. Instantiate it with
303 :class:`wizard.install.ArgSchema` (passing in arguments to get
304 some pre-canned input values), and then add application specific
305 arguments by passing instances of :class:`wizard.install.Arg`
306 to the method :meth:`~wizard.install.ArgSchema.add`. Usually you should
307 be able to get away with pre-canned attributes. You can access
308 these arguments inside :meth:`~wizard.app.Application.install` via
309 the ``options`` value.
311 Some tips and tricks for writing :meth:`wizard.app.Application.install`:
313 * Some configuration file generators will get unhappy if the
314 target directory is not chmod'ed to be writable; dropping
315 in a ``os.chmod(dir, 0777)`` and then undoing the chmod
316 when you're done is a decent workaround.
318 * :func:`wizard.install.fetch` is the standard workhorse for making
319 requests to applications. It accepts three parameters; the first
320 is ``options`` (which was the third argument to ``install`` itself),
321 the second is the page to query, relative to the installation's
322 web root, and ``post`` is a dictionary of keys to values to POST.
324 * You should log any web page output using :func:`logging.debug`.
326 * If you need to manually manipulate the database afterwards, you
327 can use :func:`wizard.sql.mysql_connect` (passing it ``options``)
328 to get a `SQLAlchemy metadata object
329 <http://www.sqlalchemy.org/docs/05/sqlexpression.html>`_, which can
330 consequently be queried. For convenience, we've bound metadata
331 to the connection, you can perform implicit execution.
335 Our installer needs to also parametrize :file:`php.ini`, which we haven't
338 To test if your installation function works, it's probably convenient to
339 create a test script in :file:`tests`; :file:`tests/test-install-wordpress.sh`
340 in the case of Wordpress. It will look something like::
345 TESTNAME="install_wordpress"
348 wizard install "wordpress-$VERSION-scripts" "$TESTDIR" --non-interactive -- --title="My Blog"
350 ``DEFAULT_HEAD=1`` indicates that this script can perform a reasonable
351 operation without any version specified (since we haven't tagged any of our
352 commits yet, we can't use the specific version functionality; not that we'd want
353 to, though). ``TESTNAME`` is simply the name of the file with the leading
354 ``test-`` stripped and dashes converted to underscores. Run the script with
355 verbose debugging information by using::
357 env WIZARD_DEBUG=1 ./test-install-wordpress.sh
362 A design decision that was made early on during Wizard's development was that
363 the scriptsified versions would contain generic copies of the configuration
364 files. You're going to generate this generic copy now and in doing so,
365 overload your previous scripts commit. Because some installers
366 exhibit different behavior depending on server configuration, you should run
367 the installation on a Scripts server. You can do this manually or use
368 the test script you created::
370 env WIZARD_NO_COMMIT=1 ./test-install-wordpress.sh
372 :envvar:`WIZARD_NO_COMMIT` (command line equivalent to ``--no-commit``)
373 prevents the installer from generating a Git commit after the install, and will
374 make it easier for us to propagate the change back to the parent repository.
376 Change into the generated directory and look at the changes the installer made::
380 There are probably now a few unversioned files lounging around; these are probably
381 the configuration files that the installer generated.
383 You will now need to implement the following data attributes and methods in your
384 :class:`~wizard.app.Application` class: :attr:`~wizard.app.Application.extractors`,
385 :attr:`~wizard.app.Application.substitutions`, :attr:`~wizard.app.Application.parametrized_files`,
386 :meth:`~wizard.app.Application.checkConfig` and :meth:`~wizard.app.Application.detectVersion`.
387 These are all closely related to the configuration files that the installer generated.
389 :meth:`~wizard.app.Application.checkConfig` is the most straightforward method to
390 write: you will usually only need to test for the existence of the configuration file.
391 Note that this function will always be called with the current working directory being
392 the deployment, so you can simplify your code accordingly:
394 .. code-block:: python
396 class Application(app.Application):
398 def checkConfig(self, deployment):
399 return os.path.isfile("wp-config.php")
401 :meth:`~wizard.app.Application.detectVersion` should detect the version of the application
402 by regexing it out of a source file. We first have to figure out where the version number
403 is stored: a quick grep tells us that it's in :file:`wp-includes/version.php`:
409 // This just holds the version number, in a separate file so we can bump it without cluttering the SVN
411 $wp_version = '2.0.4';
412 $wp_db_version = 3440;
416 We could now grab the :mod:`re` module and start constructing a regex to grab ``2.0.4``, but it
417 turns out this isn't necessary: :meth:`wizard.app.php.re_var` does this for us already!
419 With this function in hand, writing a version detection function is pretty straightforward.
420 There is one gotcha: the value that ``re_var`` returns as the second subpattern is quoted (the reasons for this
421 will become clear shortly), so you will need to trim off the last and first characters or
422 use :mod:`shlex`. In the case of version numbers, there are probably no escape characters
423 in the string, so the former is relatively safe.
425 .. code-block:: python
427 class Application(app.Application):
429 def detectVersion(self, deployment):
430 contents = open("wp-includes/version.php").read()
431 match = php.re_var("wp_version").search(contents)
432 if not match: return None
433 return distutils.version.LooseVersion(match.group(2)[1:-1])
435 :attr:`~wizard.app.Application.parametrized_files` is a simple list of files that the
436 program's installer wrote or touched during the installation process.
438 .. code-block:: python
440 class Application(app.Application):
442 parametrized_files = ['wp-config.php']
444 This is actually is a lie: we also need to include changes to :file:`php.ini` that
447 .. code-block:: python
449 class Application(app.Application):
451 parametrized_files = ['wp-config.php'] + php.parametrized_files
453 And finally, we have :attr:`~wizard.app.Application.extractors` and
454 :attr:`~wizard.app.Application.substitutions`. At the bare metal, these
455 are simply dictionaries of variable names to functions: when you call the
456 function, it performs either an extraction or a substitution. However, we can
457 use higher-level constructs to generate these functions for us.
459 The magic sauce is a data structure we'll refer to as ``seed``. Its form is a
460 dictionary of variable names to a tuple ``(filename, regular expression)``.
461 The regular expression has a slightly special form (which we mentioned
462 earlier): it contains three (not two or four) subpatterns; the second
463 subpattern matches (quotes and all) the value that the regular expression is
464 actually looking for, and the first and third subpatterns match everything to
465 the left and right, respectively.
469 The flanking subpatterns make it easier to use this regular expression
470 to perform a substitution: we are then allowed to use ``\1FOOBAR\3`` as
473 If we manually coded ``seed`` out, it might look like:
475 .. code-block:: python
478 'WIZARD_DBSERVER': ('wp-config.php', re.compile(r'''^(define\('DB_HOST', )(.*)(\))''', re.M)),
479 'WIZARD_DBNAME': ('wp-config.php', re.compile(r'''^(define\('DB_NAME', )(.*)(\))''', re.M)),
482 There's a lot of duplication, though. For one thing, the regular expressions are almost
483 identical, safe for a single substitution within the string. We have a function
484 :meth:`wizard.app.php.re_define` that does this for us:
486 .. code-block:: python
489 'WIZARD_DBSERVER': ('wp-config.php', php.re_define('DB_HOST')),
490 'WIZARD_DBNAME': ('wp-config.php', php.re_define('DB_NAME')),
495 If you find yourself needing to define a custom regular expression generation function,
496 be sure to use :func:`wizard.app.expand_re`, which will escape an incoming variable
497 to be safe for inclusion in a regular expression, and also let you pass a list,
498 and have correct behavior. Check out :mod:`wizard.app.php` for some examples.
500 Additionally, if you are implementing a function for another language, or a general pattern of
501 variables, consider placing it in an appropriate language module instead.
503 We can shorten this even further: in most cases, all of the configuration values live in
504 one file, so let's make ourselves a function that generates the whole tuple:
506 .. code-block:: python
508 def make_filename_regex_define(var):
509 return 'wp-config.php', php.re_define(var)
511 Then we can use :func:`wizard.util.dictmap` to apply this:
513 .. code-block:: python
515 seed = util.dictmap(make_filename_regex_define, {
516 'WIZARD_DBSERVER': 'DB_HOST',
517 'WIZARD_DBNAME': 'DB_NAME',
518 'WIZARD_DBUSER': 'DB_USER',
519 'WIZARD_DBPASSWORD': 'DB_PASSWORD',
522 Short and sweet. From there, setting up :attr:`~wizard.app.Application.extractors` and
523 :attr:`~wizard.app.Application.substitutions` is easy:
525 .. code-block:: python
527 class Application(app.Application):
529 extractors = app.make_extractors(seed)
530 extractors.update(php.extractors)
531 substitutions = app.make_substitutions(seed)
532 substitutions.update(php.substitutions)
534 Note how we combine our own dictionaries with the dictionaries of :mod:`wizard.app.php`, much like
535 we did for :attr:`~wizard.app.Application.parametrized_files`.
537 With all of these pieces in place, run the following command::
539 wizard prepare-config
541 If everything is working, when you open up the configuration files, any user-specific
542 variables should have been replaced by ``WIZARD_FOOBAR`` variables. If not, check
543 your regular expressions, and then try running the command again.
545 When you are satisfied with your changes, add your files, amend your previous
546 commit with these changes and force them back into the public repository::
549 git add wp-config.php
550 git commit --amend -a
556 Congratulations! You have just implemented the installation code for a new install.
557 If you have other copies of the application checked out, you can pull the forced
560 git reset --hard HEAD~
563 One last thing to do: after you are sure that your commit is correct, tag the new
564 commit as ``appname-x.y.z-scripts``, or in this specific case::
566 git tag wordpress-2.0.4-scripts
572 Here is short version for quick reference:
574 #. Create the new repository and new module,
575 #. Implement :meth:`~wizard.app.Application.download`,
576 #. *For Conversions:* Find the oldest extant version with ``wizard summary version $APP``,
577 #. Run ``wizard prepare-pristine $VERSION``,
578 #. Commit with ``-m "$APPLICATION $VERSION"`` and tag ``$APP-$VERSION``,
579 #. Create ``pristine`` branch, but stay on ``master`` branch,
580 #. *For Conversions:* Check for pre-existing patches, and apply them,
581 #. Run ``wizard prepare-new``,
582 #. *For PHP:* Copy in :file:`php.ini` file and run ``athrun scripts fix-php-ini``,
583 #. Commit with ``-m "$APPLICATION $VERSION"``, but *don't* tag,
584 #. Implement :data:`~wizard.app.Application.install_schema` and :meth:`~wizard.app.Application.install`,
585 #. Create :file:`tests/test-install-$APP.sh`,
586 #. On a scripts server, run ``wizard install $APP --no-commit`` and check changes with ``git status``,
587 #. Implement :attr:`~wizard.app.Application.extractors`,
588 :attr:`~wizard.app.Application.substitutions`, :attr:`~wizard.app.Application.parametrized_files`,
589 :meth:`~wizard.app.Application.checkConfig` and :meth:`~wizard.app.Application.detectVersion`,
590 #. Run ``wizard prepare-config``,
591 #. Amend commit and push back, and finally
592 #. Tag ``$APP-$VERSION-scripts``
597 You've only implemented a scriptsified version for only a single version; most applications
598 have multiple versions--you will have to do this process again. Fortunately, the most
599 time consuming parts (implementing logic for :class:`wizard.app.Application`) are already,
600 done so you'll only have to tweak these algorithms if the application changes their
605 Ultimately, we should have another condensed page that describes how to craft
606 an update (with emphasis on what tests to perform to make sure things still
607 work), and pages on how to implement upgrades and backups.