4 :Author: Edward Z. Yang <ezyang@mit.edu>
8 Adding Wizard support for a web application requires some glue code
9 and a specially prepared repository. Creating a new repository for an
10 application in Wizard involves placing :term:`pristine` versions of the source
11 code (from the upstream tarballs) and appropriately patched scripts versions
12 into a Git repository, as well as writing a :mod:`wizard.app` module for the
13 application that implements application specific logic, such as how to install,
14 upgrade or backup the installation.
16 Here is a tutorial for creating such a repository, using an old version of
17 Wordpress as an example. We will implement only the functions necessary for
18 installing an application--upgrades and backups are not described here.
19 We assume that you have a working setup of Wizard; consult the
20 :doc:`setup documentation <setup>` for more details.
22 From this point on, we will assume you are doing development from an AFS directory
23 named ``$WIZARD``; note that application repositories live in ``$WIZARD/srv``.
25 .. supplement:: Conversions
27 One of Wizard's goals is to replace the previous autoinstaller
28 infrastructure. These boxes will explain extra steps that you must perform
29 in order to carry out a conversion of old-style autoinstalls to a Wizard
30 autoinstall. In brief, pre-wizard autoinstalls live in
31 :file:`/mit/scripts/deploy` and consist of a tarball from upstream,
32 possibly a scripts patch, and possibly some post-install munging (such as
33 the creation of a :file:`php.ini` file and appropriate symlinks).
34 Performing a conversion means that we will recreate these changes in our
35 Wizard autoinstall, and you will start you repository with the *earliest*
36 version of the application extant on our servers.
41 This is a tutorial centered around creating a `Wordpress <http://wordpress.org/>`_
42 repository. For the sake of demonstration,
43 we shall assume that this repository hasn't been created yet.
44 The repository then doesn't exist, we should create it::
51 We also have to create a module for the application, so we
52 create :file:`$WIZARD/wizard/app/wordpress.py` and fill it in with a bare bones template:
54 .. code-block:: python
61 from wizard import app, install, resolve, sql, util
62 from wizard.app import php
64 class Application(app.Application):
67 Now we are ready to put some code in our repository: the first thing we will
68 add is the :term:`pristine` branch, which contains verbatim the code from upstream.
70 .. supplement:: Conversions
72 If we were starting a new autoinstaller, we'd pop off and use the latest version,
73 but since we're dealing with legacy we want to start our repository history
74 with the *oldest* version still extant on our servers. To find this out run::
76 wizard summary version APP
78 You'll need to be in the ``scripts-team`` list to have access rights to the
79 folder we store this information in: :file:`/mit/scripts/sec-tools/store/versions`.
81 For the purposes of demonstration, we'll use Wordpress 2.0.2; in reality you
82 should use the latest version. Try running the following command in
83 :file:`$WIZARD/srv/wordpress`::
85 wizard prepare-pristine wordpress-2.0.2
87 You should get an error complaining about :meth:`wizard.app.Application.download`
88 not being implemented yet. Let's fix that:
90 .. code-block:: python
92 class Application(app.Application):
94 def download(self, version):
95 return "http://wordpress.org/wordpress-%s.tar.gz" % version
97 We determined this by finding `Wordpress's Release Archive <http://wordpress.org/download/release-archive/>`_
98 and inferring the naming scheme by inspecting various links. You should now
99 be able to run the prepare-pristine command successfully: when it is
100 done, you'll now have a bunch of files in your repository, and they
101 will be ready to be committed. Inspect the files and commit (note that the
102 format of the commit message is a plain Appname Version.Number)::
105 git commit -asm "Wordpress 2.0.2"
106 git tag wordpress-2.0.2
110 Sometimes, ``http://wordpress.org/wordpress-2.0.2.tar.gz`` won't
111 actually exist anymore (it didn't exist when we did it). In this case,
112 you'll probably be able to find the original tarball in
113 :file:`/mit/scripts/deploy/wordpress-2.0.2`, and you can feed it
114 manually to prepare pristine with
115 ``wizard prepare-pristine /mit/scripts/deploy/wordpress-2.0.2/wordpress-2.0.2.tar.gz``
117 Some last house-keeping bits: now that you have a commit in a repository, you
118 can also create a pristine branch::
125 In a perfect world, the pristine version would be equivalent to the scriptsified
126 version that would actually get deployed. However, we have historically needed
127 to apply patches and add extra configuration files to get applications to
128 work correctly. Due to the way Git's merge algorithm works, the closer we are
129 able to reconstruct a version of the application that was actually used, the
130 better off we will be when we try to subsequently upgrade those applications.
132 First things first: verify that we are on the master branch::
136 .. supplement:: Conversions
138 Check for pre-existing patches in the old application directory,
139 :file:`/mit/scripts/deploy/wordpress-2.0.2` in the case of Wordpress,
142 patch -n0 < /mit/scripts/deploy/wordpress-2.0.2/wordpress.patch
144 Then, run the following command to setup a :file:`.scripts` directory::
148 This directory holds Wizard related files, and is also used by
149 :command:`parallel-find.pl` to determine if a directory is an autoinstall.
151 Finally, if you are running a PHP application, you'll need to setup
152 a :file:`php.ini` and symlinks to it in all subdirectories.
153 As of November 2009, all PHP applications load the same :file:`php.ini` file;
154 so just grab one from another of the PHP projects. We'll rob our own
157 cp /mit/scripts/deploy/php.ini/wordpress php.ini
158 athrun scripts fix-php-ini
160 Now commit, but don't get too attached to your commit; we're going
161 to be heavily modifying it soon::
163 git commit -asm "Wordpress 2.0.2-scripts"
168 We now need to make it possible for a user to install the application.
169 Most web applications have a number of web scripts for generating a
170 configuration file, so creating the install script involves:
172 1. Determining what input values you will need from the user, such
173 as a title for the new application or database credentials; more
176 2. Determining what POST values need to be sent to what URLs.
177 Since you're converting a repository, this job is even simpler: you just
178 need to port the Perl script that was originally used into Python.
180 There's an in-depth explanation of named input values in
181 :mod:`wizard.install`. The short version is that your application
182 contains a class-wide :data:`~wizard.app.Application.install_schema`
183 attribute that encodes this information. Instantiate it with
184 :class:`wizard.install.ArgSchema` (passing in arguments to get
185 some pre-canned input values), and then add application specific
186 arguments by passing instances of :class:`wizard.install.Arg`
187 to the method :meth:`~wizard.install.ArgSchema.add`. Usually you should
188 be able to get away with pre-canned attributes. You can access
189 these arguments inside :meth:`~wizard.app.Application.install` via
190 the ``options`` value.
192 In particular, ``options.dsn`` is a :class:`sqlalchemy.engine.url.URL`
193 which contains member variables such as :meth:`~sqlalchemy.engine.url.URL.username`,
194 :meth:`~sqlalchemy.engine.url.URL.password`, :meth:`~sqlalchemy.engine.url.URL.host` and
195 :meth:`~sqlalchemy.engine.url.URL.database` which you can use to pass in POST.
197 Some tips and tricks for writing :meth:`wizard.app.Application.install`:
199 * Some configuration file generators will get unhappy if the
200 target directory is not chmod'ed to be writable; dropping
201 in a ``os.chmod(dir, 0777)`` and then undoing the chmod
202 when you're done is a decent workaround.
204 * :func:`wizard.install.fetch` is the standard workhorse for making
205 requests to applications. It accepts three parameters; the first
206 is ``options`` (which was the third argument to ``install`` itself),
207 the second is the page to query, relative to the installation's
208 web root, and ``post`` is a dictionary of keys to values to POST.
210 * You should log any web page output using :func:`logging.debug`.
212 * If you need to manually manipulate the database afterwards, you
213 can use :func:`wizard.sql.connect` (passing it ``options.dsn``)
214 to get a `SQLAlchemy metadata object
215 <http://www.sqlalchemy.org/docs/05/sqlexpression.html>`_, which can
216 consequently be queried. For convenience, we've bound metadata
217 to the connection, you can perform implicit execution.
221 Our installer needs to also parametrize :file:`php.ini`, which we haven't
224 To test if your installation function works, it's probably convenient to
225 create a test script in :file:`tests`; :file:`tests/test-install-wordpress.sh`
226 in the case of Wordpress. It will look something like::
231 TESTNAME="install_wordpress"
234 wizard install "wordpress-$VERSION-scripts" "$TESTDIR" --non-interactive -- --title="My Blog"
236 ``DEFAULT_HEAD=1`` indicates that this script can perform a reasonable
237 operation without any version specified (since we haven't tagged any of our
238 commits yet, we can't use the specific version functionality; not that we'd want
239 to, though). ``TESTNAME`` is simply the name of the file with the leading
240 ``test-`` stripped and dashes converted to underscores. Run the script with
241 verbose debugging information by using::
243 env WIZARD_DEBUG=1 ./test-install-wordpress.sh
245 The test scripts will try to conserve databases by running ``wizard remove`` on the
246 old directory, but this requires :meth:`~wizard.app.remove` be implemented.
247 Most of the time (namely, for single database setups), this simple template will suffice:
249 .. code-block:: python
251 class Application(app.Application):
253 def remove(self, deployment)
254 app.remove_database(deployment)
259 A design decision that was made early on during Wizard's development was that
260 the scriptsified versions would contain generic copies of the configuration
261 files. You're going to generate this generic copy now and in doing so,
262 overload your previous scripts commit. Because some installers
263 exhibit different behavior depending on server configuration, you should run
264 the installation on a Scripts server. You can do this manually or use
265 the test script you created::
267 env WIZARD_NO_COMMIT=1 ./test-install-wordpress.sh
269 :envvar:`WIZARD_NO_COMMIT` (command line equivalent to ``--no-commit``)
270 prevents the installer from generating a Git commit after the install, and will
271 make it easier for us to propagate the change back to the parent repository.
273 Change into the generated directory and look at the changes the installer made::
277 There are probably now a few unversioned files lounging around; these are probably
278 the configuration files that the installer generated.
280 You will now need to implement the following data attributes and methods in your
281 :class:`~wizard.app.Application` class: :attr:`~wizard.app.Application.extractors`,
282 :attr:`~wizard.app.Application.substitutions`, :attr:`~wizard.app.Application.parametrized_files`,
283 :meth:`~wizard.app.Application.checkConfig` and :meth:`~wizard.app.Application.detectVersion`.
284 These are all closely related to the configuration files that the installer generated.
286 :meth:`~wizard.app.Application.checkConfig` is the most straightforward method to
287 write: you will usually only need to test for the existence of the configuration file.
288 Note that this function will always be called with the current working directory being
289 the deployment, so you can simplify your code accordingly:
291 .. code-block:: python
293 class Application(app.Application):
295 def checkConfig(self, deployment):
296 return os.path.isfile("wp-config.php")
298 :meth:`~wizard.app.Application.detectVersion` should detect the version of the application
299 by regexing it out of a source file. We first have to figure out where the version number
300 is stored: a quick grep tells us that it's in :file:`wp-includes/version.php`:
306 // This just holds the version number, in a separate file so we can bump it without cluttering the SVN
308 $wp_version = '2.0.4';
309 $wp_db_version = 3440;
313 We could now grab the :mod:`re` module and start constructing a regex to grab ``2.0.4``, but it
314 turns out this isn't necessary: :meth:`wizard.app.php.re_var` does this for us already!
316 With this function in hand, writing a version detection function is pretty straightforward:
317 we have a helper function that takes a file and a regex, and matches out the version number
320 .. code-block:: python
322 class Application(app.Application):
324 def detectVersion(self, deployment):
325 return self.detectVersionFromFile("wp-includes/version.php", php.re_var("wp_version"))
327 :attr:`~wizard.app.Application.parametrized_files` is a simple list of files that the
328 program's installer wrote or touched during the installation process.
330 .. code-block:: python
332 class Application(app.Application):
334 parametrized_files = ['wp-config.php']
336 This is actually is a lie: we also need to include changes to :file:`php.ini` that
339 .. code-block:: python
341 class Application(app.Application):
343 parametrized_files = ['wp-config.php'] + php.parametrized_files
347 And finally, we have :attr:`~wizard.app.Application.extractors` and
348 :attr:`~wizard.app.Application.substitutions`. At the bare metal, these
349 are simply dictionaries of variable names to functions: when you call the
350 function, it performs either an extraction or a substitution. However, we can
351 use higher-level constructs to generate these functions for us.
353 The magic sauce is a data structure we'll refer to as ``seed``. Its form is a
354 dictionary of variable names to a tuple ``(filename, regular expression)``.
355 The regular expression has a slightly special form (which we mentioned
356 earlier): it contains three (not two or four) subpatterns; the second
357 subpattern matches (quotes and all) the value that the regular expression is
358 actually looking for, and the first and third subpatterns match everything to
359 the left and right, respectively.
363 The flanking subpatterns make it easier to use this regular expression
364 to perform a substitution: we are then allowed to use ``\1FOOBAR\3`` as
367 If we manually coded ``seed`` out, it might look like:
369 .. code-block:: python
372 'WIZARD_DBSERVER': ('wp-config.php', re.compile(r'''^(define\('DB_HOST', )(.*)(\))''', re.M)),
373 'WIZARD_DBNAME': ('wp-config.php', re.compile(r'''^(define\('DB_NAME', )(.*)(\))''', re.M)),
376 There's a lot of duplication, though. For one thing, the regular expressions are almost
377 identical, safe for a single substitution within the string. We have a function
378 :meth:`wizard.app.php.re_define` that does this for us:
380 .. code-block:: python
383 'WIZARD_DBSERVER': ('wp-config.php', php.re_define('DB_HOST')),
384 'WIZARD_DBNAME': ('wp-config.php', php.re_define('DB_NAME')),
389 If you find yourself needing to define a custom regular expression generation function,
390 be sure to use :func:`wizard.app.expand_re`, which will escape an incoming variable
391 to be safe for inclusion in a regular expression, and also let you pass a list,
392 and have correct behavior. Check out :mod:`wizard.app.php` for some examples.
394 Additionally, if you are implementing a function for another language, or a general pattern of
395 variables, consider placing it in an appropriate language module instead.
397 We can shorten this even further: in most cases, all of the configuration values live in
398 one file, so let's make ourselves a function that generates the whole tuple:
400 .. code-block:: python
402 def make_filename_regex_define(var):
403 return 'wp-config.php', php.re_define(var)
405 Then we can use :func:`wizard.util.dictmap` to apply this:
407 .. code-block:: python
409 seed = util.dictmap(make_filename_regex_define, {
410 'WIZARD_DBSERVER': 'DB_HOST',
411 'WIZARD_DBNAME': 'DB_NAME',
412 'WIZARD_DBUSER': 'DB_USER',
413 'WIZARD_DBPASSWORD': 'DB_PASSWORD',
416 Short and sweet. From there, setting up :attr:`~wizard.app.Application.extractors` and
417 :attr:`~wizard.app.Application.substitutions` is easy:
419 .. code-block:: python
421 class Application(app.Application):
423 extractors = app.make_extractors(seed)
424 extractors.update(php.extractors)
425 substitutions = app.make_substitutions(seed)
426 substitutions.update(php.substitutions)
428 Note how we combine our own dictionaries with the dictionaries of :mod:`wizard.app.php`, much like
429 we did for :attr:`~wizard.app.Application.parametrized_files`.
431 With all of these pieces in place, run the following command::
433 wizard prepare-config
435 If everything is working, when you open up the configuration files, any user-specific
436 variables should have been replaced by ``WIZARD_FOOBAR`` variables. If not, check
437 your regular expressions, and then try running the command again.
439 When you are satisfied with your changes, add your files, amend your previous
440 commit with these changes and force them back into the public repository::
443 git add wp-config.php
444 git commit --amend -a
450 Congratulations! You have just implemented the installation code for a new install.
451 If you have other copies of the application checked out, you can pull the forced
454 git reset --hard HEAD~
457 One last thing to do: after you are sure that your commit is correct, tag the new
458 commit as ``appname-x.y.z-scripts``, or in this specific case::
460 git tag wordpress-2.0.4-scripts
466 Here is short version for quick reference:
468 #. Create the new repository and new module,
469 #. Implement :meth:`~wizard.app.Application.download`,
470 #. *For Conversions:* Find the oldest extant version with ``wizard summary version $APP``,
471 #. Run ``wizard prepare-pristine $VERSION``,
472 #. Commit with ``-m "$APPLICATION $VERSION"`` and tag ``$APP-$VERSION``,
473 #. Create ``pristine`` branch, but stay on ``master`` branch,
474 #. *For Conversions:* Check for pre-existing patches, and apply them,
475 #. Run ``wizard prepare-new``,
476 #. *For PHP:* Copy in :file:`php.ini` file and run ``athrun scripts fix-php-ini``,
477 #. Commit with ``-m "$APPLICATION $VERSION"``, but *don't* tag,
478 #. Implement :data:`~wizard.app.Application.install_schema` and :meth:`~wizard.app.Application.install`,
479 #. Create :file:`tests/test-install-$APP.sh`,
480 #. On a scripts server, run ``wizard install $APP --no-commit`` and check changes with ``git status``,
481 #. Implement :attr:`~wizard.app.Application.extractors`,
482 :attr:`~wizard.app.Application.substitutions`, :attr:`~wizard.app.Application.parametrized_files`,
483 :meth:`~wizard.app.Application.checkConfig` and :meth:`~wizard.app.Application.detectVersion`,
484 #. Run ``wizard prepare-config``,
485 #. Amend commit and push back, and finally
486 #. Tag ``$APP-$VERSION-scripts``
491 You've only implemented a scriptsified version for only a single version; most applications
492 have multiple versions--you will have to do this process again. Fortunately, the most
493 time consuming parts (implementing logic for :class:`wizard.app.Application`) are already,
494 done so the process of :doc:`creating upgrades <upgrade>` is much simpler.
496 There is still functionality yet undone: namely the methods for actually performing an
497 upgrade are not yet implemented. You can find instructions for this on the
498 :doc:`creating upgrades <upgrade>` page under "Implementation".