4 :Author: Edward Z. Yang <ezyang@mit.edu>
8 Adding Wizard support for a web application requires some glue code
9 and a specially prepared repository. Creating a new repository for an
10 application in Wizard involves placing :term:`pristine` versions of the source
11 code (from the upstream tarballs) and appropriately patched scripts versions
12 into a Git repository, as well as writing a :mod:`wizard.app` module for the
13 application that implements application specific logic, such as how to install,
14 upgrade or backup the installation.
16 Here is a tutorial for creating such a repository, using an old version of
17 Wordpress as an example. We will implement only the functions necessary for
18 installing an application--upgrades and backups are not described here.
19 We assume that you have a working setup of Wizard; consult the
20 :doc:`setup documentation <setup>` for more details.
22 From this point on, we will assume you are doing development from an AFS directory
23 named ``$WIZARD``; note that application repositories live in ``$WIZARD/srv``.
25 .. supplement:: Conversions
27 One of Wizard's goals is to replace the previous autoinstaller
28 infrastructure. These boxes will explain extra steps that you must perform
29 in order to carry out a conversion of old-style autoinstalls to a Wizard
30 autoinstall. In brief, pre-wizard autoinstalls live in
31 :file:`/mit/scripts/deploy` and consist of a tarball from upstream,
32 possibly a scripts patch, and possibly some post-install munging (such as
33 the creation of a :file:`php.ini` file and appropriate symlinks).
34 Performing a conversion means that we will recreate these changes in our
35 Wizard autoinstall, and you will start you repository with the *earliest*
36 version of the application extant on our servers.
41 This is a tutorial centered around creating a `Wordpress <http://wordpress.org/>`_
42 repository. It assumes you have an upstream; if you do not,
43 you can skip most of these steps: just ensure you have a Git repository
44 which contains a :term:`pristine` and :term:`master` branch,
45 as well as tags for all of the releases in the form ``appname-1.2.3``
46 and ``appname-1.2.3-scripts``.
48 For the sake of demonstration, we shall assume that this repository
49 hasn't been created yet. The repository then doesn't exist, we should
57 We also have to create a module for the application, so we
58 create :file:`$WIZARD/wizard/app/wordpress.py` and fill it in with a bare bones template:
60 .. code-block:: python
67 from wizard import app, install, resolve, sql, util
68 from wizard.app import php
70 class Application(app.Application):
73 Finally, we have to tell Wizard about this new module. If you are
74 creating this new module for Scripts, the easiest way to tell Wizard
75 about the application is to add it to the :mod:`wizard_scripts`
76 `setuptools plugin <http://aroberge.blogspot.com/2008/12/plugins-part-6-setuptools-based.html>`_.
77 Even if you don't know anything about setuptools, it's pretty easy
78 to add your application: edit the file :file:`plugins/scripts/setup.py`
79 and add your application to the ``wizard.app`` entry point by looking
80 for the following chunk of code and adding a new line::
82 'wizard.app': ['mediawiki = wizard.app.mediawiki:Application',
83 'phpBB = wizard.app.phpBB:Application',
84 'wordpress = wizard.app.wordpress:Application', # NEW LINE!
87 This tells Wizard that there is a new application named ``wordpress``,
88 with a module named ``wizard.app.wordpress`` and a class named
89 ``Application`` in that module, which Wizard should use.
91 You need to refresh plugin information by running the :file:`refresh.sh`
92 script or by running :file:`python setup.py egg_info` in the
93 :file:`plugins/scripts` directory.
97 If you do not want to place your application in the Scripts plugin,
98 you will need to create a :file:`setup.py` file from scratch in your
99 own plugin. A reasonable template file is::
104 name = 'wizard-myapp',
107 author_email = 'my-email@mit.edu',
108 description = ('My Awesome Application'),
109 license = 'My Awesome License',
110 url = 'http://www.example.com/',
111 packages = setuptools.find_packages(),
113 'wizard.app': ['wordpress = wizard.app.wordpress:Application',
118 Don't forget to run :file:`python setup.py egg_info` and add your module
119 to your :envvar:`PYTHON_PATH` (otherwise, Wizard won't know that
122 Now we are ready to put some code in our repository: the first thing we will
123 add is the :term:`pristine` branch, which contains verbatim the code from upstream.
125 .. supplement:: Conversions
127 If we were starting a new autoinstaller, we'd pop off and use the latest version,
128 but since we're dealing with legacy we want to start our repository history
129 with the *oldest* version still extant on our servers. To find this out run::
131 wizard summary version APP
133 You'll need to be in the ``scripts-team`` list to have access rights to the
134 folder we store this information in: :file:`/mit/scripts/sec-tools/store/versions`.
136 For the purposes of demonstration, we'll use Wordpress 2.0.2; in reality you
137 should use the latest version. Try running the following commands::
139 cd "$WIZARD/srv/wordpress"
140 wizard prepare-pristine wordpress-2.0.2
142 You should get an error complaining about :meth:`wizard.app.Application.download`
143 not being implemented yet. Let's fix that:
145 .. code-block:: python
147 class Application(app.Application):
149 def download(self, version):
150 return "http://wordpress.org/wordpress-%s.tar.gz" % version
152 We determined this by finding `Wordpress's Release Archive <http://wordpress.org/download/release-archive/>`_
153 and inferring the naming scheme by inspecting various links. You should now
154 be able to run the prepare-pristine command successfully: when it is
155 done, you'll now have a bunch of files in your repository, and they
156 will be ready to be committed. Inspect the files and commit (note that the
157 format of the commit message is a plain Appname Version.Number)::
160 git commit -asm "Wordpress 2.0.2"
161 git tag wordpress-2.0.2
165 Sometimes, ``http://wordpress.org/wordpress-2.0.2.tar.gz`` won't
166 actually exist anymore (it didn't exist when we did it). In this case,
167 you'll probably be able to find the original tarball in
168 :file:`/mit/scripts/deploy/wordpress-2.0.2`, and you can feed it
169 manually to prepare pristine with
170 ``wizard prepare-pristine /mit/scripts/deploy/wordpress-2.0.2/wordpress-2.0.2.tar.gz``
172 Some last house-keeping bits: now that you have a commit in a repository, you
173 can also create a pristine branch::
180 In a perfect world, the pristine version would be equivalent to the scriptsified
181 version that would actually get deployed. However, we have historically needed
182 to apply patches and add extra configuration files to get applications to
183 work correctly. Due to the way Git's merge algorithm works, the closer we are
184 able to reconstruct a version of the application that was actually used, the
185 better off we will be when we try to subsequently upgrade those applications.
187 First things first: verify that we are on the master branch::
191 .. supplement:: Conversions
193 Check for pre-existing patches in the old application directory,
194 :file:`/mit/scripts/deploy/wordpress-2.0.2` in the case of Wordpress,
197 patch -n0 < /mit/scripts/deploy/wordpress-2.0.2/wordpress.patch
199 If you are running a PHP application, you'll need to setup
200 a :file:`php.ini` and symlinks to it in all subdirectories.
201 As of November 2009, all PHP applications load the same :file:`php.ini` file;
202 so just grab one from another of the PHP projects. We'll rob our own
205 cp /mit/scripts/deploy/php.ini/wordpress php.ini
206 athrun scripts fix-php-ini
209 Now commit, but don't get too attached to your commit; we're going
210 to be heavily modifying it soon::
212 git commit -asm "Wordpress 2.0.2-scripts"
217 We now need to make it possible for a user to install the application.
218 The :meth:`~wizard.install.Application.install` method should take the
219 application from a just cloned working copy into a fully functioning web
220 application with configuration and a working database, etc. Most web
221 applications have a number of web scripts for generating a configuration
222 file, so creating the install script tend to involve:
225 1. Deleting any placeholder files that were in the repository (there
226 aren't any now, but there will be soon.)
228 2. Determining what input values you will need from the user, such
229 as a title for the new application or database credentials; more
232 3. Determining what POST values need to be sent to what URLs or to
233 what shell scripts (these are the install scripts the application
234 may have supplied to you.)
236 .. supplement:: Conversions
238 Since you're converting a repository, this job is even simpler: you
239 just need to port the Perl script that was originally used into
242 There's an in-depth explanation of named input values in
243 :mod:`wizard.install`. The short version is that your application
244 contains a class-wide :data:`~wizard.app.Application.install_schema`
245 attribute that encodes this information. Instantiate it with
246 :class:`wizard.install.ArgSchema` (passing in arguments to get
247 some pre-canned input values), and then add application specific
248 arguments by passing instances of :class:`wizard.install.Arg`
249 to the method :meth:`~wizard.install.ArgSchema.add`. Usually you should
250 be able to get away with pre-canned attributes. You can access
251 these arguments inside :meth:`~wizard.app.Application.install` via
252 the ``options`` value.
254 In particular, ``options.dsn`` is a :class:`sqlalchemy.engine.url.URL`
255 which contains member variables such as :meth:`~sqlalchemy.engine.url.URL.username`,
256 :meth:`~sqlalchemy.engine.url.URL.password`, :meth:`~sqlalchemy.engine.url.URL.host` and
257 :meth:`~sqlalchemy.engine.url.URL.database` which you can use to pass in POST.
259 Some tips and tricks for writing :meth:`wizard.app.Application.install`:
261 * Some configuration file generators will get unhappy if the
262 target directory is not chmod'ed to be writable; dropping
263 in a ``os.chmod(dir, 0777)`` and then undoing the chmod
264 when you're done is a decent workaround.
266 * :func:`wizard.install.fetch` is the standard workhorse for making
267 requests to applications. It accepts three parameters; the first
268 is ``options`` (which was the third argument to ``install`` itself),
269 the second is the page to query, relative to the installation's
270 web root, and ``post`` is a dictionary of keys to values to POST.
272 * You should log any web page output using :func:`logging.debug`.
274 * If you need to manually manipulate the database afterwards, you
275 can use :func:`wizard.sql.connect` (passing it ``options.dsn``)
276 to get a `SQLAlchemy metadata object
277 <http://www.sqlalchemy.org/docs/05/sqlexpression.html>`_, which can
278 consequently be queried. For convenience, we've bound metadata
279 to the connection, you can perform implicit execution.
281 To test if your installation function works, it's probably convenient to
282 create a test script in :file:`tests`; :file:`tests/wordpress-install-test.sh`
283 in the case of Wordpress. It will look something like::
289 TESTNAME="wordpress_install"
292 wizard install "wordpress-$VERSION-scripts" "$TESTDIR" --non-interactive -- --title="My Blog"
296 As you develop more test-scripts, you may find that you are
297 frequently copy pasting install commands. In this case, it may be
298 useful to create a 'wordpress-install' helper shell fragment and
299 source it whenever you need a vanilla installation.
301 ``DEFAULT_HEAD=1`` indicates that this script can perform a reasonable
302 operation without any version specified (since we haven't tagged any of our
303 commits yet, we can't use the specific version functionality; not that we'd want
304 to, though). ``TESTNAME`` is simply the name of the file with the trailing
305 ``-test`` stripped and dashes converted to underscores. Run the script with
306 verbose debugging information by using::
308 env WIZARD_DEBUG=1 ./wordpress-install-test.sh
310 The test scripts will try to conserve databases by running ``wizard remove`` on the
311 old directory, but this requires :meth:`~wizard.app.remove` be implemented.
312 Most of the time (namely, for single database setups), this simple template will suffice:
314 .. code-block:: python
316 class Application(app.Application):
318 def remove(self, deployment)
319 app.remove_database(deployment)
324 A design decision that was made early on during Wizard's development was that
325 the scriptsified versions would contain generic copies of the configuration
326 files. You're going to generate this generic copy now and in doing so,
327 overload your previous scripts commit. Because some installers
328 exhibit different behavior depending on server configuration, you should run
329 the installation on a Scripts server. You can do this manually or use
330 the test script you created::
332 env WIZARD_NO_COMMIT=1 ./wordpress-install-test.sh
334 :envvar:`WIZARD_NO_COMMIT` (command line equivalent to ``--no-commit``)
335 prevents the installer from generating a Git commit after the install, and will
336 make it easier for us to propagate the change back to the parent repository.
338 Change into the generated directory and look at the changes the installer made::
342 There are probably now a few unversioned files lounging around; these are probably
343 the configuration files that the installer generated.
345 You will now need to implement the following data attributes and methods in your
346 :class:`~wizard.app.Application` class: :attr:`~wizard.app.Application.extractors`,
347 :attr:`~wizard.app.Application.substitutions`, :attr:`~wizard.app.Application.parametrized_files`,
348 :meth:`~wizard.app.Application.checkConfig` and :meth:`~wizard.app.Application.detectVersion`.
349 These are all closely related to the configuration files that the installer generated.
351 :meth:`~wizard.app.Application.checkConfig` is the most straightforward method to
352 write: you will usually only need to test for the existence of the configuration file.
353 Note that this function will always be called with the current working directory being
354 the deployment, so you can simplify your code accordingly:
356 .. code-block:: python
358 class Application(app.Application):
360 def checkConfig(self, deployment):
361 return os.path.isfile("wp-config.php")
363 :meth:`~wizard.app.Application.detectVersion` should detect the version of the application
364 by regexing it out of a source file. We first have to figure out where the version number
365 is stored: a quick grep tells us that it's in :file:`wp-includes/version.php`:
371 // This just holds the version number, in a separate file so we can bump it without cluttering the SVN
373 $wp_version = '2.0.4';
374 $wp_db_version = 3440;
378 We could now grab the :mod:`re` module and start constructing a regex to grab ``2.0.4``, but it
379 turns out this isn't necessary: :meth:`wizard.app.php.re_var` does this for us already!
381 With this function in hand, writing a version detection function is pretty straightforward:
382 we have a helper function that takes a file and a regex, and matches out the version number
385 .. code-block:: python
387 class Application(app.Application):
389 def detectVersion(self, deployment):
390 return self.detectVersionFromFile("wp-includes/version.php", php.re_var("wp_version"))
392 :attr:`~wizard.app.Application.parametrized_files` is a simple list of files that the
393 program's installer wrote or touched during the installation process.
395 .. code-block:: python
397 class Application(app.Application):
399 parametrized_files = ['wp-config.php']
401 This is actually is a lie: we also need to include changes to :file:`php.ini` that
404 .. code-block:: python
406 class Application(app.Application):
408 parametrized_files = ['wp-config.php'] + php.parametrized_files
412 And finally, we have :attr:`~wizard.app.Application.extractors` and
413 :attr:`~wizard.app.Application.substitutions`. At the bare metal, these
414 are simply dictionaries of variable names to functions: when you call the
415 function, it performs either an extraction or a substitution. However, we can
416 use higher-level constructs to generate these functions for us.
418 The magic sauce is a data structure we'll refer to as ``seed``. Its form is a
419 dictionary of variable names to a tuple ``(filename, regular expression)``.
420 The regular expression has a slightly special form (which we mentioned
421 earlier): it contains three (not two or four) subpatterns; the second
422 subpattern matches (quotes and all) the value that the regular expression is
423 actually looking for, and the first and third subpatterns match everything to
424 the left and right, respectively.
428 The flanking subpatterns make it easier to use this regular expression
429 to perform a substitution: we are then allowed to use ``\1FOOBAR\3`` as
432 If we manually coded ``seed`` out, it might look like:
434 .. code-block:: python
437 'WIZARD_DBSERVER': ('wp-config.php', re.compile(r'''^(define\('DB_HOST', )(.*)(\))''', re.M)),
438 'WIZARD_DBNAME': ('wp-config.php', re.compile(r'''^(define\('DB_NAME', )(.*)(\))''', re.M)),
441 There's a lot of duplication, though. For one thing, the regular expressions are almost
442 identical, safe for a single substitution within the string. We have a function
443 :meth:`wizard.app.php.re_define` that does this for us:
445 .. code-block:: python
448 'WIZARD_DBSERVER': ('wp-config.php', php.re_define('DB_HOST')),
449 'WIZARD_DBNAME': ('wp-config.php', php.re_define('DB_NAME')),
454 If you find yourself needing to define a custom regular expression generation function,
455 be sure to use :func:`wizard.app.expand_re`, which will escape an incoming variable
456 to be safe for inclusion in a regular expression, and also let you pass a list,
457 and have correct behavior. Check out :mod:`wizard.app.php` for some examples.
459 Additionally, if you are implementing a function for another language, or a general pattern of
460 variables, consider placing it in an appropriate language module instead.
462 We can shorten this even further: in most cases, all of the configuration values live in
463 one file, so let's make ourselves a function that generates the whole tuple:
465 .. code-block:: python
467 def make_filename_regex_define(var):
468 return 'wp-config.php', php.re_define(var)
470 Then we can use :func:`wizard.util.dictmap` to apply this:
472 .. code-block:: python
474 seed = util.dictmap(make_filename_regex_define, {
475 'WIZARD_DBSERVER': 'DB_HOST',
476 'WIZARD_DBNAME': 'DB_NAME',
477 'WIZARD_DBUSER': 'DB_USER',
478 'WIZARD_DBPASSWORD': 'DB_PASSWORD',
481 Short and sweet. From there, setting up :attr:`~wizard.app.Application.extractors` and
482 :attr:`~wizard.app.Application.substitutions` is easy:
484 .. code-block:: python
486 class Application(app.Application):
488 extractors = app.make_extractors(seed)
489 extractors.update(php.extractors)
490 substitutions = app.make_substitutions(seed)
491 substitutions.update(php.substitutions)
493 Note how we combine our own dictionaries with the dictionaries of :mod:`wizard.app.php`, much like
494 we did for :attr:`~wizard.app.Application.parametrized_files`.
496 With all of these pieces in place, run the following command::
498 wizard prepare-config
500 If everything is working, when you open up the configuration files, any user-specific
501 variables should have been replaced by ``WIZARD_FOOBAR`` variables. If not, check
502 your regular expressions, and then try running the command again.
504 When you are satisfied with your changes, add your files, amend your previous
505 commit with these changes and force them back into the public repository::
508 git add wp-config.php
509 git commit --amend -a
512 You should test again if your install script works; it probably doesn't,
513 since you now have a configuration file hanging around. Use
514 :func:`wizard.util.soft_unlink` to remove the file at the very beginning
515 of the install process.
520 Congratulations! You have just implemented the installation code for a new install.
521 If you have other copies of the application checked out, you can pull the forced
524 git reset --hard HEAD~
527 One last thing to do: after you are sure that your commit is correct, tag the new
528 commit as ``appname-x.y.z-scripts``, or in this specific case::
530 git tag wordpress-2.0.4-scripts
536 Here is short version for quick reference:
538 #. Create the new repository and new module,
539 #. Implement :meth:`~wizard.app.Application.download`,
540 #. Register the application at the ``wizard_scripts`` plugin,
541 #. *For Conversions:* Find the oldest extant version with ``wizard summary version $APP``,
542 #. Run ``wizard prepare-pristine $VERSION``,
543 #. Commit with ``-m "$APPLICATION $VERSION"`` and tag ``$APP-$VERSION``,
544 #. Create ``pristine`` branch, but stay on ``master`` branch,
545 #. *For Conversions:* Check for pre-existing patches, and apply them,
546 #. Run ``wizard prepare-new``,
547 #. *For PHP:* Copy in :file:`php.ini` file and run ``athrun scripts fix-php-ini``,
548 #. Commit with ``-m "$APPLICATION $VERSION"``, but *don't* tag,
549 #. Implement :data:`~wizard.app.Application.install_schema` and :meth:`~wizard.app.Application.install`,
550 #. Create :file:`tests/$APP-install-test.sh`,
551 #. On a scripts server, run ``wizard install $APP --no-commit`` and check changes with ``git status``,
552 #. Implement :attr:`~wizard.app.Application.extractors`,
553 :attr:`~wizard.app.Application.substitutions`, :attr:`~wizard.app.Application.parametrized_files`,
554 :meth:`~wizard.app.Application.checkConfig` and :meth:`~wizard.app.Application.detectVersion`,
555 #. Run ``wizard prepare-config``,
556 #. Amend commit and push back, and finally
557 #. Tag ``$APP-$VERSION-scripts``
562 You've only implemented a scriptsified version for only a single version; most applications
563 have multiple versions--you will have to do this process again. Fortunately, the most
564 time consuming parts (implementing logic for :class:`wizard.app.Application`) are already,
565 done so the process of :doc:`creating upgrades <upgrade>` is much simpler.
567 There is still functionality yet undone: namely the methods for actually performing an
568 upgrade are not yet implemented. You can find instructions for this on the
569 :doc:`creating upgrades <upgrade>` page under "Implementation".