Creating a repository
=====================
+:Author: Edward Z. Yang <ezyang@mit.edu>
+
.. highlight:: sh
Adding Wizard support for a web application requires some glue code
Here is a tutorial for creating such a repository, using an old version of
Wordpress as an example. We will implement only the functions necessary for
installing an application--upgrades and backups are not described here.
-
-.. supplement:: Conversions
-
- One of Wizard's goals is to replace the previous autoinstaller infrastructure.
- These boxes will explain extra steps that you must perform in order to carry
- out a conversion of old-style autoinstalls to a Wizard autoinstall.
- In brief, pre-wizard autoinstalls live in :file:`/mit/scripts/deploy` and
- consist of a tarball from upstream, possibly a scripts patch, and possibly
- some post-install munging (such as the creation of a :file:`php.ini` file
- and appropriate symlinks). Performing a conversion means that we will
- recreate these changes in our Wizard autoinstall, and you will start you
- repository with the *earliest* version of the application extant on our
- servers.
-
-Setup
------
-
-Probably the easiest way to do development is entirely on :term:`AFS`: all
-of your source code should live in publically readable (i.e.
-``system:anyuser`` as read permissions) directories, so that if you
-SSH into a scripts server to perform testing, you will be able
-to invoke your tools and read your development repository. In order
-to be able to run the test scripts in the tests directory, this
-is preferably in :file:`web_scripts`. In that
-case, setup is as simple as::
-
- git clone /mit/scripts/git/wizard.git /mit/$USER/web_scripts/wizard
- athrun consult fsr /mit/$USER/web_scripts/wizard system:anyuser read
- # for any application you're going to do development on, also:
- git clone /mit/scripts/git/autoinstalls/$APP.git /mit/$USER/web_scripts/wizard/srv/$APP
-
-If you'd like to be able to develop offline, just note that you will
-have to push your changes to AFS once you start doing testing on
-scripts servers, but before your changes get incorporated into
-canonical upstream. Git doesn't exactly make this easy, but you
-can follow this recipe::
-
- git clone /mit/scripts/git/wizard.git ~/wizard
- cd /mit/$USER
- mkdir wizard.git
- cd wizard.git
- git init --bare
- cd ~/wizard
- git remote add afs /mit/$USER/wizard.git
- git push -f afs master
- git clone /mit/$USER/wizard.git /mit/$USER/wizard
-
-And then you can perform updates from your local copy with::
-
- git push afs master
- cd /mit/$USER/wizard
- git pull
-
-If :file:`/mit/$USER/wizard.git` has write permissions for
-``system:scripts-security-upd``, this is especially useful if you were hacking
-on a copy living on ``not-backward.mit.edu``, and now need to transfer the
-changes back to the canonical repository (please don't give ``not-backward.mit.edu``
-your root tickets!) You can also setup a wizard directory similar to the
-first set of directions for on-server testing.
+We assume that you have a working setup of Wizard; consult the
+:doc:`setup documentation <setup>` for more details.
From this point on, we will assume you are doing development from an AFS directory
named ``$WIZARD``; note that application repositories live in ``$WIZARD/srv``.
+.. supplement:: Conversions
+
+ One of Wizard's goals is to replace the previous autoinstaller
+ infrastructure. These boxes will explain extra steps that you must perform
+ in order to carry out a conversion of old-style autoinstalls to a Wizard
+ autoinstall. In brief, pre-wizard autoinstalls live in
+ :file:`/mit/scripts/deploy` and consist of a tarball from upstream,
+ possibly a scripts patch, and possibly some post-install munging (such as
+ the creation of a :file:`php.ini` file and appropriate symlinks).
+ Performing a conversion means that we will recreate these changes in our
+ Wizard autoinstall, and you will start you repository with the *earliest*
+ version of the application extant on our servers.
+
Pristine
--------
-This is a tutorial centered around migrating `Wordpress <http://wordpress.org/>`_
-into our Git repository. For the sake of demonstration,
+This is a tutorial centered around creating a `Wordpress <http://wordpress.org/>`_
+repository. For the sake of demonstration,
we shall assume that this repository hasn't been created yet.
The repository then doesn't exist, we should create it::
git init
We also have to create a module for the application, so we
-create ``$WIZARD/wizard/app/wordpress.py`` and fill it in with a bare bones template:
+create :file:`$WIZARD/wizard/app/wordpress.py` and fill it in with a bare bones template:
.. code-block:: python
import os
import re
import logging
+ import distutils
from wizard import app, install, resolve, sql, util
from wizard.app import php
Now we are ready to put some code in our repository: the first thing we will
add is the :term:`pristine` branch, which contains verbatim the code from upstream.
-If we were starting a new autoinstaller, we'd pop off and use the latest version,
-but since we're dealing with legacy we want to start our repository history
-with the **oldest** version still extant on our servers. To find this out run::
- wizard summary version APP
+.. supplement:: Conversions
+
+ If we were starting a new autoinstaller, we'd pop off and use the latest version,
+ but since we're dealing with legacy we want to start our repository history
+ with the *oldest* version still extant on our servers. To find this out run::
-You'll need to be in the ``scripts-team`` list in order to access the data to
-run this command.
+ wizard summary version APP
-Try running the following command in :file:`$WIZARD/srv/wordpress`::
+ You'll need to be in the ``scripts-team`` list to have access rights to the
+ folder we store this information in: :file:`/mit/scripts/sec-tools/store/versions`.
+
+For the purposes of demonstration, we'll use Wordpress 2.0.2; in reality you
+should use the latest version. Try running the following command in
+:file:`$WIZARD/srv/wordpress`::
wizard prepare-pristine wordpress-2.0.2
.. code-block:: python
class Application(app.Application):
+ # ...
def download(self, version):
return "http://wordpress.org/wordpress-%s.tar.gz" % version
git branch pristine
-From the point of view of the pristine branch pointer, history should be a
-straight-forward progression of versions. Once you have more versions,
-it will look something like:
-
-.. digraph:: pristine_dag
-
- rankdir=LR
- node [shape=square]
- subgraph cluster_pristine {
- a -> b -> c
- a [label="1.0"]
- b [label="1.1"]
- c [label="2.0"]
- label = "pristine"
- color = white
- labeljust = r
- }
-
Scriptsify
----------
able to reconstruct a version of the application that was actually used, the
better off we will be when we try to subsequently upgrade those applications.
-To give you an idea of what the Git history graph will look like when we
-are done, here is the graph from before, but augmented with the scripts versions:
-
-.. digraph:: master_dag
-
- rankdir=LR
- node [shape=square]
- subgraph cluster_pristine {
- a -> b -> c
- a [label="1.0"]
- b [label="1.1"]
- c [label="2.0"]
- label = "pristine"
- color = white
- labeljust = r
- }
- subgraph cluster_master {
- as -> bs -> cs
- as [label="1.0-scripts"]
- bs [label="1.1-scripts"]
- cs [label="2.0-scripts"]
- label = "master"
- color = white
- labeljust = r
- }
- a -> as
- b -> bs
- c -> cs
- a []
-
First things first: verify that we are on the master branch::
git checkout master
-Check for pre-existing patches in the old application directory,
-:file:`/mit/scripts/deploy/wordpress-2.0.2` in the case of Wordpress,
-and apply them::
+.. supplement:: Conversions
+
+ Check for pre-existing patches in the old application directory,
+ :file:`/mit/scripts/deploy/wordpress-2.0.2` in the case of Wordpress,
+ and apply them::
- patch -n0 < /mit/scripts/deploy/wordpress-2.0.2/wordpress.patch
+ patch -n0 < /mit/scripts/deploy/wordpress-2.0.2/wordpress.patch
Then, run the following command to setup a :file:`.scripts` directory::
:command:`parallel-find.pl` to determine if a directory is an autoinstall.
Finally, if you are running a PHP application, you'll need to setup
-a :file:`php.ini` and symlinks to it in all subdirectories::
+a :file:`php.ini` and symlinks to it in all subdirectories.
+As of November 2009, all PHP applications load the same :file:`php.ini` file;
+so just grab one from another of the PHP projects. We'll rob our own
+crib in this case::
cp /mit/scripts/deploy/php.ini/wordpress php.ini
athrun scripts fix-php-ini
-.. note::
-
- As of November 2009, all PHP applications load the same :file:`php.ini` file.
-
Now commit, but don't get too attached to your commit; we're going
to be heavily modifying it soon::
these arguments inside :meth:`~wizard.app.Application.install` via
the ``options`` value.
+In particular, ``options.dsn`` is a :class:`sqlalchemy.engine.url.URL`
+which contains member variables such as :meth:`~sqlalchemy.engine.url.URL.username`,
+:meth:`~sqlalchemy.engine.url.URL.password`, :meth:`~sqlalchemy.engine.url.URL.host` and
+:meth:`~sqlalchemy.engine.url.URL.database` which you can use to pass in POST.
+
Some tips and tricks for writing :meth:`wizard.app.Application.install`:
* Some configuration file generators will get unhappy if the
* You should log any web page output using :func:`logging.debug`.
* If you need to manually manipulate the database afterwards, you
- can use :func:`wizard.sql.mysql_connect` (passing it ``options``)
+ can use :func:`wizard.sql.connect` (passing it ``options.dsn``)
to get a `SQLAlchemy metadata object
<http://www.sqlalchemy.org/docs/05/sqlexpression.html>`_, which can
consequently be queried. For convenience, we've bound metadata
env WIZARD_DEBUG=1 ./test-install-wordpress.sh
+The test scripts will try to conserve databases by running ``wizard remove`` on the
+old directory, but this requires :meth:`~wizard.app.remove` be implemented.
+Most of the time (namely, for single database setups), this simple template will suffice:
+
+.. code-block:: python
+
+ class Application(app.Application):
+ # ...
+ def remove(self, deployment)
+ app.remove_database(deployment)
+
Versioning config
-----------------
files. You're going to generate this generic copy now and in doing so,
overload your previous scripts commit. Because some installers
exhibit different behavior depending on server configuration, you should run
-the installation on a Scripts server::
+the installation on a Scripts server. You can do this manually or use
+the test script you created::
- wizard install wordpress --no-commit
+ env WIZARD_NO_COMMIT=1 ./test-install-wordpress.sh
-The installer will interactively prompt you for some values, and then conclude
-the install. ``--no-commit`` prevents the installer from generating a Git
-commit after the install, and will make it easier for us to propagate the
-change back to the parent repository.
+:envvar:`WIZARD_NO_COMMIT` (command line equivalent to ``--no-commit``)
+prevents the installer from generating a Git commit after the install, and will
+make it easier for us to propagate the change back to the parent repository.
-Look at the changes the installer made::
+Change into the generated directory and look at the changes the installer made::
git status
We could now grab the :mod:`re` module and start constructing a regex to grab ``2.0.4``, but it
turns out this isn't necessary: :meth:`wizard.app.php.re_var` does this for us already!
-With this function in hand, writing a version detection function is pretty straightforward.
-There is one gotcha: the value that ``re_var`` returns as the second subpattern is quoted (the reasons for this
-will become clear shortly), so you will need to trim off the last and first characters or
-use :mod:`shlex`. In the case of version numbers, there are probably no escape characters
-in the string, so the former is relatively safe.
+With this function in hand, writing a version detection function is pretty straightforward:
+we have a helper function that takes a file and a regex, and matches out the version number
+for us.
.. code-block:: python
class Application(app.Application):
# ...
def detectVersion(self, deployment):
- contents = open("wp-includes/version.php").read()
- match = php.re_var("wp_version").search(contents)
- if not match: return None
- return distutils.version.LooseVersion(match.group(2)[1:-1])
+ return self.detectVersionFromFile("wp-includes/version.php", php.re_var("wp_version"))
:attr:`~wizard.app.Application.parametrized_files` is a simple list of files that the
program's installer wrote or touched during the installation process.
# ...
parametrized_files = ['wp-config.php'] + php.parametrized_files
+.. _seed:
+
And finally, we have :attr:`~wizard.app.Application.extractors` and
:attr:`~wizard.app.Application.substitutions`. At the bare metal, these
are simply dictionaries of variable names to functions: when you call the
def make_filename_regex_define(var):
return 'wp-config.php', php.re_define(var)
-Then we can use :func:`util.dictmap` to apply this:
+Then we can use :func:`wizard.util.dictmap` to apply this:
.. code-block:: python
git tag wordpress-2.0.4-scripts
git push --tags
+Summary
+-------
+
+Here is short version for quick reference:
+
+#. Create the new repository and new module,
+#. Implement :meth:`~wizard.app.Application.download`,
+#. *For Conversions:* Find the oldest extant version with ``wizard summary version $APP``,
+#. Run ``wizard prepare-pristine $VERSION``,
+#. Commit with ``-m "$APPLICATION $VERSION"`` and tag ``$APP-$VERSION``,
+#. Create ``pristine`` branch, but stay on ``master`` branch,
+#. *For Conversions:* Check for pre-existing patches, and apply them,
+#. Run ``wizard prepare-new``,
+#. *For PHP:* Copy in :file:`php.ini` file and run ``athrun scripts fix-php-ini``,
+#. Commit with ``-m "$APPLICATION $VERSION"``, but *don't* tag,
+#. Implement :data:`~wizard.app.Application.install_schema` and :meth:`~wizard.app.Application.install`,
+#. Create :file:`tests/test-install-$APP.sh`,
+#. On a scripts server, run ``wizard install $APP --no-commit`` and check changes with ``git status``,
+#. Implement :attr:`~wizard.app.Application.extractors`,
+ :attr:`~wizard.app.Application.substitutions`, :attr:`~wizard.app.Application.parametrized_files`,
+ :meth:`~wizard.app.Application.checkConfig` and :meth:`~wizard.app.Application.detectVersion`,
+#. Run ``wizard prepare-config``,
+#. Amend commit and push back, and finally
+#. Tag ``$APP-$VERSION-scripts``
+
Further reading
---------------
You've only implemented a scriptsified version for only a single version; most applications
have multiple versions--you will have to do this process again. Fortunately, the most
time consuming parts (implementing logic for :class:`wizard.app.Application`) are already,
-done so you'll only have to tweak these algorithms if the application changes their
-format.
+done so the process of :doc:`creating upgrades <upgrade>` is much simpler.
-.. todo::
+There is still functionality yet undone: namely the methods for actually performing an
+upgrade are not yet implemented. You can find instructions for this on the
+:doc:`creating upgrades <upgrade>` page under "Implementation".
- Ultimately, we should have another condensed page that describes how to craft
- an update (with emphasis on what tests to perform to make sure things still
- work), and pages on how to implement upgrades and backups.
scripts.mit.edu / wizard.git / blobdiff