Repository conversion
=====================
-Pre-wizard autoinstalls live in::
+One of Wizard's goals is to replace the previous autoinstaller infrastructure.
+Pre-wizard autoinstalls live in :file:`/mit/scripts/deploy` and consist of a
+tarball from upstream, possibly a scripts patch, and possibly some post-install
+munging (such as the creation of a :file:`php.ini` file and appropriate
+symlinks).
- /mit/scripts/deploy # production
- /mit/scripts/deploydev # development
-
-And consist of a tarball from upstream, possibly a scripts
-patch, and possibly some post-install munging (such as the creation
-of a php.ini file and appropriate symlinks).
-
-Conversion to Wizard involves placing pristine versions of the source
+Conversion to use Wizard involves placing :term:`pristine` versions of the source
code (from the upstream tarballs) and appropriately patched scripts
versions into a Git repository, as well as writing a :mod:`wizard.app`
-module for the application that defines common operations.
+module for the application that implements application specific logic, such
+as how to install, upgrade or backup the installation.
Here is a tutorial for performing a conversion, using Wordpress as
an example.
.. highlight:: sh
-Probably the easiest way to do development is entirely on AFS: all
+Probably the easiest way to do development is entirely on :term:`AFS`: all
of your source code should live in publically readable (i.e.
-system:anyuser as read permissions) directories, so that if you
+``system:anyuser`` as read permissions) directories, so that if you
SSH into a scripts server to perform testing, you will be able
-to invoke your tools and read your development repository. In that
+to invoke your tools and read your development repository. In order
+to be able to run the test scripts in the tests directory, this
+is preferably in :file:`web_scripts`. In that
case, setup is as simple as::
- git clone /mit/scripts/git/wizard.git /mit/$USER/wizard
- athrun consult fsr /mit/$USER/wizard system:anyuser read
+ git clone /mit/scripts/git/wizard.git /mit/$USER/web_scripts/wizard
+ athrun consult fsr /mit/$USER/web_scripts/wizard system:anyuser read
# for any application you're going to do development on, also:
- git clone /mit/scripts/git/autoinstalls/$APP.git ~/wizard/srv/$APP
+ git clone /mit/scripts/git/autoinstalls/$APP.git /mit/$USER/web_scripts/wizard/srv/$APP
If you'd like to be able to develop offline, just note that you will
have to push your changes to AFS once you start doing testing on
cd /mit/$USER/wizard
git pull
-If ``/mit/$USER/wizard.git`` has write permissions for
+If :file:`/mit/$USER/wizard.git` has write permissions for
``system:scripts-security-upd``, this is especially useful if you were hacking
on a copy living on ``not-backward.mit.edu``, and now need to transfer the
changes back to the canonical repository (please don't give ``not-backward.mit.edu``
-your root tickets!)
+your root tickets!) You can also setup a wizard directory similar to the
+first set of directions for on-server testing.
From this point on, we will assume you are doing development from an AFS directory
named ``$WIZARD``; note that application repositories live in ``$WIZARD/srv``.
.. highlight:: sh
-This is a tutorial for migrating Wordpress into our Git repository. As such
-a repository doesn't exist, we should create it::
+This is a tutorial centered around migrating `Wordpress <http://wordpress.org/>`_
+into our Git repository. For the sake of demonstration,
+we shall assume that this repository hasn't been created yet.
+The repository then doesn't exist, we should create it::
cd "$WIZARD/srv"
mkdir wordpress
.. highlight:: sh
Now we are ready to put some code in our repository: the first thing we will
-add is the "pristine branch", which contains verbatim the code from upstream.
+add is the :term:`pristine` branch, which contains verbatim the code from upstream.
If we were starting a new autoinstaller, we'd pop off and use the latest version,
but since we're dealing with legacy we want to start our repository history
with the **oldest** version still extant on our servers. To find this out run::
You'll need to be in the ``scripts-team`` list in order to access the data to
run this command.
-Try running the following command in "$WIZARD/srv/wordpress"::
+Try running the following command in :file:`$WIZARD/srv/wordpress`::
wizard prepare-pristine wordpress-2.0.2
-You should get an error complaining about ``download()`` not being implemented yet.
+.. highlight:: python
+
+You should get an error complaining about :meth:`wizard.app.Application.download`
+not being implemented yet. Let's fix that::
+
+ class Application(app.Application):
+ def download(self, version):
+ return "http://wordpress.org/wordpress-%s.tar.gz" % version
+
+.. highlight:: sh
+
+We determined this by finding `Wordpress's Release Archive <http://wordpress.org/download/release-archive/>`_
+and inferring the naming scheme by inspecting various links. You should now
+be able to run the prepare-pristine command successfully: when it is
+done, you'll now have a bunch of files in your repository, and they
+will be ready to be committed. Inspect the files and commit (note that the
+format of the commit message is a plain Appname Version.Number)::
+
+ git status
+ git commit -asm "Wordpress 2.0.2"
+ git tag wordpress-2.0.2
+
+.. note::
+
+ Sometimes, ``http://wordpress.org/wordpress-2.0.2.tar.gz`` won't
+ actually exist anymore (it didn't exist when we did it). In this case,
+ you'll probably be able to find the original tarball in
+ :file:`/mit/scripts/deploy/wordpress-2.0.2`, and you can feed it
+ manually to prepare pristine with
+ ``wizard prepare-pristine /mit/scripts/deploy/wordpress-2.0.2/wordpress-2.0.2.tar.gz``
+
+Some last house-keeping bits: now that you have a commit in a repository, you
+can also create a pristine branch::
+
+ git branch pristine
+
+From the point of view of the pristine branch pointer, history should be a
+straight-forward progression of versions. Once you have more versions,
+it will look something like:
+
+.. digraph:: pristine_dag
+
+ rankdir=LR
+ node [shape=square]
+ subgraph cluster_pristine {
+ a -> b -> c
+ a [label="1.0"]
+ b [label="1.1"]
+ c [label="2.0"]
+ label = "pristine"
+ color = white
+ labeljust = r
+ }
+
+Scriptsify
+----------
+
+In a perfect world, the pristine version would be equivalent to the scriptsified
+version that would actually get deployed. However, we have historically needed
+to apply patches and add extra configuration files to get applications to
+work correctly. Due to the way Git's merge algorithm works, the closer we are
+able to reconstruct a version of the application that was actually used, the
+better off we will be when we try to subsequently upgrade those applications.
+
+To give you an idea of what the Git history graph will look like when we
+are done, here is the graph from before, but augmented with the scripts versions:
+
+.. digraph:: master_dag
+
+ rankdir=LR
+ node [shape=square]
+ subgraph cluster_pristine {
+ a -> b -> c
+ a [label="1.0"]
+ b [label="1.1"]
+ c [label="2.0"]
+ label = "pristine"
+ color = white
+ labeljust = r
+ }
+ subgraph cluster_master {
+ as -> bs -> cs
+ as [label="1.0-scripts"]
+ bs [label="1.1-scripts"]
+ cs [label="2.0-scripts"]
+ label = "master"
+ color = white
+ labeljust = r
+ }
+ a -> as
+ b -> bs
+ c -> cs
+ a []
+
+.. highlight:: sh
+
+First things first: verify that we are on the master branch::
+
+ git checkout master
+
+Check for pre-existing patches in the old application directory,
+:file:`/mit/scripts/deploy/wordpress-2.0.2` in the case of Wordpress,
+and apply them::
+
+ patch -n0 < /mit/scripts/deploy/wordpress-2.0.2/wordpress.patch
+
+Then, run the following command to setup a :file:`.scripts` directory::
+
+ wizard prepare-new
+
+This directory holds Wizard related files, and is also used by
+:command:`parallel-find.pl` to determine if a directory is an autoinstall.
+
+Finally, if you are running a PHP application, you'll need to setup
+a :file:`php.ini` and symlinks to it in all subdirectories::
+
+ cp /mit/scripts/deploy/php.ini/wordpress php.ini
+ athrun scripts fix-php-ini
+
+.. note::
+
+ As of November 2009, all PHP applications load the same :file:`php.ini` file.
+
+Now commit, but don't get too attached to your commit; we're going
+to be heavily modifying it soon::
+
+ git commit -asm "Wordpress 2.0.2-scripts"
+
+Installation
+------------
+
+We now need to make it possible for a user to install the application.
+Most web applications have a number of web scripts for generating a
+configuration file, so creating the install script involves:
+
+ 1. Determining what input values you will need from the user, such
+ as a title for the new application or database credentials; more
+ on this shortly.
+
+ 2. Determining what POST values need to be sent to what URLs.
+ Since you're converting a repository, this job is even simpler: you just
+ need to port the Perl script that was originally used into Python.
+
+There's an in-depth explanation of named input values in
+:mod:`wizard.install`. The short version is that your application
+contains a class-wide :data:`~wizard.app.Application.install_schema`
+attribute that encodes this information. The constructor takes
+an arbitrary number of arguments (to be continued)
+
+Some tips and tricks for writing :meth:`wizard.app.Application.install`:
+
+ * Some configuration file generators will get unhappy if the
+ target directory is not chmod'ed to be writable; dropping
+ in a ``os.chmod(dir, 0777)`` and then undoing the chmod
+ when you're done is a decent workaround.
+
+ * :func:`wizard.install.fetch` is the standard workhorse for making
+ requests to applications. It accepts three parameters; the first
+ is ``options`` (which was the third argument to ``install`` itself),
+ the second is the page to query, relative to the installation's
+ web root, and ``post`` is a dictionary of keys to values to POST.
+
+ * You should log any web page output using :func:`logging.debug`.
+
+ * If you need to manually manipulate the database afterwards, you
+ can use :func:`wizard.sql.mysql_connect` (passing it ``options``)
+ to get a `SQLAlchemy metadata object
+ <http://www.sqlalchemy.org/docs/05/sqlexpression.html>`_, which can
+ consequently be queried. For convenience, we've bound metadata
+ to the connection, you can perform implicit execution.
+
+To test if your installation function works, it's probably convenient to
+create a test script in :file:`tests`; :file:`tests/test-install-wordpress.sh`
+in the case of Wordpress. It will look something like::
+
+ #!/bin/bash -e
+
+ DEFAULT_HEAD=1
+ TESTNAME="install_wordpress"
+ source ./setup
+
+ wizard install "wordpress-$VERSION-scripts" "$TESTDIR" --non-interactive -- --title="My Blog"
+
+``DEFAULT_HEAD=1`` indicates that this script can perform a reasonable
+operation without any version specified (since we haven't tagged any of our
+commits yet, we can't use the specific version functionality; not that we'd want
+to, though). ``TESTNAME`` is simply the name of the file with the leading
+``test-`` stripped and dashes converted to underscores.
(to be continued)