X-Git-Url: https://scripts.mit.edu/gitweb/wizard.git/blobdiff_plain/a0587472768ec8c72baa48708a4c3519814aa0f0..7b2533e1821daeb667b60c00dda4fd7d4ad3c41e:/doc/repository-conversion.rst diff --git a/doc/repository-conversion.rst b/doc/repository-conversion.rst index 5c529de..14a8547 100644 --- a/doc/repository-conversion.rst +++ b/doc/repository-conversion.rst @@ -1,19 +1,17 @@ Repository conversion ===================== -Pre-wizard autoinstalls live in:: +One of Wizard's goals is to replace the previous autoinstaller infrastructure. +Pre-wizard autoinstalls live in :file:`/mit/scripts/deploy` and consist of a +tarball from upstream, possibly a scripts patch, and possibly some post-install +munging (such as the creation of a :file:`php.ini` file and appropriate +symlinks). - /mit/scripts/deploy # production - /mit/scripts/deploydev # development - -And consist of a tarball from upstream, possibly a scripts -patch, and possibly some post-install munging (such as the creation -of a php.ini file and appropriate symlinks). - -Conversion to Wizard involves placing pristine versions of the source +Conversion to use Wizard involves placing :term:`pristine` versions of the source code (from the upstream tarballs) and appropriately patched scripts versions into a Git repository, as well as writing a :mod:`wizard.app` -module for the application that defines common operations. +module for the application that implements application specific logic, such +as how to install, upgrade or backup the installation. Here is a tutorial for performing a conversion, using Wordpress as an example. @@ -23,17 +21,19 @@ Setup .. highlight:: sh -Probably the easiest way to do development is entirely on AFS: all +Probably the easiest way to do development is entirely on :term:`AFS`: all of your source code should live in publically readable (i.e. -system:anyuser as read permissions) directories, so that if you +``system:anyuser`` as read permissions) directories, so that if you SSH into a scripts server to perform testing, you will be able -to invoke your tools and read your development repository. In that +to invoke your tools and read your development repository. In order +to be able to run the test scripts in the tests directory, this +is preferably in :file:`web_scripts`. In that case, setup is as simple as:: - git clone /mit/scripts/git/wizard.git /mit/$USER/wizard - athrun consult fsr /mit/$USER/wizard system:anyuser read + git clone /mit/scripts/git/wizard.git /mit/$USER/web_scripts/wizard + athrun consult fsr /mit/$USER/web_scripts/wizard system:anyuser read # for any application you're going to do development on, also: - git clone /mit/scripts/git/autoinstalls/$APP.git ~/wizard/srv/$APP + git clone /mit/scripts/git/autoinstalls/$APP.git /mit/$USER/web_scripts/wizard/srv/$APP If you'd like to be able to develop offline, just note that you will have to push your changes to AFS once you start doing testing on @@ -57,11 +57,12 @@ And then you can perform updates from your local copy with:: cd /mit/$USER/wizard git pull -If ``/mit/$USER/wizard.git`` has write permissions for +If :file:`/mit/$USER/wizard.git` has write permissions for ``system:scripts-security-upd``, this is especially useful if you were hacking on a copy living on ``not-backward.mit.edu``, and now need to transfer the changes back to the canonical repository (please don't give ``not-backward.mit.edu`` -your root tickets!) +your root tickets!) You can also setup a wizard directory similar to the +first set of directions for on-server testing. From this point on, we will assume you are doing development from an AFS directory named ``$WIZARD``; note that application repositories live in ``$WIZARD/srv``. @@ -71,8 +72,10 @@ Pristine .. highlight:: sh -This is a tutorial for migrating Wordpress into our Git repository. As such -a repository doesn't exist, we should create it:: +This is a tutorial centered around migrating `Wordpress `_ +into our Git repository. For the sake of demonstration, +we shall assume that this repository hasn't been created yet. +The repository then doesn't exist, we should create it:: cd "$WIZARD/srv" mkdir wordpress @@ -91,7 +94,7 @@ create ``$WIZARD/wizard/app/wordpress.py`` and fill it in with a bare bones temp .. highlight:: sh Now we are ready to put some code in our repository: the first thing we will -add is the "pristine branch", which contains verbatim the code from upstream. +add is the :term:`pristine` branch, which contains verbatim the code from upstream. If we were starting a new autoinstaller, we'd pop off and use the latest version, but since we're dealing with legacy we want to start our repository history with the **oldest** version still extant on our servers. To find this out run:: @@ -101,10 +104,197 @@ with the **oldest** version still extant on our servers. To find this out run:: You'll need to be in the ``scripts-team`` list in order to access the data to run this command. -Try running the following command in "$WIZARD/srv/wordpress":: +Try running the following command in :file:`$WIZARD/srv/wordpress`:: wizard prepare-pristine wordpress-2.0.2 -You should get an error complaining about ``download()`` not being implemented yet. +.. highlight:: python + +You should get an error complaining about :meth:`wizard.app.Application.download` +not being implemented yet. Let's fix that:: + + class Application(app.Application): + def download(self, version): + return "http://wordpress.org/wordpress-%s.tar.gz" % version + +.. highlight:: sh + +We determined this by finding `Wordpress's Release Archive `_ +and inferring the naming scheme by inspecting various links. You should now +be able to run the prepare-pristine command successfully: when it is +done, you'll now have a bunch of files in your repository, and they +will be ready to be committed. Inspect the files and commit (note that the +format of the commit message is a plain Appname Version.Number):: + + git status + git commit -asm "Wordpress 2.0.2" + git tag wordpress-2.0.2 + +.. note:: + + Sometimes, ``http://wordpress.org/wordpress-2.0.2.tar.gz`` won't + actually exist anymore (it didn't exist when we did it). In this case, + you'll probably be able to find the original tarball in + :file:`/mit/scripts/deploy/wordpress-2.0.2`, and you can feed it + manually to prepare pristine with + ``wizard prepare-pristine /mit/scripts/deploy/wordpress-2.0.2/wordpress-2.0.2.tar.gz`` + +Some last house-keeping bits: now that you have a commit in a repository, you +can also create a pristine branch:: + + git branch pristine + +From the point of view of the pristine branch pointer, history should be a +straight-forward progression of versions. Once you have more versions, +it will look something like: + +.. digraph:: pristine_dag + + rankdir=LR + node [shape=square] + subgraph cluster_pristine { + a -> b -> c + a [label="1.0"] + b [label="1.1"] + c [label="2.0"] + label = "pristine" + color = white + labeljust = r + } + +Scriptsify +---------- + +In a perfect world, the pristine version would be equivalent to the scriptsified +version that would actually get deployed. However, we have historically needed +to apply patches and add extra configuration files to get applications to +work correctly. Due to the way Git's merge algorithm works, the closer we are +able to reconstruct a version of the application that was actually used, the +better off we will be when we try to subsequently upgrade those applications. + +To give you an idea of what the Git history graph will look like when we +are done, here is the graph from before, but augmented with the scripts versions: + +.. digraph:: master_dag + + rankdir=LR + node [shape=square] + subgraph cluster_pristine { + a -> b -> c + a [label="1.0"] + b [label="1.1"] + c [label="2.0"] + label = "pristine" + color = white + labeljust = r + } + subgraph cluster_master { + as -> bs -> cs + as [label="1.0-scripts"] + bs [label="1.1-scripts"] + cs [label="2.0-scripts"] + label = "master" + color = white + labeljust = r + } + a -> as + b -> bs + c -> cs + a [] + +.. highlight:: sh + +First things first: verify that we are on the master branch:: + + git checkout master + +Check for pre-existing patches in the old application directory, +:file:`/mit/scripts/deploy/wordpress-2.0.2` in the case of Wordpress, +and apply them:: + + patch -n0 < /mit/scripts/deploy/wordpress-2.0.2/wordpress.patch + +Then, run the following command to setup a :file:`.scripts` directory:: + + wizard prepare-new + +This directory holds Wizard related files, and is also used by +:command:`parallel-find.pl` to determine if a directory is an autoinstall. + +Finally, if you are running a PHP application, you'll need to setup +a :file:`php.ini` and symlinks to it in all subdirectories:: + + cp /mit/scripts/deploy/php.ini/wordpress php.ini + athrun scripts fix-php-ini + +.. note:: + + As of November 2009, all PHP applications load the same :file:`php.ini` file. + +Now commit, but don't get too attached to your commit; we're going +to be heavily modifying it soon:: + + git commit -asm "Wordpress 2.0.2-scripts" + +Installation +------------ + +We now need to make it possible for a user to install the application. +Most web applications have a number of web scripts for generating a +configuration file, so creating the install script involves: + + 1. Determining what input values you will need from the user, such + as a title for the new application or database credentials; more + on this shortly. + + 2. Determining what POST values need to be sent to what URLs. + Since you're converting a repository, this job is even simpler: you just + need to port the Perl script that was originally used into Python. + +There's an in-depth explanation of named input values in +:mod:`wizard.install`. The short version is that your application +contains a class-wide :data:`~wizard.app.Application.install_schema` +attribute that encodes this information. The constructor takes +an arbitrary number of arguments (to be continued) + +Some tips and tricks for writing :meth:`wizard.app.Application.install`: + + * Some configuration file generators will get unhappy if the + target directory is not chmod'ed to be writable; dropping + in a ``os.chmod(dir, 0777)`` and then undoing the chmod + when you're done is a decent workaround. + + * :func:`wizard.install.fetch` is the standard workhorse for making + requests to applications. It accepts three parameters; the first + is ``options`` (which was the third argument to ``install`` itself), + the second is the page to query, relative to the installation's + web root, and ``post`` is a dictionary of keys to values to POST. + + * You should log any web page output using :func:`logging.debug`. + + * If you need to manually manipulate the database afterwards, you + can use :func:`wizard.sql.mysql_connect` (passing it ``options``) + to get a `SQLAlchemy metadata object + `_, which can + consequently be queried. For convenience, we've bound metadata + to the connection, you can perform implicit execution. + +To test if your installation function works, it's probably convenient to +create a test script in :file:`tests`; :file:`tests/test-install-wordpress.sh` +in the case of Wordpress. It will look something like:: + + #!/bin/bash -e + + DEFAULT_HEAD=1 + TESTNAME="install_wordpress" + source ./setup + + wizard install "wordpress-$VERSION-scripts" "$TESTDIR" --non-interactive -- --title="My Blog" + +``DEFAULT_HEAD=1`` indicates that this script can perform a reasonable +operation without any version specified (since we haven't tagged any of our +commits yet, we can't use the specific version functionality; not that we'd want +to, though). ``TESTNAME`` is simply the name of the file with the leading +``test-`` stripped and dashes converted to underscores. (to be continued)