X-Git-Url: https://scripts.mit.edu/gitweb/wizard.git/blobdiff_plain/64a898751caa752c5339adf122eea77ccf36153a..a504eefd8719a52b14217c2a1152db151504f99a:/doc/create.rst diff --git a/doc/create.rst b/doc/create.rst index 1e4e88d..84de71c 100644 --- a/doc/create.rst +++ b/doc/create.rst @@ -1,7 +1,7 @@ Creating a repository ===================== -:Authors: Edward Z. Yang +:Author: Edward Z. Yang .. highlight:: sh @@ -16,6 +16,11 @@ upgrade or backup the installation. Here is a tutorial for creating such a repository, using an old version of Wordpress as an example. We will implement only the functions necessary for installing an application--upgrades and backups are not described here. +We assume that you have a working setup of Wizard; consult the +:doc:`setup documentation ` for more details. + +From this point on, we will assume you are doing development from an AFS directory +named ``$WIZARD``; note that application repositories live in ``$WIZARD/srv``. .. supplement:: Conversions @@ -30,72 +35,6 @@ installing an application--upgrades and backups are not described here. Wizard autoinstall, and you will start you repository with the *earliest* version of the application extant on our servers. -Setup ------ - -Probably the easiest way to do development is entirely on :term:`AFS`, so that if you -SSH into a scripts server to perform testing, you will be able -to invoke your tools and read your development repository. In order -to be able to run the test scripts in the tests directory, this -is preferably in :file:`web_scripts`. In that -case, setup is as simple as:: - - git clone /mit/scripts/git/wizard.git /mit/$USER/web_scripts/wizard - # for any application you're going to do development on, also: - git clone /mit/scripts/git/autoinstalls/$APP.git /mit/$USER/web_scripts/wizard/srv/$APP - -From this point on, we will assume you are doing development from an AFS directory -named ``$WIZARD``; note that application repositories live in ``$WIZARD/srv``. - -.. warning:: - - Other users will not be able to access your source code. If you - change the access control to allow ``system:anyuser`` to read your - source code (in case you want to let other people test your - code), do *not* give access to the :file:`tests` directory, - which may contain sensitive data. - -Advanced setup -'''''''''''''' - -These instructions are for if you'd like to be able to develop offline and are -on ``scripts-root``. Wizard will mostly work on your local machine, but you -won't be able to do all development offline; some steps in the development -process must be performed on scripts servers. Thus, the difficult part is -marshalling commits from one repository to another. Git doesn't exactly make -this easy, but you can follow this recipe:: - - git clone ssh://scripts@scripts.mit.edu/mit/scripts/git/wizard.git ~/wizard - cd /mit/$USER - mkdir wizard.git - cd wizard.git - git init --bare - cd ~/wizard - git remote add afs /mit/$USER/wizard.git - git push -f afs master - git clone /mit/$USER/wizard.git /mit/$USER/wizard - -We create a bare repository :file:`/mit/$USER/wizard.git` that you can push and -pull from, and then setup an alternate remote ``afs`` on your offline copy. - -And then you can perform updates from your local copy with:: - - git push afs master - cd /mit/$USER/wizard - git pull - -If :file:`/mit/$USER/wizard.git` has write permissions for -``system:scripts-security-upd``, this is especially useful if you were hacking -on a copy living on ``not-backward.mit.edu``, and now need to transfer the -changes back to the canonical repository (please don't give ``not-backward.mit.edu`` -your root tickets!) You can also setup a wizard directory similar to the -first set of directions for on-server testing. - -.. warning:: - - These instructions are not well tested. Let me know if you run into - any difficulties. - Pristine -------- @@ -117,6 +56,7 @@ create :file:`$WIZARD/wizard/app/wordpress.py` and fill it in with a bare bones import os import re import logging + import distutils from wizard import app, install, resolve, sql, util from wizard.app import php @@ -139,9 +79,9 @@ add is the :term:`pristine` branch, which contains verbatim the code from upstre folder we store this information in: :file:`/mit/scripts/sec-tools/store/versions`. For the purposes of demonstration, we'll use Wordpress 2.0.2; in reality you -should use the latest version. Try running the following command in -:file:`$WIZARD/srv/wordpress`:: +should use the latest version. Try running the following commands:: + cd "$WIZARD/srv/wordpress" wizard prepare-pristine wordpress-2.0.2 You should get an error complaining about :meth:`wizard.app.Application.download` @@ -150,6 +90,7 @@ not being implemented yet. Let's fix that: .. code-block:: python class Application(app.Application): + # ... def download(self, version): return "http://wordpress.org/wordpress-%s.tar.gz" % version @@ -178,29 +119,6 @@ can also create a pristine branch:: git branch pristine -.. todo:: - - The following text and graph should be put in a more general overview of - the topology of Wizard git repositories. - -From the point of view of the pristine branch pointer, history should be a -straight-forward progression of versions. Once you have more versions, -it will look something like: - -.. digraph:: pristine_dag - - rankdir=LR - node [shape=square] - subgraph cluster_pristine { - a -> b -> c - a [label="1.0"] - b [label="1.1"] - c [label="2.0"] - label = "pristine" - color = white - labeljust = r - } - Scriptsify ---------- @@ -211,41 +129,6 @@ work correctly. Due to the way Git's merge algorithm works, the closer we are able to reconstruct a version of the application that was actually used, the better off we will be when we try to subsequently upgrade those applications. -.. todo:: - - The following text and graph should be put in a more general overview of - the topology of Wizard git repositories. - -To give you an idea of what the Git history graph will look like when we -are done, here is the graph from before, but augmented with the scripts versions: - -.. digraph:: master_dag - - rankdir=LR - node [shape=square] - subgraph cluster_pristine { - a -> b -> c - a [label="1.0"] - b [label="1.1"] - c [label="2.0"] - label = "pristine" - color = white - labeljust = r - } - subgraph cluster_master { - as -> bs -> cs - as [label="1.0-scripts"] - bs [label="1.1-scripts"] - cs [label="2.0-scripts"] - label = "master" - color = white - labeljust = r - } - a -> as - b -> bs - c -> cs - a [] - First things first: verify that we are on the master branch:: git checkout master @@ -258,14 +141,7 @@ First things first: verify that we are on the master branch:: patch -n0 < /mit/scripts/deploy/wordpress-2.0.2/wordpress.patch -Then, run the following command to setup a :file:`.scripts` directory:: - - wizard prepare-new - -This directory holds Wizard related files, and is also used by -:command:`parallel-find.pl` to determine if a directory is an autoinstall. - -Finally, if you are running a PHP application, you'll need to setup +If you are running a PHP application, you'll need to setup a :file:`php.ini` and symlinks to it in all subdirectories. As of November 2009, all PHP applications load the same :file:`php.ini` file; so just grab one from another of the PHP projects. We'll rob our own @@ -273,6 +149,7 @@ crib in this case:: cp /mit/scripts/deploy/php.ini/wordpress php.ini athrun scripts fix-php-ini + git add . Now commit, but don't get too attached to your commit; we're going to be heavily modifying it soon:: @@ -283,16 +160,29 @@ Installation ------------ We now need to make it possible for a user to install the application. -Most web applications have a number of web scripts for generating a -configuration file, so creating the install script involves: +The :meth:`~wizard.install.Application.install` method should take the +application from a just cloned working copy into a fully functioning web +application with configuration and a working database, etc. Most web +applications have a number of web scripts for generating a configuration +file, so creating the install script tend to involve: + + + 1. Deleting any placeholder files that were in the repository (there + aren't any now, but there will be soon.) - 1. Determining what input values you will need from the user, such + 2. Determining what input values you will need from the user, such as a title for the new application or database credentials; more on this shortly. - 2. Determining what POST values need to be sent to what URLs. - Since you're converting a repository, this job is even simpler: you just - need to port the Perl script that was originally used into Python. + 3. Determining what POST values need to be sent to what URLs or to + what shell scripts (these are the install scripts the application + may have supplied to you.) + +.. supplement:: Conversions + + Since you're converting a repository, this job is even simpler: you + just need to port the Perl script that was originally used into + Python. There's an in-depth explanation of named input values in :mod:`wizard.install`. The short version is that your application @@ -306,6 +196,11 @@ be able to get away with pre-canned attributes. You can access these arguments inside :meth:`~wizard.app.Application.install` via the ``options`` value. +In particular, ``options.dsn`` is a :class:`sqlalchemy.engine.url.URL` +which contains member variables such as :meth:`~sqlalchemy.engine.url.URL.username`, +:meth:`~sqlalchemy.engine.url.URL.password`, :meth:`~sqlalchemy.engine.url.URL.host` and +:meth:`~sqlalchemy.engine.url.URL.database` which you can use to pass in POST. + Some tips and tricks for writing :meth:`wizard.app.Application.install`: * Some configuration file generators will get unhappy if the @@ -322,17 +217,12 @@ Some tips and tricks for writing :meth:`wizard.app.Application.install`: * You should log any web page output using :func:`logging.debug`. * If you need to manually manipulate the database afterwards, you - can use :func:`wizard.sql.mysql_connect` (passing it ``options``) + can use :func:`wizard.sql.connect` (passing it ``options.dsn``) to get a `SQLAlchemy metadata object `_, which can consequently be queried. For convenience, we've bound metadata to the connection, you can perform implicit execution. -.. todo:: - - Our installer needs to also parametrize :file:`php.ini`, which we haven't - done yet. - To test if your installation function works, it's probably convenient to create a test script in :file:`tests`; :file:`tests/test-install-wordpress.sh` in the case of Wordpress. It will look something like:: @@ -354,6 +244,17 @@ verbose debugging information by using:: env WIZARD_DEBUG=1 ./test-install-wordpress.sh +The test scripts will try to conserve databases by running ``wizard remove`` on the +old directory, but this requires :meth:`~wizard.app.remove` be implemented. +Most of the time (namely, for single database setups), this simple template will suffice: + +.. code-block:: python + + class Application(app.Application): + # ... + def remove(self, deployment) + app.remove_database(deployment) + Versioning config ----------------- @@ -362,16 +263,16 @@ the scriptsified versions would contain generic copies of the configuration files. You're going to generate this generic copy now and in doing so, overload your previous scripts commit. Because some installers exhibit different behavior depending on server configuration, you should run -the installation on a Scripts server:: +the installation on a Scripts server. You can do this manually or use +the test script you created:: - wizard install wordpress --no-commit + env WIZARD_NO_COMMIT=1 ./test-install-wordpress.sh -The installer will interactively prompt you for some values, and then conclude -the install. ``--no-commit`` prevents the installer from generating a Git -commit after the install, and will make it easier for us to propagate the -change back to the parent repository. +:envvar:`WIZARD_NO_COMMIT` (command line equivalent to ``--no-commit``) +prevents the installer from generating a Git commit after the install, and will +make it easier for us to propagate the change back to the parent repository. -Look at the changes the installer made:: +Change into the generated directory and look at the changes the installer made:: git status @@ -414,21 +315,16 @@ is stored: a quick grep tells us that it's in :file:`wp-includes/version.php`: We could now grab the :mod:`re` module and start constructing a regex to grab ``2.0.4``, but it turns out this isn't necessary: :meth:`wizard.app.php.re_var` does this for us already! -With this function in hand, writing a version detection function is pretty straightforward. -There is one gotcha: the value that ``re_var`` returns as the second subpattern is quoted (the reasons for this -will become clear shortly), so you will need to trim off the last and first characters or -use :mod:`shlex`. In the case of version numbers, there are probably no escape characters -in the string, so the former is relatively safe. +With this function in hand, writing a version detection function is pretty straightforward: +we have a helper function that takes a file and a regex, and matches out the version number +for us. .. code-block:: python class Application(app.Application): # ... def detectVersion(self, deployment): - contents = open("wp-includes/version.php").read() - match = php.re_var("wp_version").search(contents) - if not match: return None - return distutils.version.LooseVersion(match.group(2)[1:-1]) + return self.detectVersionFromFile("wp-includes/version.php", php.re_var("wp_version")) :attr:`~wizard.app.Application.parametrized_files` is a simple list of files that the program's installer wrote or touched during the installation process. @@ -448,6 +344,8 @@ we made: # ... parametrized_files = ['wp-config.php'] + php.parametrized_files +.. _seed: + And finally, we have :attr:`~wizard.app.Application.extractors` and :attr:`~wizard.app.Application.substitutions`. At the bare metal, these are simply dictionaries of variable names to functions: when you call the @@ -548,6 +446,11 @@ commit with these changes and force them back into the public repository:: git commit --amend -a git push --force +You should test again if your install script works; it probably doesn't, +since you now have a configuration file hanging around. Use +:func:`wizard.util.soft_unlink` to remove the file at the very beginning +of the install process. + Ending ceremonies ----------------- @@ -564,17 +467,40 @@ commit as ``appname-x.y.z-scripts``, or in this specific case:: git tag wordpress-2.0.4-scripts git push --tags +Summary +------- + +Here is short version for quick reference: + +#. Create the new repository and new module, +#. Implement :meth:`~wizard.app.Application.download`, +#. *For Conversions:* Find the oldest extant version with ``wizard summary version $APP``, +#. Run ``wizard prepare-pristine $VERSION``, +#. Commit with ``-m "$APPLICATION $VERSION"`` and tag ``$APP-$VERSION``, +#. Create ``pristine`` branch, but stay on ``master`` branch, +#. *For Conversions:* Check for pre-existing patches, and apply them, +#. Run ``wizard prepare-new``, +#. *For PHP:* Copy in :file:`php.ini` file and run ``athrun scripts fix-php-ini``, +#. Commit with ``-m "$APPLICATION $VERSION"``, but *don't* tag, +#. Implement :data:`~wizard.app.Application.install_schema` and :meth:`~wizard.app.Application.install`, +#. Create :file:`tests/test-install-$APP.sh`, +#. On a scripts server, run ``wizard install $APP --no-commit`` and check changes with ``git status``, +#. Implement :attr:`~wizard.app.Application.extractors`, + :attr:`~wizard.app.Application.substitutions`, :attr:`~wizard.app.Application.parametrized_files`, + :meth:`~wizard.app.Application.checkConfig` and :meth:`~wizard.app.Application.detectVersion`, +#. Run ``wizard prepare-config``, +#. Amend commit and push back, and finally +#. Tag ``$APP-$VERSION-scripts`` + Further reading --------------- You've only implemented a scriptsified version for only a single version; most applications have multiple versions--you will have to do this process again. Fortunately, the most time consuming parts (implementing logic for :class:`wizard.app.Application`) are already, -done so you'll only have to tweak these algorithms if the application changes their -format. +done so the process of :doc:`creating upgrades ` is much simpler. -.. todo:: +There is still functionality yet undone: namely the methods for actually performing an +upgrade are not yet implemented. You can find instructions for this on the +:doc:`creating upgrades ` page under "Implementation". - Ultimately, we should have another condensed page that describes how to craft - an update (with emphasis on what tests to perform to make sure things still - work), and pages on how to implement upgrades and backups.