X-Git-Url: https://scripts.mit.edu/gitweb/wizard.git/blobdiff_plain/31253eca10ab83a98b0921ac6bbbb22b90ff2af3..c4a68ce8e6a291ca66d2f73c7e3fdded87aa5f1f:/doc/create.rst diff --git a/doc/create.rst b/doc/create.rst index 352cc63..0295383 100644 --- a/doc/create.rst +++ b/doc/create.rst @@ -16,6 +16,11 @@ upgrade or backup the installation. Here is a tutorial for creating such a repository, using an old version of Wordpress as an example. We will implement only the functions necessary for installing an application--upgrades and backups are not described here. +We assume that you have a working setup of Wizard; consult the +:doc:`setup documentation ` for more details. + +From this point on, we will assume you are doing development from an AFS directory +named ``$WIZARD``; note that application repositories live in ``$WIZARD/srv``. .. supplement:: Conversions @@ -30,72 +35,6 @@ installing an application--upgrades and backups are not described here. Wizard autoinstall, and you will start you repository with the *earliest* version of the application extant on our servers. -Setup ------ - -Probably the easiest way to do development is entirely on :term:`AFS`, so that if you -SSH into a scripts server to perform testing, you will be able -to invoke your tools and read your development repository. In order -to be able to run the test scripts in the tests directory, this -is preferably in :file:`web_scripts`. In that -case, setup is as simple as:: - - git clone /mit/scripts/git/wizard.git /mit/$USER/web_scripts/wizard - # for any application you're going to do development on, also: - git clone /mit/scripts/git/autoinstalls/$APP.git /mit/$USER/web_scripts/wizard/srv/$APP - -From this point on, we will assume you are doing development from an AFS directory -named ``$WIZARD``; note that application repositories live in ``$WIZARD/srv``. - -.. warning:: - - Other users will not be able to access your source code. If you - change the access control to allow ``system:anyuser`` to read your - source code (in case you want to let other people test your - code), do *not* give access to the :file:`tests` directory, - which may contain sensitive data. - -Advanced setup -'''''''''''''' - -These instructions are for if you'd like to be able to develop offline and are -on ``scripts-root``. Wizard will mostly work on your local machine, but you -won't be able to do all development offline; some steps in the development -process must be performed on scripts servers. Thus, the difficult part is -marshalling commits from one repository to another. Git doesn't exactly make -this easy, but you can follow this recipe:: - - git clone ssh://scripts@scripts.mit.edu/mit/scripts/git/wizard.git ~/wizard - cd /mit/$USER - mkdir wizard.git - cd wizard.git - git init --bare - cd ~/wizard - git remote add afs /mit/$USER/wizard.git - git push -f afs master - git clone /mit/$USER/wizard.git /mit/$USER/wizard - -We create a bare repository :file:`/mit/$USER/wizard.git` that you can push and -pull from, and then setup an alternate remote ``afs`` on your offline copy. - -And then you can perform updates from your local copy with:: - - git push afs master - cd /mit/$USER/wizard - git pull - -If :file:`/mit/$USER/wizard.git` has write permissions for -``system:scripts-security-upd``, this is especially useful if you were hacking -on a copy living on ``not-backward.mit.edu``, and now need to transfer the -changes back to the canonical repository (please don't give ``not-backward.mit.edu`` -your root tickets!) You can also setup a wizard directory similar to the -first set of directions for on-server testing. - -.. warning:: - - These instructions are not well tested. Let me know if you run into - any difficulties. - Pristine -------- @@ -140,9 +79,9 @@ add is the :term:`pristine` branch, which contains verbatim the code from upstre folder we store this information in: :file:`/mit/scripts/sec-tools/store/versions`. For the purposes of demonstration, we'll use Wordpress 2.0.2; in reality you -should use the latest version. Try running the following command in -:file:`$WIZARD/srv/wordpress`:: +should use the latest version. Try running the following commands:: + cd "$WIZARD/srv/wordpress" wizard prepare-pristine wordpress-2.0.2 You should get an error complaining about :meth:`wizard.app.Application.download` @@ -180,29 +119,6 @@ can also create a pristine branch:: git branch pristine -.. todo:: - - The following text and graph should be put in a more general overview of - the topology of Wizard git repositories. - -From the point of view of the pristine branch pointer, history should be a -straight-forward progression of versions. Once you have more versions, -it will look something like: - -.. digraph:: pristine_dag - - rankdir=LR - node [shape=square] - subgraph cluster_pristine { - a -> b -> c - a [label="1.0"] - b [label="1.1"] - c [label="2.0"] - label = "pristine" - color = white - labeljust = r - } - Scriptsify ---------- @@ -213,41 +129,6 @@ work correctly. Due to the way Git's merge algorithm works, the closer we are able to reconstruct a version of the application that was actually used, the better off we will be when we try to subsequently upgrade those applications. -.. todo:: - - The following text and graph should be put in a more general overview of - the topology of Wizard git repositories. - -To give you an idea of what the Git history graph will look like when we -are done, here is the graph from before, but augmented with the scripts versions: - -.. digraph:: master_dag - - rankdir=LR - node [shape=square] - subgraph cluster_pristine { - a -> b -> c - a [label="1.0"] - b [label="1.1"] - c [label="2.0"] - label = "pristine" - color = white - labeljust = r - } - subgraph cluster_master { - as -> bs -> cs - as [label="1.0-scripts"] - bs [label="1.1-scripts"] - cs [label="2.0-scripts"] - label = "master" - color = white - labeljust = r - } - a -> as - b -> bs - c -> cs - a [] - First things first: verify that we are on the master branch:: git checkout master @@ -285,16 +166,29 @@ Installation ------------ We now need to make it possible for a user to install the application. -Most web applications have a number of web scripts for generating a -configuration file, so creating the install script involves: +The :meth:`~wizard.install.Application.install` method should take the +application from a just cloned working copy into a fully functioning web +application with configuration and a working database, etc. Most web +applications have a number of web scripts for generating a configuration +file, so creating the install script tend to involve: + - 1. Determining what input values you will need from the user, such + 1. Deleting any placeholder files that were in the repository (there + aren't any now, but there will be soon.) + + 2. Determining what input values you will need from the user, such as a title for the new application or database credentials; more on this shortly. - 2. Determining what POST values need to be sent to what URLs. - Since you're converting a repository, this job is even simpler: you just - need to port the Perl script that was originally used into Python. + 3. Determining what POST values need to be sent to what URLs or to + what shell scripts (these are the install scripts the application + may have supplied to you.) + +.. supplement:: Conversions + + Since you're converting a repository, this job is even simpler: you + just need to port the Perl script that was originally used into + Python. There's an in-depth explanation of named input values in :mod:`wizard.install`. The short version is that your application @@ -308,6 +202,11 @@ be able to get away with pre-canned attributes. You can access these arguments inside :meth:`~wizard.app.Application.install` via the ``options`` value. +In particular, ``options.dsn`` is a :class:`sqlalchemy.engine.url.URL` +which contains member variables such as :meth:`~sqlalchemy.engine.url.URL.username`, +:meth:`~sqlalchemy.engine.url.URL.password`, :meth:`~sqlalchemy.engine.url.URL.host` and +:meth:`~sqlalchemy.engine.url.URL.database` which you can use to pass in POST. + Some tips and tricks for writing :meth:`wizard.app.Application.install`: * Some configuration file generators will get unhappy if the @@ -324,17 +223,12 @@ Some tips and tricks for writing :meth:`wizard.app.Application.install`: * You should log any web page output using :func:`logging.debug`. * If you need to manually manipulate the database afterwards, you - can use :func:`wizard.sql.mysql_connect` (passing it ``options``) + can use :func:`wizard.sql.connect` (passing it ``options.dsn``) to get a `SQLAlchemy metadata object `_, which can consequently be queried. For convenience, we've bound metadata to the connection, you can perform implicit execution. -.. todo:: - - Our installer needs to also parametrize :file:`php.ini`, which we haven't - done yet. - To test if your installation function works, it's probably convenient to create a test script in :file:`tests`; :file:`tests/test-install-wordpress.sh` in the case of Wordpress. It will look something like:: @@ -356,6 +250,17 @@ verbose debugging information by using:: env WIZARD_DEBUG=1 ./test-install-wordpress.sh +The test scripts will try to conserve databases by running ``wizard remove`` on the +old directory, but this requires :meth:`~wizard.app.remove` be implemented. +Most of the time (namely, for single database setups), this simple template will suffice: + +.. code-block:: python + + class Application(app.Application): + # ... + def remove(self, deployment) + app.remove_database(deployment) + Versioning config ----------------- @@ -416,21 +321,16 @@ is stored: a quick grep tells us that it's in :file:`wp-includes/version.php`: We could now grab the :mod:`re` module and start constructing a regex to grab ``2.0.4``, but it turns out this isn't necessary: :meth:`wizard.app.php.re_var` does this for us already! -With this function in hand, writing a version detection function is pretty straightforward. -There is one gotcha: the value that ``re_var`` returns as the second subpattern is quoted (the reasons for this -will become clear shortly), so you will need to trim off the last and first characters or -use :mod:`shlex`. In the case of version numbers, there are probably no escape characters -in the string, so the former is relatively safe. +With this function in hand, writing a version detection function is pretty straightforward: +we have a helper function that takes a file and a regex, and matches out the version number +for us. .. code-block:: python class Application(app.Application): # ... def detectVersion(self, deployment): - contents = open("wp-includes/version.php").read() - match = php.re_var("wp_version").search(contents) - if not match: return None - return distutils.version.LooseVersion(match.group(2)[1:-1]) + return self.detectVersionFromFile("wp-includes/version.php", php.re_var("wp_version")) :attr:`~wizard.app.Application.parametrized_files` is a simple list of files that the program's installer wrote or touched during the installation process. @@ -450,6 +350,8 @@ we made: # ... parametrized_files = ['wp-config.php'] + php.parametrized_files +.. _seed: + And finally, we have :attr:`~wizard.app.Application.extractors` and :attr:`~wizard.app.Application.substitutions`. At the bare metal, these are simply dictionaries of variable names to functions: when you call the @@ -550,6 +452,11 @@ commit with these changes and force them back into the public repository:: git commit --amend -a git push --force +You should test again if your install script works; it probably doesn't, +since you now have a configuration file hanging around. Use +:func:`wizard.util.soft_unlink` to remove the file at the very beginning +of the install process. + Ending ceremonies ----------------- @@ -597,11 +504,9 @@ Further reading You've only implemented a scriptsified version for only a single version; most applications have multiple versions--you will have to do this process again. Fortunately, the most time consuming parts (implementing logic for :class:`wizard.app.Application`) are already, -done so you'll only have to tweak these algorithms if the application changes their -format. +done so the process of :doc:`creating upgrades ` is much simpler. -.. todo:: +There is still functionality yet undone: namely the methods for actually performing an +upgrade are not yet implemented. You can find instructions for this on the +:doc:`creating upgrades ` page under "Implementation". - Ultimately, we should have another condensed page that describes how to craft - an update (with emphasis on what tests to perform to make sure things still - work), and pages on how to implement upgrades and backups.