X-Git-Url: https://scripts.mit.edu/gitweb/wizard.git/blobdiff_plain/11df8e59c2a3fae7db21ad8d983b77d561826667..3a2a3f6faeb318d11d9474e72c0b5fb9d29ad14c:/doc/testing.rst diff --git a/doc/testing.rst b/doc/testing.rst index 31b4265..6a4b8e0 100644 --- a/doc/testing.rst +++ b/doc/testing.rst @@ -12,7 +12,8 @@ Unit tests Unit tests can be run by running ``nosetests`` in the Wizard root directory (the directory that contains a :file:`wizard` and :file:`bin` -directory). These should run and pass on all platforms. +directory). These should run and pass on all platforms. Test files +tend to live in :file:`tests` directories under :file:`wizard`. Doctests -------- @@ -20,6 +21,9 @@ Doctests Certain pure functions contain doctests for their functionality. You can run them using ``sphinx-build -b doctest doc doc/_build`` in the Wizard root directory. These should run and pass on all platforms. +These are found inside of docstrings and are noted by a ``>>>`` marker. +They should be written for simple, self-contained functions that would +benefit from an example code usage. Acceptance tests ---------------- @@ -28,41 +32,64 @@ These are the most useful metrics of whether or not Wizard is working: they run Wizard commands and try to see if any of the commands return a non-zero exit code (future development may allow for pre-conditions and post-conditions to be checked). The test scripts are located in the -:file:`tests` directory, and are identifiably by their prefix ``test-`` -and their suffix ``.sh``. They should run out of the box on +:file:`tests` directory, and are identifiably by their suffix +``-test.sh``. They should run out of the box on scripts servers, so long as the Wizard source tree is inside of your :file:`web_scripts` directory, and require some configuration if you plan on running them locally. -Custom configuration can be specified in the :file:`config` file (located -at :file:`tests/config`. This is actually a Bash script to be sourced -by the real test script (:file:`tests/setup`), which exports various -environment variables that Wizard will use during installation. +These test scripts are the key to ensuring that Wizard is functioning +properly, and exercise most of the common code paths (and a few, though +not all, of the uncommon ones). They're organized by application name +and then test name. Every application should have a :file:`appname-install-test.sh` +which can be run without any parameters; you can also specify a version +number to test installing a particular version installation. There is +also :file:`appname-upgrade-test.sh`, which requires specifying a version +number and tests upgrading from that version to the latest version in +the repository. -Here is a sample file:: +.. note:: + + There are substantially more MediaWiki test scripts than for other applications, + a mixture of MediaWiki-specific tests and more general tests of functionality. + There may be some possible restructuring to reduce duplication for the + general tests. + +There are also utility scripts :file:`all.sh`, which takes a application +name and a version number as parameters, and runs all of the corresponding +test scripts, as well as :file:`clean.sh` which removes all test directories. + +Custom configuration can be specified in the :file:`config` file (located at +:file:`tests/config`). This is actually a Bash script to be sourced by the +real test script (:file:`tests/setup`), which exports various environment +variables that Wizard will use during installation. You should only need to +set these variables if you're attempting to run these tests off of a +non-scripts server. + +Here is a sample :file:`config` file:: + + MYSQL_ARGS="-uroot -ppassword" export WIZARD_WEB_HOST="localhost" export WIZARD_WEB_PATH="/wizard/tests/$TESTDIR" - export WIZARD_MYSQL_HOST="localhost" - export WIZARD_MYSQL_USER="root" - export WIZARD_MYSQL_PASSWORD="password" - export WIZARD_MYSQL_DB="wizard_test_$TESTID" + export WIZARD_DSN="mysql://root:password@localhost/wizard_test_$TESTID" export WIZARD_EMAIL="bob@example.com" -You will need to specify all of these environment variables. :envvar:`WIZARD_WEB_HOST` -and :envvar:`WIZARD_WEB_PATH` indicate Wizard's configuration with respect to -your web server. ``http://$WIZARD_WEB_HOST/$WIZARD_WEB_PATH`` will be the location -that a newly installed application will be accessible. You will notice that -we used :envvar:`TESTDIR`; this is the directory that will be created in -:file:`tests` for the application. :envvar:`WIZARD_MYSQL_HOST`, -:envvar:`WIZARD_MYSQL_USER`, :envvar:`WIZARD_MYSQL_PASSWORD` and :envvar:`WIZARD_MYSQL_DB` -are standard configuration variables for accessing a local MySQL database. You can use -:envvar:`TESTID` to uniquely identify any particular test. Finally, :envvar:`WIZARD_EMAIL` -is any email address you own that will be configured as an administrative email. - -It may be useful to include a little bit of code to handle dropping and creating -databases. Here is a sample:: - - MYSQL_ARGS="-uroot -prootpassword" - mysql $MYSQL_ARGS -e "DROP DATABASE \`$WIZARD_MYSQL_DB\`;" || true - mysql $MYSQL_ARGS -e "CREATE DATABASE \`$WIZARD_MYSQL_DB\`;" +You will need to specify all of the environment variables in the +:file:`config`. Those prefixed with ``WIZARD`` are directly used by Wizard, +while the ``MYSQL`` environment variables are used if a test script wants to +interactive directly with a MYSQL database. We don't quite have a good story +for alternative databases in test scripts. + +* :envvar:`WIZARD_WEB_HOST` and :envvar:`WIZARD_WEB_PATH` indicate Wizard's + configuration with respect to your web server. + ``http://$WIZARD_WEB_HOST/$WIZARD_WEB_PATH`` will be the location that a newly + installed application will be accessible. You will notice that we used + :envvar:`TESTDIR`; this is the directory that will be created in :file:`tests` + for the application. +* :envvar:`WIZARD_DSN` is the database source name, which + should be used to access a MySQL database (as Wizard does not contain support + for any other database system yet.) You can use :envvar:`TESTID` to uniquely + identify any particular test. +* :envvar:`WIZARD_EMAIL` is any email address you + own that will be configured as an administrative email.