As a system administration tool, Wizard employs a variety of testing mechanisms in order to verify that the code is working and that there are no regressions.
Unit tests can be run by running nosetests in the Wizard root directory (the directory that contains a wizard and bin directory). These should run and pass on all platforms. Test files tend to live in tests directories under wizard.
Certain pure functions contain doctests for their functionality. You can run them using sphinx-build -b doctest doc doc/_build in the Wizard root directory. These should run and pass on all platforms. These are found inside of docstrings and are noted by a >>> marker. They should be written for simple, self-contained functions that would benefit from an example code usage.
These are the most useful metrics of whether or not Wizard is working: they run Wizard commands and try to see if any of the commands return a non-zero exit code (future development may allow for pre-conditions and post-conditions to be checked). The test scripts are located in the tests directory, and are identifiably by their suffix -test.sh. They should run out of the box on scripts servers, so long as the Wizard source tree is inside of your web_scripts directory, and require some configuration if you plan on running them locally.
These test scripts are the key to ensuring that Wizard is functioning properly, and exercise most of the common code paths (and a few, though not all, of the uncommon ones). They’re organized by application name and then test name. Every application should have a appname-install-test.sh which can be run without any parameters; you can also specify a version number to test installing a particular version installation. There is also appname-upgrade-test.sh, which requires specifying a version number and tests upgrading from that version to the latest version in the repository.
Note
There are substantially more MediaWiki test scripts than for other applications, a mixture of MediaWiki-specific tests and more general tests of functionality. There may be some possible restructuring to reduce duplication for the general tests.
There are also utility scripts all.sh, which takes a application name and a version number as parameters, and runs all of the corresponding test scripts, as well as clean.sh which removes all test directories.
Custom configuration can be specified in the config file (located at tests/config). This is actually a Bash script to be sourced by the real test script (tests/setup), which exports various environment variables that Wizard will use during installation. You should only need to set these variables if you’re attempting to run these tests off of a non-scripts server.
Here is a sample config file:
MYSQL_ARGS="-uroot -ppassword"
export WIZARD_WEB_HOST="localhost"
export WIZARD_WEB_PATH="/wizard/tests/$TESTDIR"
export WIZARD_DSN="mysql://root:password@localhost/wizard_test_$TESTID"
export WIZARD_EMAIL="bob@example.com"
You will need to specify all of the environment variables in the config. Those prefixed with WIZARD are directly used by Wizard, while the MYSQL environment variables are used if a test script wants to interactive directly with a MYSQL database. We don’t quite have a good story for alternative databases in test scripts.