Handle Wordpress random keys correctly on install and upgrade.

[wizard.git] / doc / testing.rst

diff --git a/doc/testing.rst b/doc/testing.rst
index cabe3a989523de28dfaef6b85ec0edbd4eb9eb24..6a4b8e0f4bbcd1c84c0c4e30f81ea2e765c59d56 100644 (file)
--- a/doc/testing.rst
+++ b/doc/testing.rst
@@ -12,7 +12,8 @@ Unit tests
 
 Unit tests can be run by running ``nosetests`` in the Wizard root
 directory (the directory that contains a :file:`wizard` and :file:`bin`
-directory).  These should run and pass on all platforms.
+directory).  These should run and pass on all platforms.  Test files
+tend to live in :file:`tests` directories under :file:`wizard`.
 
 Doctests
 --------
@@ -20,6 +21,9 @@ Doctests
 Certain pure functions contain doctests for their functionality.  You
 can run them using ``sphinx-build -b doctest doc doc/_build`` in
 the Wizard root directory.  These should run and pass on all platforms.
+These are found inside of docstrings and are noted by a ``>>>`` marker.
+They should be written for simple, self-contained functions that would
+benefit from an example code usage.
 
 Acceptance tests
 ----------------
@@ -28,18 +32,41 @@ These are the most useful metrics of whether or not Wizard is working:
 they run Wizard commands and try to see if any of the commands return
 a non-zero exit code (future development may allow for pre-conditions and
 post-conditions to be checked).  The test scripts are located in the
-:file:`tests` directory, and are identifiably by their prefix ``test-``
-and their suffix ``.sh``.  They should run out of the box on
+:file:`tests` directory, and are identifiably by their suffix
+``-test.sh``.  They should run out of the box on
 scripts servers, so long as the Wizard source tree is inside of your
 :file:`web_scripts` directory, and require some configuration if you
 plan on running them locally.
 
-Custom configuration can be specified in the :file:`config` file (located
-at :file:`tests/config`.  This is actually a Bash script to be sourced
-by the real test script (:file:`tests/setup`), which exports various
-environment variables that Wizard will use during installation.
+These test scripts are the key to ensuring that Wizard is functioning
+properly, and exercise most of the common code paths (and a few, though
+not all, of the uncommon ones).  They're organized by application name
+and then test name.  Every application should have a :file:`appname-install-test.sh`
+which can be run without any parameters; you can also specify a version
+number to test installing a particular version installation.  There is
+also :file:`appname-upgrade-test.sh`, which requires specifying a version
+number and tests upgrading from that version to the latest version in
+the repository.
 
-Here is a sample file::
+.. note::
+
+    There are substantially more MediaWiki test scripts than for other applications,
+    a mixture of MediaWiki-specific tests and more general tests of functionality.
+    There may be some possible restructuring to reduce duplication for the
+    general tests.
+
+There are also utility scripts :file:`all.sh`, which takes a application
+name and a version number as parameters, and runs all of the corresponding
+test scripts, as well as :file:`clean.sh` which removes all test directories.
+
+Custom configuration can be specified in the :file:`config` file (located at
+:file:`tests/config`).  This is actually a Bash script to be sourced by the
+real test script (:file:`tests/setup`), which exports various environment
+variables that Wizard will use during installation.  You should only need to
+set these variables if you're attempting to run these tests off of a
+non-scripts server.
+
+Here is a sample :file:`config` file::
 
     MYSQL_ARGS="-uroot -ppassword"
 
@@ -48,11 +75,11 @@ Here is a sample file::
     export WIZARD_DSN="mysql://root:password@localhost/wizard_test_$TESTID"
     export WIZARD_EMAIL="bob@example.com"
 
-You will need to specify all of these environment variables.  Those prefixed
-with ``WIZARD`` are directly used by Wizard, while the ``MYSQL`` environment
-variables are used if a test script wants to interactive directly with a
-MYSQL database.  We don't quite have a good story for alternative databases
-in test scripts.
+You will need to specify all of the environment variables in the
+:file:`config`.  Those prefixed with ``WIZARD`` are directly used by Wizard,
+while the ``MYSQL`` environment variables are used if a test script wants to
+interactive directly with a MYSQL database.  We don't quite have a good story
+for alternative databases in test scripts.
 
 * :envvar:`WIZARD_WEB_HOST` and :envvar:`WIZARD_WEB_PATH`  indicate Wizard's
   configuration with respect to your web server.
@@ -66,4 +93,3 @@ in test scripts.
   identify any particular test.
 * :envvar:`WIZARD_EMAIL` is any email address you
   own that will be configured as an administrative email.
-