Issue 3985 - Unified test runner for sitescripts

README.md

 # Sitescripts
 
 ## Introduction
 
 The sitescripts repository contains many of the server side Python scripts that
 we use. There are both web handlers, which can be used via the multiplexer, and
 scripts that perform other tasks.
 
 The scripts are often unrelated and as such will not all be documented here. For
 more information about the individual scripts you will need to look at their
@@ skipping 63 matching lines @@
 So, to test any of the URL handlers in development do the following:
 
 1. Create a sitescripts configuration file that lists the web modules that you
 are testing under the `multiplexer` section. (Depending on the modules you are
 testing you may need to add additional configuration as well.)
 2. Save the configuration file somewhere where it will be found, for example
 `~/.sitescripts`.
 3. Type `python multiplexer.py`, it will start a web server at
 http://localhost:5000/ . This web server will use any URL handlers that have
 been defined in the modules you are testing to respond to requests.
+
+
+## Testing
+
+There are tests for some parts of the functionality of sitescripts. They are
+located in `test` directories and can be run via
+[Tox](https://tox.readthedocs.org/). There are two parts of the test suite:
+the tests that only depend on the packages that can be automatically installed
+by tox and the tests that have other dependencies (a database). The first part
+is run by tox by default:
+
+    $ tox
+
+The tests dependent on the database are covering the packages `filterhits` and

[Sebastian Noack, 2016/05/24 07:58:36]

Nice idea to use Vagrant for tests with non-python dependencies. Just some
thoughts on that approach:

1. The way you intend those tests to run, couldn't one simply setup the machine
with the infrastructure repository, rather than having a separate Vagrantfile
for testing in here?
2. If we integrate Vagrant for the tests here, couldn't we do it the other way
around, that people just have to run tox, which then setups the VM and runs
tests in there if necessary?

+`notifications`. They can be run in a standard prepared environment using
+[vagrant](https://www.vagrantup.com://www.vagrantup.com/):
+
+    $ vagrant up
+    Bringing machine 'default' up with 'virtualbox' provider...
+    ==> default: Importing base box 'debian/contrib-jessie64'...
+    ...
+    $ vagrant ssh
+    ...
+    vagrant@debiancontrib-jessie:~$ cd /vagrant
+    vagrant@debiancontrib-jessie:/vagrant$ tox -e filtehits,notifications

[mathias, 2016/05/23 22:25:32]

Do you really want a binary image from an un-trusted source being able to write
back files you could potentially execute on your host system? Even if not
malicious by intention: I do not see how one would trust a test environment with
potentially broken code and potentially just slightly different executable
objects that much.

+
+It's also possible to run the other tests inside of vagrant environment by not
+giving any arguments to `tox`.
+
+Please note that since the subpackages of sitescripts are not installed via
+`setup.py` but are instead imported directly from the working directory, the
+`.pyc` files that are created next to the `.py` files might be obsolete and or
+not matching the environment. This might cause the tests to fail and Tox/Pytest
+normally hints at the source of the problem in these cases. The problem can be
+fixed by deleting all `.pyc` files:
+
+    $ find . -name '*.pyc' | xargs rm

[mathias, 2016/05/23 22:25:32]

After addressing my comment above, you could transform this section into a
lecture about not mixing up the environments in the first place, and what to do
if it happened by accident.