Issue 5383 - Add tests for the Chrome and Firefox packagers

tests/README.md

Index: tests/README.md
diff --git a/tests/README.md b/tests/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..4e4216de4d81183fb1dc7e6586f3240b69e4b1f1
--- /dev/null
+++ b/tests/README.md
@@ -0,0 +1,71 @@
+# Buildtools unit tests
+
+## Introduction
+
+Unit tests for buildtools, using the pytest framework.
+
+## Approach
+
+In order to test the buildtools capability of creating WebExtension-packages
+for Chrome, Edge and Firefox, an example configuration for each platform is
+provided.
+Running the tests calls the same API as the cli would, except for releases
+(we don't want to trigger our releaseAutomation during tests) - in this case,
+a build is manually created and verified)
+
+Each extension is build with different parameters, the resulting package and
+it's content is compared against provided expected results:
+
+- Release or build-only, with or without a specific build-number (Edge)
+- Release, build-only or developement environment, with or without a specifc
+  build-number (Chrome, Firefox), with or without a predefined signing key
+  (Chrome)
+
+The expected results for each manifest are provided with the files in
+`expecteddata/`.
+
+## Test cases
+
+_(covered platforms are referred to as C=Chrome, E=Edge, F=Firefox)_
+
+- Metadata inheritance (EF)
+- correct package filename (CEF)
+- Printed warning about non-square icons (CF)
+- Presence of javascript unit test files in developement environment (CF) 
+- Absence of javascript unit test files in build-only or release (CF)
+- Inlcusion of defined contentScripts into the manifest (CF)
+- Packaging (and moving) of included icons / scripts / htmls (CEF)
+- Presence of files for all configured locales (CEF)
+- Correct import of translations with or without placeholders, with or without
+  access keys (CF)
+- Translation presence and validity for the Chrome Web Store (C)
+- Packaging of modularized script files (CF)
+- Adherence to provided build-number in the manifest (CEF)
+- Other content in the manifest (CEF)
+- Correct encrypted signature of the package (C)
+
+## Requirements
+
+- [Tox](https://pypi.python.org/pypi/tox) (>= 2.7.0)
+
+_(Tox will take care of installing the other dependencies such as flake8 and
+flake8-abp inside a virtualenv.)_
+
+## Usage
+
+To run the tests simply run
+
+   `tox`

[Vasily Kuznetsov, 2017/09/19 17:52:26]

You can use a 4 space indent here instead, and I would also suggest adding a
shell prompt before "tox" so it's clear that we run it from shell prompt.
Something like this:

    $ tox

[Sebastian Noack, 2017/09/20 00:58:01]

Actually, since the whole paragraph consists of code, you should rather use
triple backticks and no indentation at all:

```
$ tox
```

[tlucas, 2017/09/20 08:52:50]

Done.

[Vasily Kuznetsov, 2017/09/20 14:53:04]

Seems like it's more cluttered and harder to read source code that produces the
same output. Why do you think this is better than just a simple code block?

[Sebastian Noack, 2017/09/20 21:36:09]

The output is usually not the same (it might depend on the markdown renderer
though). Single backticks usually (e.g. on GitHub) don't render as code block,
but render a code snippet, and are meant for code inside a text paragraph.

[Vasily Kuznetsov, 2017/09/21 09:29:02]

I agree about the single backticks, but my proposal was a code block
(https://daringfireball.net/projects/markdown/syntax#precode).

[Sebastian Noack, 2017/09/21 19:09:02]

Oh, I didn't know of that syntax, I was assuming you were suggesting to keep the
single backticks but just change the indentation. But indeed indented code is
rendered the same as code wrapped in triple backticks, but reads nicer in plain
text. The only downside seems to be that you cannot annotate the programming
language for code hightlighting, but that doesn't matter here.

Tristan, so if you want to go with Vasily's suggestion, fine with me.

+
+in the buildtools' rootfolder.

[Sebastian Noack, 2017/09/20 00:58:01]

In English, you can not make up new words by putting two words together. So it
is "root folder", not "rootfolder".

[tlucas, 2017/09/20 08:52:51]

Done.

+
+## Coverage
+
+Pytest will create a coverage report as output to the terminal, using the
+pytest plugin `pytest-cov`.
+
+In order to get an html report you can add `--cov-report=html` to the
+pytest-command in tox.ini.
+
+For more information please refer to the
+[coverage documentation](https://coverage.readthedocs.io/)