Issue 4970 - Document the library API of python-abp

README.md

 # python-abp
 
 This repository contains a library for working with Adblock Plus filter lists
 and the script that is used for building Adblock Plus filter lists from the
 form in which they are authored into the format suitable for consumption by the
 adblocking software.

[mathias, 2017/08/08 12:24:35]

For an introduction that is a bit too much. How about making two or three short
sentences out of this?

 
 ## Installation
 
 Prerequisites:
 
 * Linux, Mac OS X or Windows (any modern Unix should work too),
 * Python (2.7 or 3.5+),
 * pip.
 
 To install:
@@ skipping 30 matching lines @@
 The first instruction contains a URL that will be fetched and inserted at the
 point of reference. 
 The second one contains a path inside easylist repository.
 `flrender` needs to be able to find a copy of the repository on the local
 filesystem. We use `-i` option to point it to to the right directory:
 
     $ flrender -i easylist=/home/abc/easylist input.txt output.txt
 
 Now the second reference above will be resolved to
 `/home/abc/easylist/easylist/easylist_general_block.txt` and the fragment will
-be read from this file.
+be loaded from this file.
 
 Directories that contain filter list fragments that are used during rendering
 are called sources.
 They are normally working copies of the repositories that contain filter list
 fragments.
 Each source is identified by a name: that's the part that comes before ":"
 in the include instruction and it should be the same as what comes before "="
 in the `-i` option.
 
 Commonly used sources have generally accepted names. For example the main
@@ skipping 22 matching lines @@
 If `filterlist.txt` contains a filter list:
 
     [Adblock Plus 2.0]
     ! Title: Example list
 
     abc.com,cdf.com##div#ad1
     abc.com/ad$image
     @@/abc\.com/
     ...
 
-the output will look similar to the following:
+the output will look something like:
 
     Header(version='Adblock Plus 2.0')
     Metadata(key='Title', value='Example list')
     EmptyLine()
     Filter(text='abc.com,cdf.com##div#ad1', selector={'type': 'css', 'value': 'div#ad1'}, action='hide', options=[('domain', [('abc .com', True), ('cdf.com', True)])])
     Filter(text='abc.com/ad$image', selector={'type': 'url-pattern', 'value': 'abc.com/ad'}, action='block', options=[('image', True)])
     Filter(text='@@/abc\\.com/', selector={'type': 'url-regexp', 'value': 'abc\\.com'}, action='allow', options=[])
     ...
 
-In general `parse_filterlist` takes an iterable of strings (such as a list or
-an open file) and returns an iterable of parsed filter list lines. Each line
-will have its `.type` attribute set to a string indicating its type. It will
-also have a `.to_string()` method that converts it to a unicode string in the
-filter list format (most of the time it's the same as the string from which the
-filter was parsed). Further attributes depend on the type of the line.
-
-**Note:** `parse_filterlist` returns an iterator, not a list, and only consumes
-the input lines when its output is iterated over. This allows much more memory
-efficient handling of large filter lists, however there are two things to watch
-out for:
-
-**Note:** iteration over parsed lines may throw a `ParseError` exception if a
-line cannot be parsed. The exception will contain the information about the
-error and the original line that failed parsing.

[mathias, 2017/08/08 12:24:35]

It is not clear what bits this is about (I assume the generator that is function
parse_filterlist) and the following section/list seems a bit unrelated.

Also, as a side-note, I am not sure this is the desired behavior - analogous to
EmptyLine there should be an InvalidLine or something that contains an .error
reference to the exception object and the .raw string. Otherwise one would
always abort processing a list just because of something being allegedly
malformed, which could very well be something one just wants to skip over and
log about.

[Vasily Kuznetsov, 2017/08/08 14:31:12]

Yeah, we've discussed this. But for now that change is not in yet, so I'm
documenting the actual behavior.

-
-- When you're parsing filters from a file, you need to complete the iteration
-  before you close the file.
-- Once you iterate over the output of `parse_filterlist` once, it will be
-  consumed and you won't be iterate over it again.
-
-If you find that this is bothering you, you probably want to convert the output

[mathias, 2017/08/08 12:24:34]

Everything in this section from here on, maybe including the two items listed
above, seems terribly obvious after the first **Note:** earlier about
parse_filterlist() being a generator. Do you really think it is worth the effort
documenting that stuff in explicit fashion?

-of `parse_filterlist` to a list:
-
-    lines_list = list(parse_filterlist(filterlist))
-
-This will load the whole file into memory but unless you're dealing with a
-gigantic filter list that should not be a problem.
-
-### Line types
-
-As mentioned above, lines of different types have different attributes:
-
-| type       | attributes                                                             |

[mathias, 2017/08/08 12:24:35]

Are you sure this kind of table markup is supported by all the systems we want
to properly render the README.md file? That'll be news to me, I was under the
impression that this is basically a part of the "GitHub flavor" just loosely
adopted by some libraries. 

[Vasily Kuznetsov, 2017/08/08 14:31:12]

Indeed the table markup was not part of the original Markdown. However, it does
seem to be supported by all websites, tools and libraries that I've seen.

-|------------|------------------------------------------------------------------------|
-| header     | `version` - plugin version string                                      |
-| emptyline  | no options                                                             |
-| comment    | `text` - text of the comment                                           |
-| metadata   | `key` - name of the metadata field, `value` - value of the field       |
-| include    | `target` - url/path of the file to include                             |
-| filter     | `text` - text of the filter, `selector` - what to look for, `action` - what to do with selected items, `options` - filter options |
-
-#### Filter atributes

[mathias, 2017/08/08 12:24:35]

This section mentions "Selector" but not ".selector", "Action" but not
".action", etc. - hence it's not obvious to the reader what to do with that
information and what it is about. 

-
-Selector is a dictionary with two keys:
-
-| key          | meaning                                                          |
-|--------------|------------------------------------------------------------------|
-| type         | 'css', 'abp-simple', 'url-pattern', 'url-regexp', 'extended-css' |
-| value        | the selector itself, the meaning is type-dependent               |
-
-It's preferable to import `SELECTOR_TYPE` namespace from `abp.filters` to refer
-to filter types instead of using strings. `SELECTOR_TYPE` contains constants
-for each filter type: `SELECTOR_TYPE.CSS`, `SELECTOR_TYPE.ABP_SIMPLE`,
-`SELECTOR_TYPE.URL_PATTERN`, `SELECTOR_TYPE.URL_REGEXP` and
-`SELECTOR_TYPE.XCSS`.
-
-Action instructs adblocking software on what should be done with the items
-matching the selector:
-
-| action | meaning                                                                | 
-|--------|------------------------------------------------------------------------|
-| block  | block http(s) request that matches the selector                        |
-| allow  | allow http(s) request that matches the filter (whitelist the resource) |
-| hide   | hide the DOM element that matches the selector                         |
-| show   | show the DOM element that matches the selector (whitelist the element) |
-
-The action constants are contained in `FILTER_ACTION` namespace, which can also
-be imported from `abp.filters` (`FILTER_ACTION.BLOCK`, `FILTER_ACTION.ALLOW`,
-etc.)
-
-Options is a list of tuples consisting of option name and option value. The
-option value is `True` or `False` for flags or, for options with a value, it's
-a string, list of strings or a list of `(string, boolean)` tuples. See
-[documentation on authoring the filter rules][2] for the list of existing
-options and their meanings.
-
-### Other functions
-
 `abp.filters` module also exports a lower-level function for parsing individual
 lines of a filter list: `parse_line`. It returns a parsed line object just like
 the items in the iterator returned by `parse_filterlist`. 
+
+For further information on the library API use `help()` on `abp.filters` and
+its contents in interactive Python session, read the docstrings or look at the
+tests for some usage examples.
 
 ## Testing
 
 Unit tests for `python-abp` are located in the `/tests` directory.
 [Pytest][3] is used for quickly running the tests
 during development.
 [Tox][4] is used for testing in different
 environments (Python 2.7, Python 3.5+ and PyPy) and code quality
 reporting.
 
 In order to execute the tests, first create and activate development
 virtualenv:
 
     $ python setup.py devenv
     $ . devenv/bin/activate
 
 With the development virtualenv activated use pytest for a quick test run:
 
     (devenv) $ pytest tests
 
 and tox for a comprehensive report:
 
     (devenv) $ tox
 
+## Development
+
+When adding new functionality, add tests for it (preferably first). Code
+coverage (as measured by `tox -e qa`) should not decrease and the tests
+should pass in all Tox environments.
+
+All public functions, classes and methods should have docstrings compliant with
+[NumPy/SciPy documentation guide][5]. One exception is the constructors of
+classes that the user is not expected to instantiate (such as exceptions).
 
  [1]: https://adblockplus.org/filters#special-comments
  [2]: https://adblockplus.org/filters#options
  [3]: http://pytest.org/
  [4]: https://tox.readthedocs.org/
+ [5]: https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt