Issue 6685 - Offer incremental filter list downloads

abp/filters/renderer.py

Index: abp/filters/renderer.py
===================================================================
--- a/abp/filters/renderer.py
+++ b/abp/filters/renderer.py
@@ -24,7 +24,7 @@
 from .parser import parse_filterlist, Comment, Metadata
 from .sources import NotFound
 
-__all__ = ['IncludeError', 'MissingHeader', 'render_filterlist']
+__all__ = ['IncludeError', 'MissingHeader', 'render_filterlist', 'render_diff']
 
 _logger = logging.getLogger(__name__)
 
@@ -180,3 +180,51 @@
                  _validate]:
         lines = proc(lines)
     return lines
+
+
+def _split_list_for_diff(list_in):
+    filterlist, metadata, keys = set(), set(), set()
+    for line in parse_filterlist(list_in):
+        if line.type == 'metadata' and 'Checksum' not in line.to_string():

[Sebastian Noack, 2018/08/28 19:52:18]

I think it has been agreed on that this isn't necessary, as any diff-aware
client ignores checksums anyway (besides most filter list if not all wouldn't
have a checksum anymore anyway).

[rhowell, 2018/08/29 21:43:35]

Done.

+            metadata.add(line.to_string())
+            keys.add(line.key)

[Sebastian Noack, 2018/08/28 19:52:18]

This code has to be changed as well, to handle case-insensitive keys:

  def _split_list_for_diff(list_in):
      metadata = {}
      for line in parse_filterlist(list_in):
          if line.type == 'metadata':
              metadata[line.key.lower()] = line
      ...

  def render_diff(base, latest):
      ...
      for key, line in latest_md.items():
          other = base_md.get(key)
          if not other or other.value != line.value:
              yield line.to_string()
      for key in set(base_md) - set(latest_md):
          yield '! {}:'.format(base_md[key].key)
      ...

[rhowell, 2018/08/29 21:43:34]

Any reason to use a dict instead of a set? I guess one advantage would be: if a
metadata item is removed in the new list, we would convert it to lower-case and
give it a blank value. If that's a concern (converting it to lower case), then I
can use dicts instead of sets, as you suggested.

[Sebastian Noack, 2018/08/29 21:56:37]

The algorithm I suggest here, is the (probably) simplest way to treat keys
case-insensitive while preserving the original capitalization. With a set alone
you cannot associate the normalized keys with the original.

[rhowell, 2018/08/29 22:52:23]

Done.

+        elif line.type == 'filter':
+            filterlist.add(line.to_string())
+    return filterlist, metadata, keys
+
+
+def render_diff(base, latest):
+    """Return a diff between two filter lists.
+
+    Parameters
+    ----------
+    base : iterator of str
+        The base (old) list that we want to update to latest.
+    lastest : iterator  of str
+        The latest (most recent) list that we want to update to.
+
+    Returns
+    -------
+    iterable of str
+        A diff between two lists (https://issues.adblockplus.org/ticket/6685)
+
+    """
+    latest_fl, latest_md, latest_keys = _split_list_for_diff(latest)
+    base_fl, base_md, base_keys = _split_list_for_diff(base)
+
+    new_md = latest_md - base_md
+    removed_keys = base_keys - latest_keys
+    add_fl = latest_fl - base_fl
+    remove_fl = base_fl - latest_fl
+
+    yield '[Adblock Plus Diff]'
+    for item in new_md:
+        yield item
+    for key in removed_keys:
+        # If a special comment has been removed, enter it with a blank value
+        # so the client will set it back to the default value
+        yield '! {}:'.format(key)
+    for item in remove_fl:
+        yield '- {}'.format(item)
+    for item in add_fl:
+        yield '+ {}'.format(item)