Issue 3176 - Add metadata to content blocker lists

sitescripts/content_blocker_lists/bin/generate_lists.py

 #!/usr/bin/env python
 # coding: utf-8
 
 # This file is part of Adblock Plus <https://adblockplus.org/>,
 # Copyright (C) 2006-2015 Eyeo GmbH
 #
 # Adblock Plus is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License version 3 as
 # published by the Free Software Foundation.
 #
@@ skipping 27 matching lines @@
                               stdout=devnull)
     else:
       subprocess.check_call(("hg", "clone", config["abp2blocklist_url"],
                              abp2blocklist_path), stdout=devnull)
     subprocess.check_call(("npm", "install"), cwd=abp2blocklist_path,
                           stdout=devnull)
 
 def download_filter_list(url):
   with closing(urllib2.urlopen(url)) as response:
     body = response.read()
-  version = re.search(r"^(?:[^[!])|!\s*Version:\s*(.+)$",

[Sebastian Noack, 2015/12/08 15:08:38]

Indeed, we can simply use re.search instead iterating over all matches.

However, note that with the regexp you use here begin of string is only matched
for the pattern before the alternation. So for example you would also match "!
foo ! Version: foo".

Also note that re.search() will return None, and therefore the .group() causes
an exception, if there is neither a Version header nor some filters in the list.

So this should rather be:

  match = re.search(r"^(?:(?:[^[!])|!\s*Version:\s*(.+)$)", body, re.MULTILINE)
  version = match and match.group(1)

[kzar, 2015/12/08 15:37:43]

Well it's true about the "! foo ! Version: foo" match, we're not expecting that
to happen but I guess checking for it can't harm. Done.

As for the other point, I don't think it's true that we're expecting the match
to be None. re.search() is returning a match for the version, or a match for a
line not starting with [ or !. In the latter case there is still a match object,
but .group(1) simply returns None as no capturing groups matched.

So the end result is we either get the version number or None, and the filter
list after the header is not searched.

[Sebastian Noack, 2015/12/08 15:45:51]

Well, theoretically the filter list can be empty. But it's probably not
important. LGTM.

+  version = re.search(r"^(?:[^[!])|^!\s*Version:\s*(.+)$",
                       body, re.MULTILINE).group(1)
   return body, url, version
 
 def generate_metadata(filter_lists, expires):
   metadata = OrderedDict((
     ("version", time.strftime("%Y%m%d%H%M", time.gmtime())),
     ("expires", expires),
     ("sources", [])
   ))
   for body, url, version in filter_lists:
@@ skipping 25 matching lines @@
 
   easylist = download_filter_list(config["easylist_url"])
   exceptionrules = download_filter_list(config["exceptionrules_url"])
 
   write_block_list([easylist],
                    config["easylist_content_blocker_path"],
                    config["easylist_content_blocker_expires"])
   write_block_list([easylist, exceptionrules],
                    config["combined_content_blocker_path"],
                    config["combined_content_blocker_expires"])