Issue 1170 - [adblockplus.org Anwiki to CMS migration] Migrate content

filters/get_subscriptions.py

 # This file is part of the Adblock Plus website,
 # Copyright (C) 2006-2015 Eyeo GmbH
 #
 # Adblock Plus is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License version 3 as
 # published by the Free Software Foundation.
 #
 # Adblock Plus is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU General Public License
 # along with Adblock Plus.  If not, see <http://www.gnu.org/licenses/>.
 
 import codecs
+import os
 import tarfile
 import urllib
 
 from sitescripts.subscriptions.subscriptionParser import parse_file, calculate_supplemented
 
 def get_subscriptions(_):

[Sebastian Noack, 2015/02/27 11:10:49]

I feel that we begin to misuse template filters. Do you know that in addition to
filters you can also register global functions with Jinja2:
http://jinja.pocoo.org/docs/dev/api/#jinja2.Environment.globals

However, our CMS currently only supports custom filters. But I wonder whether we
should add support for custom globals, for cases like this?

[Sebastian Noack, 2015/02/27 14:42:45]

Since Dave apparently doesn't have an opinion here, what do you think Wladimir?

[Wladimir Palant, 2015/02/27 15:37:02]

Sure, we are misusing the filters mechanism here, I am fully aware of that. And
we probably want to support custom global functions in the long term. However,
I'd rather not touch the CMS right now and finish the migration sooner - this
isn't the only hack we left around, and it's a comparably small one. I'm all for
filing a follow-up issue on this however.

[Sebastian Noack, 2015/02/27 15:40:43]

Sounds good to me.

   result = {}
+  utf8_reader = codecs.getreader('utf8')
   source = urllib.urlopen("https://hg.adblockplus.org/subscriptionlist/archive/default.tar.gz")

[Sebastian Noack, 2015/02/27 11:10:49]

The file-like object returned by urllib.urlopen() wants to be closed. ;)

[kzar, 2015/02/27 14:21:07]

Done.

-  with tarfile.open(fileobj=source, mode="r|gz") as archive:
-    for fileinfo in archive:
-      if not fileinfo.name.endswith(".subscription"):

[Sebastian Noack, 2015/02/27 11:10:49]

Nit: os.path.splitext(fileinfo.name)[1] == ".subscription"

[kzar, 2015/02/27 14:21:07]

Done.

-        continue
+  try:
+    with tarfile.open(fileobj=source, mode="r|gz") as archive:
+      for fileinfo in archive:
+        if os.path.splitext(fileinfo.name)[1] != ".subscription":
+          continue
 
-      filedata = parse_file(fileinfo.name, codecs.getreader('utf8')(archive.extractfile(fileinfo)))

[Sebastian Noack, 2015/02/27 11:10:49]

Nit: Maybe initializing the utf8 reader above the loop, instead calling it on
every iteration?

[kzar, 2015/02/27 14:21:07]

Done.

-      if filedata.unavailable:
-        continue
+        filedata = parse_file(fileinfo.name, utf8_reader(archive.extractfile(fileinfo)))
+        if filedata.unavailable:
+          continue
 
-      result[filedata.name] = filedata
+        result[filedata.name] = filedata
+  finally:
+    source.close()
 
   calculate_supplemented(result)
   return result.values()