Issue 1170 - [adblockplus.org Anwiki to CMS migration] Migrate content

filters/get_subscriptions.py

+# This file is part of the Adblock Plus website,
+# Copyright (C) 2006-2015 Eyeo GmbH
+#
+# Adblock Plus is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# Adblock Plus is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Adblock Plus.  If not, see <http://www.gnu.org/licenses/>.
+
+import codecs
+import os
+import tarfile
+import urllib
+
+from sitescripts.subscriptions.subscriptionParser import parse_file, calculate_supplemented
+
+def get_subscriptions(_):
+  result = {}
+  utf8_reader = codecs.getreader('utf8')
+  source = urllib.urlopen("https://hg.adblockplus.org/subscriptionlist/archive/default.tar.gz")
+  try:
+    with tarfile.open(fileobj=source, mode="r|gz") as archive:
+      for fileinfo in archive:
+        if os.path.splitext(fileinfo.name)[1] != ".subscription":

[Wladimir Palant, 2015/02/27 15:37:02]

Actually, I'm unsure that this is really better than using endswith() - we don't
have a file system path here, rather a path within a tar archive which might
have different conventions (Unix-like paths regardless of operating system).

[Sebastian Noack, 2015/02/27 15:40:43]

AFAIK splitext() is implemented identically for all file systems. But if in
doubt you could still use posixpath.split().

[Sebastian Noack, 2015/02/27 15:41:18]

Operating systems, I meant.

[Wladimir Palant, 2015/02/27 16:19:58]

Sure, it is. However, this doesn't change the fact that here we are applying it
to something it isn't meant for - this isn't a file system path.

[Sebastian Noack, 2015/02/27 16:32:32]

As I said, strictly it would be more correct to use posixpath.splitext() here.
But IMO since splitext() is the same for all operating systems, one could argue
that importing posixpath just for this purpose isn't worth it. Let's leave it up
to Dave, what he prefers. I don't mind.

+          continue
+
+        filedata = parse_file(fileinfo.name, utf8_reader(archive.extractfile(fileinfo)))
+        if filedata.unavailable:
+          continue
+
+        result[filedata.name] = filedata
+  finally:
+    source.close()
+
+  calculate_supplemented(result)
+  return result.values()