Issue 1093 - Separate update manifest generation

sitescripts/extensions/bin/updateUpdateManifests.py

 # coding: utf-8
 
 # This file is part of the Adblock Plus web scripts,
 # Copyright (C) 2006-2014 Eyeo GmbH
 #
 # Adblock Plus is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License version 3 as
 # published by the Free Software Foundation.
 #
 # Adblock Plus is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU General Public License
 # along with Adblock Plus.  If not, see <http://www.gnu.org/licenses/>.
 
 """
-Update update manifests

[Wladimir Palant, 2014/07/22 11:24:26]

Generate update manifests?

[Felix Dahlke, 2014/07/22 12:01:11]

Done.

-=======================
+Generate update manifests
+=========================
 
   This script generates update manifests for all extensions and apps
 """
 
+import re
+import subprocess
+from buildtools.packagerGecko import KNOWN_APPS
 from ConfigParser import SafeConfigParser
+from sitescripts.utils import get_config, get_template
 from sitescripts.extensions.utils import (Configuration, getDownloadLinks,
-  readMetadata)
-from sitescripts.utils import get_config, get_template
+                                          getSafariCertificateID)
+from sitescripts.extensions.android import get_min_sdk_version
+
+def readMetadata(repo, version):
+  """
+  reads extension ID and compatibility information from metadata file in the
+  extension's repository
+  """
+  if repo.type == 'android':
+    command = ['hg', '-R', repo.repository, 'id', '-r', version, '-n']
+    result = subprocess.check_output(command)
+    revision = re.sub(r'\D', '', result)
+
+    return {
+      'revision': revision,
+      'minSdkVersion': get_min_sdk_version(repo, version),
+    }
+  elif repo.type == 'safari':
+    metadata = repo.readMetadata(version)
+    return {
+      'certificateID': getSafariCertificateID(repo.keyFile),
+      'version': version,
+      'shortVersion': version,
+      'basename': metadata.get('general', 'basename'),
+    }
+  elif repo.type == 'gecko':
+    metadata = repo.readMetadata(version)
+    result = {
+      'extensionID': metadata.get('general', 'id'),
+      'version': version,
+      'compat': []
+    }
+    for key, value in KNOWN_APPS.iteritems():
+      if metadata.has_option('compat', key):
+        minVersion, maxVersion = metadata.get('compat', key).split('/')
+        result['compat'].append({'id': value, 'minVersion': minVersion, 'maxVersion': maxVersion})
+    return result
+  else:
+    raise Exception('unknown repository type %r' % repo.type)
 
 def writeUpdateManifest(links):

[Felix Dahlke, 2014/07/22 08:11:11]

This hasn't been changed, just moved.

Since I kept the logic as it is, we have to keep a few repositories in sync on
the update server (in addition to downloads): adblockplus, adblockpluschrome and
adblockplusandroid. I'm a bit undecided - shall we create a follow-up for
removing that requirement?

[Wladimir Palant, 2014/07/22 11:24:26]

Quite a few more actually: elemhidehelper, abpcustomization, abpwatcher etc.
How would you do that? You need parameters like compatibility info for the
update manifest. The only alternative is IMHO downloading
https://hg.adblockplus.org/adblockpluschrome/raw-file/1.8.3/metadata.chrome.
However, this path is dependent on the server software (e.g. will be
https://hg.adblockplus.org/adblockpluschrome/raw/1.8.3/metadata.chrome with
RhodeCode) and it also won't be simple (things like inheritance need to be
considered).

[Felix Dahlke, 2014/07/22 12:01:11]

It doesn't look to me like we're generating update manifests for those - do we? 
Alright, let's stick to syncing the repositories then. No real harm in that.

[Wladimir Palant, 2014/07/22 18:41:22]

We do, see https://adblockplus.org/update.rdf. Extension Auto-installer and JS
Deobfuscator should be removed, the rest of them has to stay however. In fact,
abpwatcher can only be updated via our servers, we don't host it on AMO.

   """
   writes an update manifest for all extensions and Android apps
   """
 
   extensions = {'gecko': [], 'android': [], 'safari': []}
   for repo in Configuration.getRepositoryConfigurations():
     if repo.type not in extensions or not links.has_section(repo.repositoryName):
       continue
     data = readMetadata(repo, links.get(repo.repositoryName, 'version'))
     data['updateURL'] = links.get(repo.repositoryName, 'downloadURL')
     if data['updateURL'].startswith(repo.downloadsURL):
       data['updateURL'] += "?update"
     extensions[repo.type].append(data)
 
   if len(extensions['android']) > 1:
     print >>sys.stderr, 'Warning: more than one Android app defined, update manifest only works for one'
 
   for repoType in extensions.iterkeys():
     manifestPath = get_config().get('extensions', '%sUpdateManifestPath' % repoType)
     template = get_template(get_config().get('extensions', '%sUpdateManifest' % repoType))
-    print extensions[repoType]

[Wladimir Palant, 2014/07/22 11:24:26]

Debug code?

[Felix Dahlke, 2014/07/22 12:01:11]

Ouch, done.

     template.stream({'extensions': extensions[repoType]}).dump(manifestPath)
 
 def updateUpdateManifests():
   """
   updates all update manifests with the current versions
   """
 
   parser = SafeConfigParser()
   getDownloadLinks(parser)
   writeUpdateManifest(parser)
 
 if __name__ == "__main__":
   updateUpdateManifests()