Issue 4767 - Improve error reporting in update_update_manifests

sitescripts/extensions/utils.py

 # This file is part of the Adblock Plus web scripts,
 # Copyright (C) 2006-2016 Eyeo GmbH
 #
 # Adblock Plus is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License version 3 as
 # published by the Free Software Foundation.
 #
 # Adblock Plus is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU General Public License
 # along with Adblock Plus.  If not, see <http://www.gnu.org/licenses/>.
 
 import codecs
 import os
 import json
 import re
 import subprocess
 import traceback
 import time
 import urlparse
 import urllib
 import xml.dom.minidom as dom
 from ConfigParser import SafeConfigParser, NoOptionError
 from StringIO import StringIO
 from sitescripts.utils import get_config
+from xml.parsers.expat import ExpatError
 
 PACKAGE_SUFFIXES = {
     'gecko': '.xpi',
     'gecko-webext': '.xpi',
     'chrome': '.crx',
     'safari': '.safariextz',
     'ie': '.exe',
     'android': '.apk'
 }
 
@@ skipping 274 matching lines @@
     """
     gets the download links for all extensions and puts them into the config
     object
     """
     for repo in Configuration.getRepositoryConfigurations():
         try:
             (downloadURL, version) = _getDownloadLink(repo)
             if downloadURL is None:
                 raise Exception('No download link found for repo: ' +
                                 repo.repositoryName)
+        except ExpatError:
+            traceback.print_exc()
+            print "Error found while parsing xml from %s link" % repo.repositoryName

[Jon Sonesen, 2017/01/09 07:48:22]

Here you should be using the .format() method for string formatting as per our
style guidelines. Guideline text:

 "Use the + operator when concatenating exactly two strings, use the format()
method for more complex string formatting, use the join() method when
concatenating pre-existing sequences."

[Jon Sonesen, 2017/01/09 08:44:10]

Oops, I also just noticed that you are using double quotes. We are using single
quotes so that string literals match the repr 

[f.nicolaisen, 2017/01/10 08:58:04]

This adds some context (i.e. which repo) to the error, which is good.

However, in the case of the stacktrace shown in the issue, it is a special case
for _getMozillaDownloadLink - meaning that some external service is returning
something we cannot parse:

    ExpatError: mismatched tag: line 6, column 2

I'm guessing that the mozilla service returns something different than we
expect. I would like to know what this is, so that we know more about what
action to take next time it occurs. For example, maybe they return an xml
payload saying <response msg="This service is deprecated, please switch to using
http://blabla.mozilla.org">.

Perhaps it is even possible to test cover this by mocking the external call to
deliver something unexpected like that (ref test_updateManifests.py).

However, I think with these considerations we are going outside the scope of
'infrastructure' and deeper into sitescripts territory. @Vasily/Jon is this
something you want to adopt and roll out in the near future?

+            continue
         except:
             traceback.print_exc()
             continue
         if not result.has_section(repo.repositoryName):
             result.add_section(repo.repositoryName)
         result.set(repo.repositoryName, 'downloadURL', downloadURL)
         result.set(repo.repositoryName, 'version', version)
 
         qrcode = _getQRCode(downloadURL)
         if qrcode is not None:
@@ skipping 46 matching lines @@
     if not extensions:
         return
 
     updates = {}
     for extension in extensions:
         updates[extension['basename']] = {
             'url': extension['updateURL'],
             'version': extension['version']
         }
     writeLibabpUpdateManifest(path, updates)