Issue 5250 - Add copyright update script

update-copyright/update_copyright.py

 #!/usr/bin/env python3

[Sebastian Noack, 2017/07/05 15:46:53]

I suppose we should make this script executable (i.e. the mode above should be
755 instead of 644). Note that if the file is not executable, the shebang (i.e.
#!) here would be useless.

Once you made the script executable, you might also want to adapt the README to
instruct calling ./update_copyright.py instead of "python3 update_copyright.py",
which as a side effect removes some duplication.

[rosie, 2017/07/07 15:55:48]

Done.

 
 import os
 import subprocess
 import re
 import datetime
 import shutil
 import urllib.parse
 import urllib.request
 import html.parser
 import argparse
@@ skipping 39 matching lines @@
     finally:
         shutil.rmtree(repo, ignore_errors=True)
 
 
 def text_replace(dirpath, filename):
     with open(os.path.join(dirpath, filename), 'r+',
               encoding='utf-8', newline='') as file:
         try:
             text = file.read()
         except UnicodeDecodeError:
-            print("Error: Couldn't read {}{}".format(dirpath, filename))

[Sebastian Noack, 2017/07/05 15:46:53]

Failing silently (in the original code) was intended here, as this script is
expected to encounter many binary files (e.g. images). If we output an error
message every time, this will generate a huge amount of noise.

[rosie, 2017/07/07 15:55:49]

Done.

             return
 
         text = re.sub(
             r'(copyright.*?\d{4})(?:-\d{4})?\s+eyeo gmbh',
             r'\1-{} eyeo GmbH'.format(CURRENT_YEAR), text, 0, re.I
         )
         file.seek(0)
         file.write(text)
         file.truncate()
 
@@ skipping 35 matching lines @@
     def handle_endtag(self, tag):
         if tag == 'tr':
             self.recordtr = False
             self.cell = 0
 
     def handle_data(self, data):
         if self.cell == 2 and self.recordtr is True:
             self.recordtr = False
             self.cell = 0
             # Only process the URL if the description is not Deprecated
-            deprecated = (re.search(r'\*DEPRECATED\*', data) or
-                          re.search(r'(Deprecated)', data))

[Sebastian Noack, 2017/07/05 15:46:53]

This regular expression seems incorrect. The parentheses annotate a group. So it
won't only match substrings like "(Deprecated)" but any occurrence of
"Deprecated", storing the result in the 1st group of the match (which is
discarded here). If you want to match parantheses in the input string, you have
to escape them in the pattern:

  re.search(r'\(Deprecated\)', data)

Anyway, if all you want to do is to find a fixed substring, it would be much
simpler (and less error-prone) to just use the in-operator (instead of regular
expressions):

  '*DEPRECATED*' in data or '(Deprecated)' in data

[rosie, 2017/07/07 15:55:48]

Done.

-            if not deprecated and len(self.current_url) > 2:
+            if ('*DEPRECATED*' not in data and
+                    '(Deprecated)' not in data and
+                    len(self.current_url) > 2):
                 self.result += [self.current_url]
                 return self.result
 
 
 def extract_urls(hg_page):
     base_url = os.path.dirname(hg_page) + '/'
     parser = Parser()
     with urllib.request.urlopen(hg_page) as response:
         parser.feed(response.read().decode('utf-8'))
     parser.close()
     repo_urls = []
     for url in parser.result:
         repo_urls.append(urllib.parse.urljoin(base_url, url))
     return repo_urls
 
 
 def main(hg_page, hg_upstream):
     for repo in extract_urls(hg_page):
         process_repo(repo, hg_upstream)
 
 
 if __name__ == '__main__':
     arg_parser = argparse.ArgumentParser()
     arg_parser.add_argument('-u', '--hg-url',
-                            help='specify which Mercurial URL site to scrape',
-                            required=True)
+                            help='specify which Mercurial URL site to scrape')
     arg_parser.add_argument('-p', '--push-url',
-                            default=None,
                             help='specify where to push the repository')
     args = arg_parser.parse_args()
-    main(args.hg_url, args.push_url)
+    if args.hg_url and not args.push_url:
+        arg_parser.error('If -u is provided, -p is mandatory')
+    main(args.hg_url or 'https://hg.adblockplus.org/',
+         args.push_url or 'ssh://hg@hg.adblockplus.org/')