Crawler backend

sitescripts/crawler/bin/extract_crawler_sites.py

 # coding: utf-8
 
 # This Source Code is subject to the terms of the Mozilla Public License
 # version 2.0 (the "License"). You can obtain a copy of the License at
 # http://mozilla.org/MPL/2.0/.
 
-import os, re, subprocess
+import MySQLdb, os, re, subprocess
 from sitescripts.utils import get_config
 
 def hg(args):
   return subprocess.Popen(["hg"] + args, stdout = subprocess.PIPE)
 
 def extract_urls(filter_list_dir):
   os.chdir(filter_list_dir)
   process = hg(["log", "--template", "{desc}\n"])
   urls = set([])
 
-  while True:
-    line = process.stdout.readline()
-    if line == "":

[Wladimir Palant, 2012/09/14 17:24:18]

Is this really a good break condition? An empty commit line is unlikely but
possible... Anyway, how about "for line in process.stdout:" instead of a while
loop?

[Felix Dahlke, 2012/09/26 15:20:30]

Done. An empty commit line would be "\n", but you're definitely right that for
... in is nicer.

-      break
-
-    matches = re.match(r"[A-Z]:.*(https?://.*)", line)

[Wladimir Palant, 2012/09/14 17:24:18]

What if we have some additional text following the URL? Or an inexperienced
EasyList author deviates from the "A:" format? I would prefer something more
generic like r"\bhttps?://\S+", maybe even with an additional .strip(",.!?")
below.

[Felix Dahlke, 2012/09/26 15:20:30]

Done.

-    if not matches:
+  for line in process.stdout:
+    match = re.search(r"\b(https?://\S*)", line)
+    if not match:
       continue
 
-    url = matches.group(1).strip()
+    url = match.group(1).strip()
     urls.add(url)
 
   return urls
 
 def print_statements(urls):
   for url in urls:
-    print "INSERT INTO crawler_sites (url) VALUES ('" + url + "');"

[Wladimir Palant, 2012/09/14 17:24:18]

While this might not be a big issue here, failing to escape values in SQL
statements is a bad habit. Please add escaping or - better - eliminate the SQL
file as intermediate step. Just add the URLs to the database immediately using
prepared statements.

[Felix Dahlke, 2012/09/26 15:20:30]

Done. I'd rather stick to generating a file with statements. More convenient to
test/verify, and we can keep old files for regression testing.

+    escaped_url = MySQLdb.escape_string(url)
+    print "INSERT INTO crawler_sites (url) VALUES ('" + escaped_url + "');"
 
 if __name__ == "__main__":
   filter_list_dir = get_config().get("crawler", "filter_list_repository")
   urls = extract_urls(filter_list_dir)
   print_statements(urls)