Implemented extraction of URL Fixer data

Implemented extraction of URL Fixer data

[Thomas Greiner, 2012-11-28 10:05:42]

http://codereview.adblockplus.org/8943045/diff/1/sitescripts/urlfixer/bin/top...
File sitescripts/urlfixer/bin/topDomains.py (right):

http://codereview.adblockplus.org/8943045/diff/1/sitescripts/urlfixer/bin/top...
sitescripts/urlfixer/bin/topDomains.py:41: re.search(r"['\"_,<>;]|^\.|\.$|\.\.",
domain)):
That's a small selection of special characters. It will still accept URLs such
as "www.a$%&§b.c0m". A whitelist should be able to do a better job at that.

My recommendation:
not re.search(r"^([\da-z\.-]+)\.([a-z\.]{2,6})$", domain)

http://codereview.adblockplus.org/8943045/diff/1/sitescripts/urlfixer/bin/top...
sitescripts/urlfixer/bin/topDomains.py:58: curr_year * 0.2 + prev_year * 0.1 AS
result
|result| is not very descriptive here. Please rename. (e.g. |weighted_count|)

http://codereview.adblockplus.org/8943045/diff/1/sitescripts/urlfixer/bin/top...
sitescripts/urlfixer/bin/topDomains.py:65: return result["result"]
|result["result"]| is not very descriptive here. Please rename. (e.g.
|result["weighted_score"]|)

[Wladimir Palant, 2012-11-28 10:49:24]

http://codereview.adblockplus.org/8943045/diff/1/sitescripts/urlfixer/bin/top...
File sitescripts/urlfixer/bin/topDomains.py (right):

http://codereview.adblockplus.org/8943045/diff/1/sitescripts/urlfixer/bin/top...
sitescripts/urlfixer/bin/topDomains.py:31: def getTopDomains(count=1000):
Note that this is count=5000 on the server currently. Ideally we will soon
increase it to 10000, provided our data sample is good enough.

http://codereview.adblockplus.org/8943045/diff/1/sitescripts/urlfixer/bin/top...
sitescripts/urlfixer/bin/topDomains.py:41: re.search(r"['\"_,<>;]|^\.|\.$|\.\.",
domain)):
What about international domain names? I explicitly didn't want to reject
something that might be valid, that's why I limited it to a few things I noticed
in our data.

[Thomas Greiner, 2012-11-28 11:17:39]

http://codereview.adblockplus.org/8943045/diff/1/sitescripts/urlfixer/bin/top...
File sitescripts/urlfixer/bin/topDomains.py (right):

http://codereview.adblockplus.org/8943045/diff/1/sitescripts/urlfixer/bin/top...
sitescripts/urlfixer/bin/topDomains.py:31: def getTopDomains(count=1000):
Ok.

http://codereview.adblockplus.org/8943045/diff/1/sitescripts/urlfixer/bin/top...
sitescripts/urlfixer/bin/topDomains.py:41: re.search(r"['\"_,<>;]|^\.|\.$|\.\.",
domain)):
Ok. For now I think it would be sufficient to check for special characters from
the ASCII character set so that we at least detect $, %, & and such as well.

We should resume that discussion when we include the filters in URL Fixer which
might result in sending a different version number to the server.

[Thomas Greiner, 2012-11-28 12:23:45]

LGTM