Centralize stats processing, have the stats server pull in logs

Centralize stats processing, have the stats server pull in logs

[Wladimir Palant, 2013-12-20 14:49:18]

Uploaded new patchset to fix two more issues:

* We should use standard json rather than simplejson
* Invalid IP addresses (from corrupted log files) shouldn't stop processing

[Sebastian Noack, 2013-12-20 15:48:36]

http://codereview.adblockplus.org/5182947690807296/diff/5685265389584384/site...
File sitescripts/stats/bin/logprocessor.py (right):

http://codereview.adblockplus.org/5182947690807296/diff/5685265389584384/site...
sitescripts/stats/bin/logprocessor.py:44: match =
re.search(r"^ssh://(\w+)@([^/:]+)(?::(\d+))?/([^/]+)", path)
You can use urlparse (from the built-in module "urlparse").

[Wladimir Palant, 2013-12-22 16:05:06]

I fixed various issues in addition to the one mentioned in review:

* We shouldn't e running on a single CPU core now that we have a quad-core
server. This part is suboptimal right now, it should be solvable without having
to send a huge chunk of data back to the main process.
* Using the built-in gzip module meant wasting lots of memory because it doesn't
support streaming.
* The check for numerical data types was wrong (this didn't cause any issues
however).

What I am currently still unable to figure out: somehow the data gets corrupted
on server, with json.dump() saving the same dict key twice (once with its
original value and once with the modified value).

[Wladimir Palant, 2013-12-22 22:04:36]

Finally fixed the remaining issue as well. JSON converts all dict keys to
strings - and some of our dicts were using numbers or booleans as keys. So
merging them with JSON data produced weird results.

[Sebastian Noack, 2013-12-23 10:01:41]

http://codereview.adblockplus.org/5182947690807296/diff/6031560012726272/site...
File sitescripts/stats/bin/logprocessor.py (right):

http://codereview.adblockplus.org/5182947690807296/diff/6031560012726272/site...
sitescripts/stats/bin/logprocessor.py:512: pool = multiprocessing.Pool()
You should call pool.close(), when you are done.

[Wladimir Palant, 2013-12-23 21:43:37]

The new patch set is closing the pool after use now. It also adds an option to
subtract log data from the database in case a lot was applied mistakenly or
twice (unfortunately we need that option right now).

[Sebastian Noack, 2013-12-24 10:08:42]

http://codereview.adblockplus.org/5182947690807296/diff/5110435556622336/site...
File sitescripts/stats/bin/logprocessor.py (right):

http://codereview.adblockplus.org/5182947690807296/diff/5110435556622336/site...
sitescripts/stats/bin/logprocessor.py:519: pool.close()
You should put the code above into a try-finally block.

[Wladimir Palant, 2013-12-26 14:11:46]

I fixed that review comment. I also fixed an issue we had with processing a
specific log file - non-Unicode keys could not be saved (originally this was
throwing in json.dump(), now the exception moved to our own string conversion of
keys). For reference, the data in question came from SQL injection attempts.

[Sebastian Noack, 2013-12-26 14:44:55]

LGTM