sitescripts: Collect unmatched filters

In order to collect all unmatched domain-specific filters, I import all domain-specific filters from easylist and associate them with requests send by the crawler.

As discussed, we will probably import all filters, not just domain-specific ones, from compiled lists in the future. However, since we've decided to put the crawler on hold for now, I think we should leave that part as it is and take it from there in a few months.

The part where I actually figure out which domain-specific filters are unused is still missing. But we should be able to extract that from the database, all the data is there.

I also used the opportunity to turn the extract_crawler_sites script into one that doesn't directly import the data into the database (import_sites) and made a few improvements.

[Wladimir Palant, 2012-12-13 16:16:23]

How about we agree that we don't want this change at the moment (particularly
the part putting filters into our database)? I guess Eric has other plans
anyway.

[Felix Dahlke, 2012-12-14 10:11:33]

On 2012/12/13 16:16:23, Wladimir Palant wrote:
> How about we agree that we don't want this change at the moment (particularly
> the part putting filters into our database)? I guess Eric has other plans
> anyway.

I'm not sure if the changes in 8492018 are going to work without some of these.
And even though Eric decided to disregard the backend for now, I'm pretty sure
we'll need something like this then.

[Wladimir Palant, 2012-12-14 16:39:58]

Frankly, I am unhappy with the general approach taken here - importing filters
into the database shouldn't be necessary. We only need them to run the
evaluation and we should get them directly from Mercurial or from the web server
when we do that. In fact, that's something I would have preferred for the list
of URLs already...

[Felix Dahlke, 2012-12-17 14:30:22]

On 2012/12/14 16:39:58, Wladimir Palant wrote:
> Frankly, I am unhappy with the general approach taken here - importing filters
> into the database shouldn't be necessary. We only need them to run the
> evaluation and we should get them directly from Mercurial or from the web
server
> when we do that.

Wouldn't it still make sense to keep this version around? How I see it, we will
need the code in import_filters.py. I believe it's easier to adopt this version
to call it directly rather than rewrite it.

> In fact, that's something I would have preferred for the list
> of URLs already...

My main reason for putting this in two steps was to have a more flexible data
source, especially for testing. Eric's plan of having the Crawler run
independently from the backend seems better though.

[Wladimir Palant, 2012-12-19 16:12:10]

On 2012/12/17 14:30:22, Felix H. Dahlke wrote:
> Wouldn't it still make sense to keep this version around? How I see it, we
will
> need the code in import_filters.py. I believe it's easier to adopt this
version
> to call it directly rather than rewrite it.

It will still be in this code review but I doubt that it makes sense to commit
this code to the repository...

> My main reason for putting this in two steps was to have a more flexible data
> source, especially for testing.

My original idea was having the list of URLs in the Mercurial repositories along
with the lists themselves - which would allow filter list authors do add/remove
URLs easily.

[Felix Dahlke, 2012-12-19 16:18:23]

On 2012/12/19 16:12:10, Wladimir Palant wrote:
> On 2012/12/17 14:30:22, Felix H. Dahlke wrote:
> > Wouldn't it still make sense to keep this version around? How I see it, we
> will
> > need the code in import_filters.py. I believe it's easier to adopt this
> version
> > to call it directly rather than rewrite it.
> 
> It will still be in this code review but I doubt that it makes sense to commit
> this code to the repository...
> 
> > My main reason for putting this in two steps was to have a more flexible
data
> > source, especially for testing.
> 
> My original idea was having the list of URLs in the Mercurial repositories
along
> with the lists themselves - which would allow filter list authors do
add/remove
> URLs easily.

I see. Well in that case we can just close this issue, it's unlikely we'll need
any of this code.