Issue 3775 - fix saving of requests

# abpcrawler project

Major points to be noted:  
- Each entry of "requests" array is not necessarily a real request. For each request issued from a whitelisted nested frame there are at least two entries, one for the real request with matching blocking filter and the second one is the frame location with the whitelisting filter matching that location.
- Now there is no "blocked" property of the request, because we cannot actually get such information because we don't know that request has been blocked because there is no any relation between entries for the request and for the whitelisting frame.

E.g. for the following page
root-page
- resource with block-me-URL
- whitelisted frame with frame-URL by exception-filter (@@||frame-URL-domain^$document)
--resource with block-me-URL

the stored requests are:
{block-me-URL, blocking-filter, contentType}
{block-me-URL, blocking-filter, contentType}
{frame-URL, filter:null, contentType:SUBDOCUMENT}
{frame-URL, filter:exception-filter, contentType:DOCUMENT}
{frame-URL, filter:exception-filter, contentType:DOCUMENT}
{frame-URL, filter:exception-filter, contentType:DOCUMENT}

[Wladimir Palant, 2016-03-14 19:50:43]

https://codereview.adblockplus.org/29338121/diff/29338122/lib/crawler.js
File lib/crawler.js (right):

https://codereview.adblockplus.org/29338121/diff/29338122/lib/crawler.js#newc...
lib/crawler.js:267: let requestNotifier = new
RequestNotifier(tab.linkedBrowser.outerWindowID, function({type, location,
filter}, scanComplete)
When the scan is completed the listener is called with null as first parameter -
this will produce an error right now. You need to check the parameter first,
before decomposing.

Style nit: while we didn't strictly enforce this rule here, please keep the code
reasonably within 80 characters.

https://codereview.adblockplus.org/29338121/diff/29338122/lib/crawler.js#newc...
lib/crawler.js:270: });
You need to shut down this notifier when you are done, just like the tab is
being released. Otherwise the content side won't be shut down - you'll leak
memory and produce unnecessary IPC messages.

[sergei, 2016-03-15 10:59:56]

https://codereview.adblockplus.org/29338121/diff/29338122/lib/crawler.js
File lib/crawler.js (right):

https://codereview.adblockplus.org/29338121/diff/29338122/lib/crawler.js#newc...
lib/crawler.js:267: let requestNotifier = new
RequestNotifier(tab.linkedBrowser.outerWindowID, function({type, location,
filter}, scanComplete)
> Style nit: while we didn't strictly enforce this rule here, please keep the
code
> reasonably within 80 characters.
Fixed, is the indentation correct now?

https://codereview.adblockplus.org/29338121/diff/29338122/lib/crawler.js#newc...
lib/crawler.js:267: let requestNotifier = new
RequestNotifier(tab.linkedBrowser.outerWindowID, function({type, location,
filter}, scanComplete)
On 2016/03/14 19:50:43, Wladimir Palant wrote:
> When the scan is completed the listener is called with null as first parameter
-
> this will produce an error right now. You need to check the parameter first,
> before decomposing.
Fixed.

However, I guess, it should be documented in `RequestNotifier` constructor and I
actually cannot reproduce it, how to reproduce it in crawler?

https://codereview.adblockplus.org/29338121/diff/29338122/lib/crawler.js#newc...
lib/crawler.js:270: });
On 2016/03/14 19:50:43, Wladimir Palant wrote:
> You need to shut down this notifier when you are done, just like the tab is
> being released. Otherwise the content side won't be shut down - you'll leak
> memory and produce unnecessary IPC messages.

Done. Thanks, overlooked it.

[Wladimir Palant, 2016-03-15 11:05:44]

LGTM

https://codereview.adblockplus.org/29338121/diff/29338122/lib/crawler.js
File lib/crawler.js (right):

https://codereview.adblockplus.org/29338121/diff/29338122/lib/crawler.js#newc...
lib/crawler.js:267: let requestNotifier = new
RequestNotifier(tab.linkedBrowser.outerWindowID, function({type, location,
filter}, scanComplete)
On 2016/03/15 10:59:56, sergei wrote:
> However, I guess, it should be documented in `RequestNotifier` constructor and
I
> actually cannot reproduce it, how to reproduce it in crawler?

Yes, it should be documented. As to reproducing - I don't know, but
RequestNotifier.onComplete() will trigger the listener with null parameter.