Issue 4916 - Request compression in CURL WebRequest.

src/DefaultWebRequestCurl.cpp

 /*
  * This file is part of Adblock Plus <https://adblockplus.org/>,
  * Copyright (C) 2006-2016 Eyeo GmbH
  *
  * Adblock Plus is free software: you can redistribute it and/or modify
  * it under the terms of the GNU General Public License version 3 as
  * published by the Free Software Foundation.
  *
  * Adblock Plus is distributed in the hope that it will be useful,
  * but WITHOUT ANY WARRANTY; without even the implied warranty of
@@ skipping 120 matching lines @@
 
   CURL *curl = curl_easy_init();
   if (curl)
   {
     std::stringstream responseText;
     HeaderData headerData;
     curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
     curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
     curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, ReceiveData);
     curl_easy_setopt(curl, CURLOPT_WRITEDATA, &responseText);
+    // Request compressed data. Using any supported aglorithm
+    curl_easy_setopt(curl, CURLOPT_ACCEPT_ENCODING, "");

[Felix Dahlke, 2017/03/02 14:33:38]

Won't it be possible to overwrite this by supplying the `Accept-Encoding` header
in `requestHeaders`? It shouldn't be possible for XHR users to overwrite this,
it's user agent specific after all.

In issue 4916, I suggested to blacklist the headers not allowed in XHR's
`setRequestHeader` function according to the spec, see:
https://codereview.adblockplus.org/29377775/diff/29377796/test/WebRequest.cpp...

I figure the easiest way is to reject those in lib/compat.js. I figure clients
can still overwrite it if they really want to, but it'll be pretty obvious that
this shouldn't be done.

I'm not sure if that should be tackled in the same issue however, feel free to
create a new one for that. I'm not all that worried about it.

[hub, 2017/03/02 15:36:00]

It is my understanding that passing "" will set the only valid values and make
CURL do the proper decoding of the response:
https://github.com/curl/curl/blob/a08db496429e09561c8b8d243e79b21ef04f98f2/do...

An override will cause:
1. no change if the encoding is supported by CURL
2. an error, it the encoding is not supported by CURL

Simply setting the Accept-Encoding header doesn't work because then we need to
handle the decoding out of CURL.

I think blacklisting is a different issue, because with or without this change
it should be done.

[Felix Dahlke, 2017/03/02 15:39:47]

Fair enough.

     curl_easy_setopt(curl, CURLOPT_HEADERFUNCTION, ReceiveHeader);
     curl_easy_setopt(curl, CURLOPT_HEADERDATA, &headerData);
 
     struct curl_slist* headerList = 0;
     for (HeaderList::const_iterator it = requestHeaders.begin();
       it != requestHeaders.end(); ++it)
     {
       headerList = curl_slist_append(headerList, (it->first + ": " + it->second).c_str());
     }
     if (headerList)
@@ skipping 30 matching lines @@
         }
       }
     }
 
     if (headerList)
       curl_slist_free_all(headerList);
     curl_easy_cleanup(curl);
   }
   return result;
 }