Issue 4977 - Request for compressed response in AndroidWebRequest

libadblockplus-android/src/org/adblockplus/libadblockplus/android/AndroidWebRequest.java

 /*
  * This file is part of Adblock Plus <https://adblockplus.org/>,
  * Copyright (C) 2006-2016 Eyeo GmbH
  *
  * Adblock Plus is free software: you can redistribute it and/or modify
  * it under the terms of the GNU General Public License version 3 as
  * published by the Free Software Foundation.
  *
  * Adblock Plus is distributed in the hope that it will be useful,
  * but WITHOUT ANY WARRANTY; without even the implied warranty of
  * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
  * GNU General Public License for more details.
  *
  * You should have received a copy of the GNU General Public License
  * along with Adblock Plus.  If not, see <http://www.gnu.org/licenses/>.
  */
 
 package org.adblockplus.libadblockplus.android;
 
 import java.io.BufferedReader;
+import java.io.InputStream;
 import java.io.InputStreamReader;
 import java.net.HttpURLConnection;
 import java.net.URL;
 import java.util.HashSet;
 import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
+import java.util.zip.GZIPInputStream;
 
 import org.adblockplus.libadblockplus.AdblockPlusException;
 import org.adblockplus.libadblockplus.FilterEngine;
 import org.adblockplus.libadblockplus.HeaderEntry;
 import org.adblockplus.libadblockplus.ServerResponse;
 import org.adblockplus.libadblockplus.ServerResponse.NsStatus;
 import org.adblockplus.libadblockplus.WebRequest;
 
 import android.util.Log;
 
 public class AndroidWebRequest extends WebRequest
 {
+  protected static final String ENCODING_GZIP = "gzip";
+  protected static final String ENCODING_IDENTITY = "identity";
+
   public final static String TAG = Utils.getTag(WebRequest.class);
 
   private final HashSet<String> subscriptionURLs = new HashSet<String>();
   private final boolean elemhideEnabled;
+  private final boolean compressedStream;
 
-  public AndroidWebRequest(boolean enableElemhide)
+  /**
+   * Ctor
+   * @param enableElemhide Enable element hiding?
+   *                       Element hiding requires significantly more memory
+   *                       but allows better ad blocking
+   * @param compressedStream Request for gzip compressed stream from the server
+   */
+  public AndroidWebRequest(boolean enableElemhide, boolean compressedStream)

[Felix Dahlke, 2017/03/27 14:13:16]

I don't really see a use case for passing `compressedStream` as `false` - why
not always compress?

[anton, 2017/03/27 18:56:13]

The server can not support it or some or compressing takes more battery - who
knows why it can be undesired.
In ctor with no arguments i use `true` it seems to be the best by default.

[anton, 2017/03/28 07:18:54]

* decompressing (on user's device)

[Felix Dahlke, 2017/03/28 08:52:00]

If the server doesn't support compression it'll just return an uncompressed
response. But yeah, maybe clients will want to overwrite this for reasons we
cannot think of, fine by me if we leave it like this.

   {
     this.elemhideEnabled = enableElemhide;
+    this.compressedStream = compressedStream;
   }
 
   public AndroidWebRequest()
   {
-    this(false);
+    this(false, true);
   }
 
   private boolean isListedSubscriptionUrl(final URL url)
   {
     String toCheck = url.toString();
 
     final int idx = toCheck.indexOf('?');
     if (idx != -1)
     {
       toCheck = toCheck.substring(0, idx);
@@ skipping 14 matching lines @@
   @Override
   public ServerResponse httpGET(final String urlStr, final List<HeaderEntry> headers)
   {
     try
     {
       final URL url = new URL(urlStr);
       Log.d(TAG, "Downloading from: " + url);
 
       final HttpURLConnection connection = (HttpURLConnection) url.openConnection();
       connection.setRequestMethod("GET");
+      connection.setRequestProperty("Accept-Encoding",
+        (compressedStream ? ENCODING_GZIP : ENCODING_IDENTITY));

[Felix Dahlke, 2017/03/27 14:13:16]

Why not just leave the `Accept-Encoding` header out if `compressedStream` is
false? That's how the behaviour was before.

[anton, 2017/03/27 18:56:13]

Not true sometimes, previously it depended on android version:
https://android-developers.googleblog.com/2011/09/androids-http-clients.html
"In Gingerbread, we added transparent response compression"

So i decided to add ENCODING_IDENTITY if the compressed stream is not desired
for some reason.

[Felix Dahlke, 2017/03/28 08:52:00]

Acknowledged.

       connection.connect();
 
       final ServerResponse response = new ServerResponse();
       response.setResponseStatus(connection.getResponseCode());
 
       if (response.getResponseStatus() == 200)
       {
-        final BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream(), "UTF-8"));
+        final InputStream inputStream =
+          (compressedStream && ENCODING_GZIP.equals(connection.getContentEncoding())
+            ? new GZIPInputStream(connection.getInputStream())
+            : connection.getInputStream());
+        final BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream, "UTF-8"));
         final StringBuilder sb = new StringBuilder();
 
         String line;
         while ((line = reader.readLine()) != null)
         {
           // We're only appending non-element-hiding filters here.
           //
           // See:
           //      https://issues.adblockplus.org/ticket/303
           //
@@ skipping 35 matching lines @@
       Log.d(TAG, "Downloading finished");
       return response;
     }
     catch (final Throwable t)
     {
       Log.e(TAG, "WebRequest failed", t);
       throw new AdblockPlusException("WebRequest failed", t);
     }
   }
 }