Issue 402 - Use a redirector script for downloads, not a direct link

sitescripts/extensions/web/downloads.py

Index: sitescripts/extensions/web/downloads.py
===================================================================
new file mode 100644
--- /dev/null
+++ b/sitescripts/extensions/web/downloads.py
@@ -0,0 +1,63 @@
+# coding: utf-8
+
+# This file is part of the Adblock Plus web scripts,
+# Copyright (C) 2006-2014 Eyeo GmbH
+#
+# Adblock Plus is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# Adblock Plus is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Adblock Plus.  If not, see <http://www.gnu.org/licenses/>.
+
+import re
+import time
+import posixpath
+import urlparse
+import multiprocessing
+import threading
+from ConfigParser import SafeConfigParser
+from sitescripts.web import url_handler
+from sitescripts.extensions.utils import getDownloadLinks
+
+links = {}
+UPDATE_INTERVAL = 10 * 60   # 10 minutes
+
+@url_handler('/latest/')
+def handle_request(environ, start_response):
+  request = urlparse.urlparse(environ.get('REQUEST_URI', ''))
+  basename = posixpath.splitext(posixpath.basename(request.path))[0]
+  if basename in links:
+    start_response('302 Found', [('Location', links[basename].encode("utf-8"))])
+  else:
+    start_response('404 Not Found', [])
+  return []
+
+def update_links():

[Sebastian Noack, 2014/09/17 13:32:11]

Since you are apparently fine with keeping a thread around, why not just using a
thread instead a process (or a combination of both)?

This would be way simpler, and there isn't a real benefit of using a subprocess
(instead a thread) for the actual work here.

def _set_links():
  global links

  while True:
    links = _update_links()
    time.sleep(UPDATE_INTERVAL)
    
thread = threading.Thread(target=_set_links)
thread.start()

[Wladimir Palant, 2014/09/17 17:45:44]

That was the idea originally. However,
https://stackoverflow.com/a/2846695/785541 and other sources suggest that using
threads only makes sense for I/O operations in Python, any Python code running
still blocks the main thread.

[Sebastian Noack, 2014/09/17 17:53:42]

That is correct, but IMO not an issue here:

1. The reason why generating those links takes so long is only IO time.
2. As far as I know the server this is supposed to run on has only one CPU core
anyway.

As long as at least one of those assumptions hold true, there is no advantage of
subprocesses over threads. And even then it would be questionable if that
advantage justifies the added complexity.

[Sebastian Noack, 2014/09/17 18:13:53]

Note that CPU time is distributed across all threads, similar as when running
multiple processes with a single CPU. The main thread won't block until other
threads are finished.

+  try:
+    pool = multiprocessing.Pool(1)
+    pool.apply_async(_update_links, callback=_set_links)
+    pool.close()
+  finally:
+    t = threading.Timer(UPDATE_INTERVAL, update_links)
+    t.daemon = True
+    t.start()
+
+def _update_links():
+  parser = SafeConfigParser()
+  getDownloadLinks(parser)
+  result = {}
+  for section in parser.sections():
+    result[section] = parser.get(section, "downloadURL")
+  return result
+
+def _set_links(newlinks):
+  global links
+  links = newlinks
+
+update_links()