Issue 402 - Use a redirector script for downloads, not a direct link

sitescripts/extensions/web/downloads.py

 # coding: utf-8
 
 # This file is part of the Adblock Plus web scripts,
 # Copyright (C) 2006-2014 Eyeo GmbH
 #
 # Adblock Plus is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License version 3 as
 # published by the Free Software Foundation.
 #
 # Adblock Plus is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU General Public License
 # along with Adblock Plus.  If not, see <http://www.gnu.org/licenses/>.
 
 import re
 import time
 import posixpath
 import urlparse
-import multiprocessing
 import threading
+import traceback
 from ConfigParser import SafeConfigParser
 from sitescripts.web import url_handler
 from sitescripts.extensions.utils import getDownloadLinks
 
 links = {}
 UPDATE_INTERVAL = 10 * 60   # 10 minutes
 
 @url_handler('/latest/')
 def handle_request(environ, start_response):
   request = urlparse.urlparse(environ.get('REQUEST_URI', ''))
   basename = posixpath.splitext(posixpath.basename(request.path))[0]
   if basename in links:
     start_response('302 Found', [('Location', links[basename].encode("utf-8"))])
   else:
     start_response('404 Not Found', [])
   return []
 
-def update_links():

[Sebastian Noack, 2014/09/17 13:32:11]

Since you are apparently fine with keeping a thread around, why not just using a
thread instead a process (or a combination of both)?

This would be way simpler, and there isn't a real benefit of using a subprocess
(instead a thread) for the actual work here.

def _set_links():
  global links

  while True:
    links = _update_links()
    time.sleep(UPDATE_INTERVAL)
    
thread = threading.Thread(target=_set_links)
thread.start()

[Wladimir Palant, 2014/09/17 17:45:44]

That was the idea originally. However,
https://stackoverflow.com/a/2846695/785541 and other sources suggest that using
threads only makes sense for I/O operations in Python, any Python code running
still blocks the main thread.

[Sebastian Noack, 2014/09/17 17:53:42]

That is correct, but IMO not an issue here:

1. The reason why generating those links takes so long is only IO time.
2. As far as I know the server this is supposed to run on has only one CPU core
anyway.

As long as at least one of those assumptions hold true, there is no advantage of
subprocesses over threads. And even then it would be questionable if that
advantage justifies the added complexity.

[Sebastian Noack, 2014/09/17 18:13:53]

Note that CPU time is distributed across all threads, similar as when running
multiple processes with a single CPU. The main thread won't block until other
threads are finished.

-  try:
-    pool = multiprocessing.Pool(1)
-    pool.apply_async(_update_links, callback=_set_links)
-    pool.close()
-  finally:
-    t = threading.Timer(UPDATE_INTERVAL, update_links)
-    t.daemon = True
-    t.start()
-
-def _update_links():
+def _get_links():
   parser = SafeConfigParser()
   getDownloadLinks(parser)
   result = {}
   for section in parser.sections():
     result[section] = parser.get(section, "downloadURL")
   return result
 
-def _set_links(newlinks):
+def _update_links():
   global links
-  links = newlinks
 
-update_links()
+  while True:
+    try:
+      links = _get_links()
+    except:
+      traceback.print_exc()
+    time.sleep(UPDATE_INTERVAL)
+
+t = threading.Thread(target = _update_links)
+t.daemon = True
+t.start()