1377 - Blacklist dependencies from SCM tracking

ensure_dependencies.py

 #!/usr/bin/env python
 # coding: utf-8
 
 # This file is part of the Adblock Plus build tools,
 # Copyright (C) 2006-2014 Eyeo GmbH
 #
 # Adblock Plus is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License version 3 as
 # published by the Free Software Foundation.
 #
 # Adblock Plus is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU General Public License
 # along with Adblock Plus.  If not, see <http://www.gnu.org/licenses/>.
 
 import sys
 import os
 import posixpath
 import re
 import io
 import errno
 import logging
 import subprocess
 import urlparse
+
 from collections import OrderedDict
+from ConfigParser import RawConfigParser
 
 USAGE = """
 A dependencies file should look like this:
 
   # VCS-specific root URLs for the repositories
   _root = hg:https://hg.adblockplus.org/ git:https://github.com/adblockplus/
   # File to update this script from (optional)
   _self = buildtools/ensure_dependencies.py
   # Check out elemhidehelper repository into extensions/elemhidehelper directory
   # at tag "1.2".
@@ skipping 21 matching lines @@
     # should simply return an empty string.
     result = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[0]
     return result.strip()
 
   def pull(self, repo):
     subprocess.check_call(["hg", "pull", "--repository", repo, "--quiet"])
 
   def update(self, repo, rev):
     subprocess.check_call(["hg", "update", "--repository", repo, "--quiet", "--check", "--rev", rev])
 
-  def ignore(self, target):
+  def ignore(self, target, repo):
+
     if not self.istype(target):
-      logging.warning("Mercurial won't ignore dependency %s by default. You should do it manually." % (target))

[Wladimir Palant, 2014/09/23 15:04:13]

Quite the opposite actually - Mercurial will automatically ignore other
repositories. So nothing needs to be done here, this method should simply say
"pass."

Btw, just for reference as this isn't necessary:
http://mercurial.selenic.com/wiki/TipsAndTricks#Ignore_files_in_local_working...

[mathias, 2014/09/23 16:20:05]

Mercurial does not ignore Git submodules (just verified that again). And it's
quite possible to have an HG root repo but a Git submodule, especially when
using both HG and Git together in the root repo (which is, again, quite common)
or just synchronizing repos, debugging dependencies or whatever.

We are aware that Mercurial ignores Mercurial repositories in the working tree
though, which is why the "if not self.istype(target)" condition is in there. But
I assume that the use of the word "dependency" is a bit misleading, and the "You
should..." part is overhead.

[Wladimir Palant, 2014/09/25 19:52:20]

You are right, so a local hgignore file should be the way to go here as well...

[mathias, 2014/09/26 05:06:39]

Thank you for the "local" keyword! I've been searching for something similar to
.git/info/exclude in HG, but couldn't find anything before trying it again with
that one in addition.. sometimes it may be too simple I guess :)

[mathias, 2014/09/28 00:24:25]

Done.

+
+      config_path = os.path.join(repo, ".hg", "hgrc")
+      ignore_path = os.path.abspath(os.path.join(repo, ".hg", "dependencies"))
+
+      config = RawConfigParser()
+      config.read(config_path)
+
+      if not config.has_section("ui"):
+        config.add_section("ui")
+
+      config.set("ui", "ignore.dependencies", ignore_path)
+      with open(config_path, "w") as stream:
+        config.write(stream)
+
+      module = os.path.relpath(target, repo)
+      _ensure_line_exists(ignore_path, module)
 
 class Git():
   def istype(self, repodir):
     return os.path.exists(os.path.join(repodir, ".git"))
 
   def clone(self, source, target):
     source = source.rstrip("/")
     if not source.endswith(".git"):
       source += ".git"
     subprocess.check_call(["git", "clone", "--quiet", source, target])
 
   def get_revision_id(self, repo, rev="HEAD"):
     command = ["git", "rev-parse", "--revs-only", rev]
     return subprocess.check_output(command, cwd=repo).strip()
 
   def pull(self, repo):
     subprocess.check_call(["git", "fetch", "--quiet", "--all", "--tags"], cwd=repo)
 
   def update(self, repo, rev):
     subprocess.check_call(["git", "checkout", "--quiet", rev], cwd=repo)
 
-  def ignore(self, target):
-    module = str(target).replace(os.getcwd(),'')[1:]

[Wladimir Palant, 2014/09/23 15:04:13]

There is a reason why this script never relies on the current work directory -
we don't know what the current work directory is. Even if you run
ensure_dependencies.py without parameters - it applies to the repository in
current work directory initially but it will recurse to other repositories after
that, located in other directories.

This function should have both repo and target as parameters. And a relative
path is being determined by calling os.path.relpath(), not by doing some
adventurous string replacements.

[mathias, 2014/09/23 16:20:05]

Agreed, will be addressed in the next patch-set.

[mathias, 2014/09/28 00:24:25]

Done.

-    exclude_file = '.git/info/exclude'
-    if os.path.exists(exclude_file):

[Wladimir Palant, 2014/09/23 15:04:13]

1) Please don't use relative file paths (see above).
2) Please don't assume Unix-style paths.

This should be:

  exclude_file = os.path.join(repo, ".git", "info", "exclude")

[mathias, 2014/09/23 16:20:05]

Agreed, will be addressed in the next patch-set.

[mathias, 2014/09/28 00:24:25]

Done.

-      with open(exclude_file, 'r') as f:
-        exclude_file_content = f.read().splitlines()

[Wladimir Palant, 2014/09/23 15:04:13]

I'd rather do [l.strip() for l in f.readlines()] here - this will remove other
whitespace as well, not just newlines.

[mathias, 2014/09/23 16:20:05]

Agreed, will be addressed in the next patch-set.

[mathias, 2014/09/28 00:24:25]

Done.

-        if not module in exclude_file_content:
-          logging.warning("Adding dependency %s to %s" % (module, exclude_file) )

[Wladimir Palant, 2014/09/23 15:04:13]

1) Why is this a warning? Nothing is wrong, a completely normal operation is
being performed - we call logging.info() for those.

2) This message is very misleading, what's being added here definitely isn't a
dependency.

3) The repository classes merely abstract away repository specifics, any
informational messages should be produced by the code using them.

Luckily, this message should no longer be necessary if this is called by
ensure_repo().

[mathias, 2014/09/23 16:20:05]

Agreed, we just remove the line entirely.

[mathias, 2014/09/28 00:24:25]

Done.

-          exclude_file_content = open(exclude_file, 'a')

[Wladimir Palant, 2014/09/23 15:04:13]

This will fail on Windows - you are opening a file for writing that is still
open for reading, file locking won't allow it. You should really only open this
file once, file mode "r+" - this will allow you overwriting the file contents
after reading it.

Bonus obfuscation points for overwriting a string variable with a file handle.

[mathias, 2014/09/23 16:20:05]

Agreed, will be addressed in the next patch-set.

[mathias, 2014/09/28 00:24:25]

Done.

-          exclude_file_content.write("\n"+module)

[Wladimir Palant, 2014/09/23 15:04:13]

What if there is already a line break at the end of the file? It's better to
overwrite the file completely - and make sure there is always a line break at
the end of file while at it.

  exclude_file.seek(0)
  exclude_file.truncate(0)
  exclude_file.write("\n".join(exclude_file_content) + "\n")

[mathias, 2014/09/23 16:20:05]

Agreed, will be addressed in the next patch-set.

[mathias, 2014/09/28 00:24:25]

Done.

-          exclude_file_content.close()

[Wladimir Palant, 2014/09/23 15:04:13]

If your code throws an exception that file will never be closed - that's why we
use the with statement.

[mathias, 2014/09/23 16:20:05]

Agreed, will be addressed as well.

[mathias, 2014/09/28 00:24:25]

Done.

-
-repo_types = {
-  "hg": Mercurial(),
-  "git": Git(),
-}
+  def ignore(self, target, repo):
+    module = os.path.relpath(target, repo)
+    exclude_file = os.path.join(repo, ".git", "info", "exclude")
+    _ensure_line_exists(exclude_file, module)
+
+repo_types = OrderedDict((
+  ("hg", Mercurial()),
+  ("git", Git()),
+))
 
 def parse_spec(path, line):
   if "=" not in line:
     logging.warning("Invalid line in file %s: %s" % (path, line))
     return None, None
 
   key, value = line.split("=", 1)
   key = key.strip()
   items = value.split()
   if not len(items):
@@ skipping 63 matching lines @@
   for key in roots:
     if key == parenttype or (key in repo_types and type is None):
       type = key
   if type is None:
     raise Exception("No valid source found to create %s" % target)
 
   url = urlparse.urljoin(roots[type], sourcename)
   logging.info("Cloning repository %s into %s" % (url, target))
   repo_types[type].clone(url, target)
 
+  for repo in repo_types.itervalues():
+    if repo.istype(parentrepo):
+      repo.ignore(target, parentrepo)
+
 def update_repo(target, revisions):
   type = get_repo_type(target)
   if type is None:
     logging.warning("Type of repository %s unknown, skipping update" % target)
     return
 
   if type in revisions:
     revision = revisions[type]
   elif "*" in revisions:
     revision = revisions["*"]
@@ skipping 26 matching lines @@
   if overrideroots is not None:
     config["_root"] = overrideroots
 
   for dir, revisions in config.iteritems():
     if dir.startswith("_") or revisions["_source"] in skipdependencies:
       continue
     target = safe_join(repodir, dir)
     ensure_repo(repodir, target, config.get("_root", {}), revisions["_source"])
     update_repo(target, revisions)
     resolve_deps(target, level + 1, self_update=False, overrideroots=overrideroots, skipdependencies=skipdependencies)
-    for repo in repo_types.itervalues():
-      if repo.istype(repodir):
-        repo.ignore(target)

[Wladimir Palant, 2014/09/23 15:04:13]

IMHO, this should be part of ensure_repo():

1) The repository type is known there, no need to call get_repo_type() again.

2) If we didn't create the repository, we shouldn't touch the configuration for
it.

3) The recursive call is being performed once we are finished processing a
repository - it should stay that way.

[mathias, 2014/09/23 16:20:05]

Valid point, we move it to `ensure_repo()`. Note, however, that
`get_repo_type()` is not called in this part, because it considers all repo
types (the parallel use thingy, again).

[mathias, 2014/09/28 00:24:25]

Done.

 
   if self_update and "_self" in config and "*" in config["_self"]:
     source = safe_join(repodir, config["_self"]["*"])
     try:
       with io.open(source, "rb") as handle:
         sourcedata = handle.read()
     except IOError, e:
       if e.errno != errno.ENOENT:
         raise
       logging.warning("File %s doesn't exist, skipping self-update" % source)
       return
 
     target = __file__
     with io.open(target, "rb") as handle:
       targetdata = handle.read()
 
     if sourcedata != targetdata:
       logging.info("Updating %s from %s, don't forget to commit" % (source, target))
       with io.open(target, "wb") as handle:
         handle.write(sourcedata)
       if __name__ == "__main__":
         logging.info("Restarting %s" % target)
         os.execv(sys.executable, [sys.executable, target] + sys.argv[1:])
       else:
         logging.warning("Cannot restart %s automatically, please rerun" % target)
 
+def _ensure_line_exists(path, pattern):
+  with open(path, 'a+') as f:
+    file_content = [l.strip() for l in f.readlines()]
+    if not pattern in file_content:
+      file_content.append(pattern)
+      f.seek(0, os.SEEK_SET)
+      f.truncate()
+      for l in file_content:
+        print >>f, l
+
 if __name__ == "__main__":
   logging.basicConfig(format='%(levelname)s: %(message)s', level=logging.INFO)
   repos = sys.argv[1:]
   if not len(repos):
     repos = [os.getcwd()]
   for repo in repos:
     resolve_deps(repo)