Issue 122 - Puppet ENC via Hiera

Vagrantfile

Index: Vagrantfile
===================================================================
--- a/Vagrantfile
+++ b/Vagrantfile
@@ -1,3 +1,5 @@
+require 'yaml'
+
 VAGRANTFILE_API_VERSION = "2"
 REPOSITORY_DIR = File.dirname(__FILE__)
 DEPENDENCY_SCRIPT = File.join(REPOSITORY_DIR, "ensure_dependencies.py")
@@ -8,77 +10,64 @@
   raise error
 end
 
-def define_standard_vm(config, host_name, ip)
+def define_standard_vm(config, host_name, ip, role=nil)
   config.vm.define host_name do |config|
+

[Felix Dahlke, 2015/03/09 23:14:51]

I'd rather not introduce extraneous whitespace here, also below. Let's keep
unrelated changes low.

     config.vm.box = 'precise64'
     config.vm.box_url = 'http://cloud-images.ubuntu.com/vagrant/precise/current/precise-server-cloudimg-amd64-vagrant-disk1.box'
     config.vm.host_name = "#{host_name}.adblockplus.org"
     config.vm.network :private_network, ip: ip
     config.vm.provider :virtualbox do |vb|
+
       vb.customize ["modifyvm", :id, "--cpus", 1]
 
       # Work around https://www.virtualbox.org/ticket/11649
       vb.customize ['modifyvm', :id, '--natdnshostresolver1', 'on']
+
+      setup_path = File.join(REPOSITORY_DIR, "hiera", "roles", "#{role}.yaml")
+      setup = YAML.load_file(setup_path) rescue {}

[Felix Dahlke, 2015/03/09 23:14:51]

How I see it, if we get an exception here, setup is gonna be an empty hash.
Won't the setup.fetch() call on the next line fail then? If so, I'd opt for
leaving the rescue out here - then we get a meaningful exception at least.

[mathias, 2015/03/16 12:07:11]

The next call works as well, because it also uses a default: Here
setup.fetch(.., default) is pretty similar to Python's {}.get(.., default) or
the .. rescue default before.

[Felix Dahlke, 2015/03/16 18:10:15]

But setup would be nill - can we call "fetch" on nil without exceptions?

[mathias, 2015/03/16 19:09:17]

It won't be nil. It is either the parsed YAML or an empty hash after the
load_file/rescue operation, and the subsequent fetch() also results in either a
YAML object or another, empty hash.

By the way, the load_file() operation failing happens all the time. We only have
3 roles integrated yet, and all the others trigger an exception here. And the
script does indeed continue as expected in these cases. Furthermore, only the
infraforumserver.yaml file features a "requirements" section so far - yet
monitoringserver.yaml and codereviewserver.yaml work perfectly fine.

However, I do agree that a future version should be more verbose here. E.g. when
the #{role}.yaml exists, but parsing fails, actually print an error message and
probably even exit..

[Wladimir Palant, 2015/03/16 19:09:42]

As you said yourself, setup would be {} in case of an exception rather than nil
- and {}.fetch("requirements", {}) will give you  {}, all fine.

[Felix Dahlke, 2015/03/17 03:48:25]

Right, missed that yesterday, works fine indeed.

+      requirements = setup.fetch("requirements", {})
+
+      requirements.each do |key, value|
+        vb.customize ['modifyvm', :id, "--#{key}", "#{value}"]
+      end
+
     end
 
+    # The repository location in the production system's puppet master
+    config.vm.synced_folder ".", "/etc/puppet/infrastructure"
+
     config.vm.provision :shell, :inline => '
-if ! test -f /usr/bin/puppet; then
-  sudo apt-get update && sudo apt-get install -y puppet
-fi'
+      sudo /etc/puppet/infrastructure/hiera/install-precise.py

[Felix Dahlke, 2015/03/09 23:14:51]

Bit of a nit, but we're using underscores in Python scripts fairly consistently.
Mandatory for modules (PEP8), but not strictly speaking for standalone scripts.
Still, we don't have a single one using dashes, lets stay consistent.

[mathias, 2015/03/16 12:07:11]

Done.

+    '
 
-    manifest_files = ['vagrant.pp', 'nodes.pp']
-    manifest_files.each do |manifest_file|
-      config.vm.provision :puppet do |puppet|
-        puppet.options = ['--environment=development']
-        puppet.manifests_path = 'manifests'
-        puppet.manifest_file = manifest_file
-        puppet.module_path = 'modules'
-      end
+    config.vm.provision :puppet do |puppet|
+      puppet.options = [
+        '--environment=development',
+        '--external_nodes=/etc/puppet/infrastructure/hiera/puppet-node-classifier.rb',

[Felix Dahlke, 2015/03/09 23:14:51]

While we don't have that many Ruby scripts so far, I'd opt for keeping the
naming consistent with our Python scripts (i.e. underscores) - unless there's a
reason against it.

[mathias, 2015/03/16 12:07:11]

Done.

+        '--node_terminus=exec',
+        '--verbose',
+        '--debug',
+      ]
+      puppet.manifests_path = 'manifests'
+      puppet.manifest_file = 'nodes.pp'
+      puppet.module_path = 'modules'
+      # Requires Puppet 3.x or later
+      #puppet.hiera_config_path = 'hiera/vagrant.yaml'

[Felix Dahlke, 2015/03/09 23:14:51]

I'd rather remove this - not a fan of commented out code, and we'd also have to
remove the --external_nodes line above anyway. I'd rather have this in an issue
about switching to Puppet 3.x (which I really get the impression is something we
want to do).

     end
 
     yield(config) if block_given?
+
   end
 end
 
 Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
-  define_standard_vm config, 'server4', '10.8.0.99'
-  define_standard_vm config, 'server5', '10.8.0.100'
-  define_standard_vm config, 'server6', '10.8.0.101'
-  define_standard_vm config, 'server7', '10.8.0.102'
-  define_standard_vm config, 'server10', '10.8.0.105' do |config|
-    config.vm.provider :virtualbox do |vb|
-      vb.customize ["modifyvm", :id, "--memory", 1024]
-    end
+  config_path = File.join(REPOSITORY_DIR, "hiera/private/hosts.yaml")
+  config_data = YAML.load_file(config_path)
+  servers = config_data["servers"]
+  servers.each do |server, items|
+    ip = items["ip"][0]
+    role = items.fetch("role", "default")
+    define_standard_vm(config, server, ip, role)
   end
-  define_standard_vm config, 'server11', '10.8.0.106'
-  define_standard_vm config, 'server12', '10.8.0.107'
-  define_standard_vm config, 'server15', '10.8.0.110'
-  define_standard_vm config, 'server19', '10.8.0.114'
-  define_standard_vm config, 'notification1', '10.8.0.118'
-  define_standard_vm config, 'notification2', '10.8.0.119'
-  define_standard_vm config, 'filter1', '10.8.0.120'
-  define_standard_vm config, 'filter2', '10.8.0.121'
-  define_standard_vm config, 'filter3', '10.8.0.122'
-  define_standard_vm config, 'filter4', '10.8.0.123'
-  define_standard_vm config, 'filter5', '10.8.0.124'
-  define_standard_vm config, 'filter6', '10.8.0.125'
-  define_standard_vm config, 'download1', '10.8.0.126'
-  define_standard_vm config, 'filtermaster1', '10.8.0.127'
-  define_standard_vm config, 'update1', '10.8.0.128'
-  define_standard_vm config, 'web1', '10.8.0.129'
-  define_standard_vm config, 'stats1', '10.8.0.130'
-  define_standard_vm config, 'issues1', '10.8.0.131'
-  define_standard_vm config, 'codereview1', '10.8.0.132'
-  define_standard_vm config, 'filter7', '10.8.0.133'
-  define_standard_vm config, 'filter8', '10.8.0.134'
-  define_standard_vm config, 'filter9', '10.8.0.135'
-  define_standard_vm config, 'filter10', '10.8.0.136'
-  define_standard_vm config, 'filter11', '10.8.0.137'
-  define_standard_vm config, 'filter12', '10.8.0.138'
-  define_standard_vm config, 'filter13', '10.8.0.139'
-  define_standard_vm config, 'filter14', '10.8.0.140'
-  define_standard_vm config, 'filter15', '10.8.0.141'
-  define_standard_vm config, 'filter16', '10.8.0.142'
-  define_standard_vm config, 'filter17', '10.8.0.143'
-  define_standard_vm config, 'filter18', '10.8.0.144'
 end