Issue 122 - Puppet ENC via Hiera

README.md

 Adblock Plus infrastructure
 ===========================
 
 The Adblock Plus infrastructure uses [Puppet](http://puppetlabs.com/)
 to set up servers, and to have a realistic development environment.
 
 Our Puppet manifests are only tested with Ubuntu 12.04 right now.
 
-Private files
--------------
+Host specific setup

[Felix Dahlke, 2015/03/03 16:43:32]

Is that really "Host specific"? You're calling servers "hosts" below, so IMO
this is a bit confusing. Maybe something like "Environment specific setup"? Or
just "Private files" or maybe "Private resources"? :)

[mathias, 2015/03/04 12:32:37]

Done.

+-------------------
 
-Some parts of our infrastructure are, obviously, confidential. We have
-htpasswd files, SSH keys and SSL certificates that we need to be
-careful with.
+Some infrastructure parts are specific to the environment (such as e.g.
+*development*, *test* and *production*) whilst passwords, for example,
+are confidential. In order to allow for such specific configuration, the
+repository requires a set of manual operations during the initial setup:
 
-That's why _modules/private_ is missing, and needs to be placed there
-manually. We provide stub versions of all those files in
-_modules/private-stub_, so just linking or copying that to
-_modules/private_ will make everything work locally.
+### `modules/private`
+
+The `private` module is destined to store confidential information such as
+[RSA](http://en.wikipedia.org/wiki/RSA_%28cryptosystem%29) keys, `htpasswd`
+files and so on. The repository provides a `private-stub` module containing
+defaults suitable for development and testing purpose. One can create a
+symbolic link to start using the resource:
+
+    # UNIX-oid

[Felix Dahlke, 2015/03/03 16:43:32]

Why the "-old" here?

Also, I would go for something like a #### headline instead of code comments
here. Also because "#" doesn't introduce a comment in batch scripts, so it
doesn't really make sense for the Windows part below.

[Wladimir Palant, 2015/03/03 20:00:19]

I guess using "Unix-like" will avoid that confusion...

[mathias, 2015/03/04 12:32:37]

Done.

+    user@host:~/infrastructure$ ln -s private-stub modules/private

[Felix Dahlke, 2015/03/03 16:43:32]

I'd opt for just having `$ ln -s private-stub modules/private` - the rest of the
prompt doesn't matter, and it's quite customary to leave it out.

[mathias, 2015/03/04 12:32:37]

Done.

+
+    # Windows
+    C:\infrastructure\> MKLINK /D modules\private private-stub
+
+When creating a custom version, one may inspect the `modules/private-stub`
+directory to determine which resources have to be provided.
+
+### `hiera/private`
+
+Analogous to `modules/private`, [Hiera](https://docs.puppetlabs.com/hiera/1/)
+configuration files specific to the current environment are expected to be
+found in `hiera/private`. Default resources for development (and testing)
+purpose are provided within `modules/private-stub/hiera`:
+
+    # UNIX-oid
+    user@host:~/infrastructure$ ln -s ../modules/private-stub/hiera hiera/private
+
+    # Windows
+    C:\infrastructure\> MKLINK /D ..\modules\private-stub\hiera hiera\private
+
+Note that custom versions are recommended to be tracked together with the
+custom `private` module, if any.
 
 Development environment
 -----------------------
 
 As with our other projects, all changes to our infrastructure should
 be made in a local development environment, and reviewed before
 deployment. Thanks to Puppet, we can easily set up local VMs that
 mirror our production environment.
 
 The most convenient way to do this is to use Vagrant, as described
 below.
 
 ### Requirements

[Felix Dahlke, 2015/03/03 16:43:32]

Shouldn't we also add hiera here? We should also describe the
PUPPET_HIERA_CONFIG and PUPPET_HOSTS_CONFIG variables.

[mathias, 2015/03/04 12:32:37]

Yes we should improve on documentation, yet this is not the right place do
document either Hiera or Puppet. These are the requirements you need to set up a
development environment. And in fact one does not need Puppet or Hiera for that
purpose - they are only used in the guest boxes.

[Felix Dahlke, 2015/03/04 14:27:53]

Oh, I actually presumed we need to have Hiera in the development environment -
nevermind then.

 
 * [VirtualBox](https://www.virtualbox.org/)
 * [Vagrant](http://vagrantup.com/)
-* _modules/private_ exists (see above)
+* Both `modules/private` and `hiera/private` exist (see above)
 
 ### Start a VM
 
 For each production server, we have a Vagrant VM with the same host
 name.
 
 To start the _filter1_ VM:
 
         vagrant up filter1
 
@@ skipping 10 matching lines @@
 
         vagrant ssh server5
 
 If you want to test "real" SSH access you can use the test user account defined
 in _private-stub_:
 
         ssh -i modules/private/files/id_rsa test@10.8.0.100
 
 The default password for this user (required for the _sudo_ command) is "test".
 
-Adding a server
----------------
+Adding a host
+-------------
 
-To set up a new server, you should first add it to the development
-environment and test the setup, then set up a corresponding production
-server.
+To set up a new host, extend the custom `hiera/private/host.yaml` by another
+`servers:` item, e.g.:
 
-### Development environment
+    # ...
+    custom1:
+        ip: [10.8.0.254]
+        dns: foobar.example.com
+        ssh_public_key: AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAA...
+        role: codereviewserver
 
-1. Add entries in _Vagrantfile_ and _manifests/vagrant.pp_
+See `modules/base/manifests/init.pp`, especially the `explicit_host_record()`
+named type within class `base`, for more information on the possible option
+keys and values.
 
-2. Add the host name to one of the manifests imported by
-_manifests/nodes.pp_
+Configuring Puppet

[Felix Dahlke, 2015/03/03 16:43:32]

This part of "Adding a host", so it should rather be a ### headline I'd say.

[mathias, 2015/03/04 12:32:37]

Not exclusively, it also applies to the master. Nonetheless I've extended the
above section, and updated this one. That should improve it in the sense of your
comment.

+------------------
 
-3. Make sure the server uses the _nagios::client_ class and add a
-_nagios\_host_ to _manifests/monitoringserver.pp_
+Below please find brief instructions for setting up Puppet on both master

[Felix Dahlke, 2015/03/03 16:43:32]

Sounds awfully formal and doesn't really add anything IMO, I'd just get rid of
this paragraph :)

[mathias, 2015/03/04 12:32:37]

Done.

+and agents:
 
-### Production environment
+### Prerequisites
 
 1. Install Ubuntu Server 12.04 LTS
 2. Perform an update and install Puppet
 
         apt-get -y update && apt-get -y upgrade && apt-get -y install puppet
 
 3. Enable pluginsync (Add the following to the _main_ section in
    _/etc/puppet/puppet.conf_)
 
         pluginsync=true
@@ skipping 73 matching lines @@
 Monitoring is fully functional in any environment, including development.
 Here, after bootstrapping the `server4` box, one can access the Nagios GUI
 from the host machine via <https://nagiosadmin:nagiosadmin@10.8.0.99/>.
 
 The monitoring service of our production environment, however, is accessible
 via <https://monitoring.adblockplus.org/>.
 Add yourself to _files/nagios-htpasswd_ in the _private_ module used on the
 server, or have someone add you if you don't have access.