Issue 122 - Puppet ENC via Hiera

README.md

 Adblock Plus infrastructure
 ===========================
 
 The Adblock Plus infrastructure uses [Puppet](http://puppetlabs.com/)
 to set up servers, and to have a realistic development environment.
 
 Our Puppet manifests are only tested with Ubuntu 12.04 right now.
 
-Private files
--------------
+Environment specific setup
+--------------------------
 
-Some parts of our infrastructure are, obviously, confidential. We have
-htpasswd files, SSH keys and SSL certificates that we need to be
-careful with.
+Some infrastructure parts are specific to the environment (such as e.g.
+*development*, *test* and *production*) whilst passwords, for example,
+are confidential. In order to allow for such specific configuration, the
+repository requires a set of manual operations during the initial setup:
 
-That's why _modules/private_ is missing, and needs to be placed there
-manually. We provide stub versions of all those files in
-_modules/private-stub_, so just linking or copying that to
-_modules/private_ will make everything work locally.
+### `modules/private`
+
+The `private` module is destined to store confidential information such as
+[RSA](http://en.wikipedia.org/wiki/RSA_%28cryptosystem%29) keys, `htpasswd`
+files and so on. The repository provides a `private-stub` module containing
+defaults suitable for development and testing purposes. One can create a
+symbolic link to start using the resource:
+
+#### UNIX-like
+
+    ln -s private-stub modules/private
+
+#### Windows
+
+    MKLINK /D modules\private private-stub
+
+When creating a custom version, one may inspect the `modules/private-stub`
+directory to determine which resources have to be provided.
+
+### `hiera/private`
+
+Analogous to `modules/private`, [Hiera](https://docs.puppetlabs.com/hiera/1/)
+configuration files specific to the current environment are expected to be
+found in `hiera/private`. Default resources for development (and testing)
+purposes are provided within `modules/private-stub/hiera`:
+
+#### UNIX-like
+
+    ln -s ../modules/private-stub/hiera hiera/private
+
+#### Windows
+
+    MKLINK /D ..\modules\private-stub\hiera hiera\private
+
+Note that custom versions are recommended to be tracked together with the
+custom `private` module, if any.
 
 Development environment
 -----------------------
 
 As with our other projects, all changes to our infrastructure should
 be made in a local development environment, and reviewed before
 deployment. Thanks to Puppet, we can easily set up local VMs that
 mirror our production environment.
 
 The most convenient way to do this is to use Vagrant, as described
 below.
 
 ### Requirements
 
 * [VirtualBox](https://www.virtualbox.org/)
 * [Vagrant](http://vagrantup.com/)
-* _modules/private_ exists (see above)
+* Both `modules/private` and `hiera/private` exist (see above)
 
 ### Start a VM
 
 For each production server, we have a Vagrant VM with the same host
 name.
 
 To start the _filter1_ VM:
 
         vagrant up filter1
 
@@ skipping 10 matching lines @@
 
         vagrant ssh server5
 
 If you want to test "real" SSH access you can use the test user account defined
 in _private-stub_:
 
         ssh -i modules/private/files/id_rsa test@10.8.0.100
 
 The default password for this user (required for the _sudo_ command) is "test".
 
-Adding a server
----------------
+Adding a host
+-------------
 
-To set up a new server, you should first add it to the development
-environment and test the setup, then set up a corresponding production
-server.
+To set up a new host, extend the custom `hiera/private/host.yaml` by another
+`servers:` item, e.g.:
 
-### Development environment
+    # ...
+    custom1:
+        ip: [10.8.0.254]
+        dns: foobar.example.com
+        ssh_public_key: AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAA...
+        role: codereviewserver
 
-1. Add entries in _Vagrantfile_ and _manifests/vagrant.pp_
+See `modules/base/manifests/init.pp`, especially the definition of the named
+type `explicit_host_record()` within class `base`, for more information on the
+possible option keys and values.
 
-2. Add the host name to one of the manifests imported by
-_manifests/nodes.pp_
+In development, this is all that needs to be done before the new box can be
+started using `vagrant up ...`. Production servers, however, need a working
+Puppet configuration first (see below).
 
-3. Make sure the server uses the _nagios::client_ class and add a
-_nagios\_host_ to _manifests/monitoringserver.pp_
+Configuring Puppet
+------------------
 
-### Production environment
+### Prerequisites
 
 1. Install Ubuntu Server 12.04 LTS
-2. Perform an update and install Puppet
-
-        apt-get -y update && apt-get -y upgrade && apt-get -y install puppet
-
+2. Run `hiera/install_precise.py` as user `root` to install Puppet and Hiera
 3. Enable pluginsync (Add the following to the _main_ section in
    _/etc/puppet/puppet.conf_)
 
         pluginsync=true
 
 4. Configure the master address (Add the following to the bottom of
         _/etc/puppet/puppet.conf_)
 
         [agent]
         server = puppetmaster.adblockplus.org
 
 Now you can either set it up as a pure agent or as a master. The
 master provides the configuration, agents fetch it from the master and
 apply it locally. The master is also an agent, fetching configuration
 from itself.
 
-#### Puppet agent
+### Puppet agent
 
 1. Attempt an initial provisioning, this will fail
 
         puppet agent --test
 
 2. On the master: List the certificates to get the name of the new
    agent's certificate
 
         puppet cert list
 
 3. Still on the master: Sign the certificate, e.g. for serverx:
 
         puppet cert sign serverx
 
 4. Back on the agent: Attempt another provisioning, it should work now
 
         puppet agent --test
 
-#### Puppet master
+### Puppet master
 
 1. Configure the certificate name (Add the following to the _master_
    section in _/etc/puppet/puppet.conf_)
 
         certname = puppetmaster.adblockplus.org
 
 2. Install the required packages
 
         apt-get install puppetmaster mercurial
 
@@ skipping 31 matching lines @@
 Monitoring is fully functional in any environment, including development.
 Here, after bootstrapping the `server4` box, one can access the Nagios GUI
 from the host machine via <https://nagiosadmin:nagiosadmin@10.8.0.99/>.
 
 The monitoring service of our production environment, however, is accessible
 via <https://monitoring.adblockplus.org/>.
 Add yourself to _files/nagios-htpasswd_ in the _private_ module used on the
 server, or have someone add you if you don't have access.