Issue 122 - Puppet ENC via Hiera

README.md

 Adblock Plus infrastructure
 ===========================
 
 The Adblock Plus infrastructure uses [Puppet](http://puppetlabs.com/)
 to set up servers, and to have a realistic development environment.
 
 Our Puppet manifests are only tested with Ubuntu 12.04 right now.
 
-Host specific setup

[Felix Dahlke, 2015/03/03 16:43:32]

Is that really "Host specific"? You're calling servers "hosts" below, so IMO
this is a bit confusing. Maybe something like "Environment specific setup"? Or
just "Private files" or maybe "Private resources"? :)

[mathias, 2015/03/04 12:32:37]

Done.

--------------------
+Environment specific setup
+--------------------------
 
 Some infrastructure parts are specific to the environment (such as e.g.
 *development*, *test* and *production*) whilst passwords, for example,
 are confidential. In order to allow for such specific configuration, the
 repository requires a set of manual operations during the initial setup:
 
 ### `modules/private`
 
 The `private` module is destined to store confidential information such as
 [RSA](http://en.wikipedia.org/wiki/RSA_%28cryptosystem%29) keys, `htpasswd`
 files and so on. The repository provides a `private-stub` module containing
-defaults suitable for development and testing purpose. One can create a
+defaults suitable for development and testing purposes. One can create a
 symbolic link to start using the resource:
 
-    # UNIX-oid

[Felix Dahlke, 2015/03/03 16:43:32]

Why the "-old" here?

Also, I would go for something like a #### headline instead of code comments
here. Also because "#" doesn't introduce a comment in batch scripts, so it
doesn't really make sense for the Windows part below.

[Wladimir Palant, 2015/03/03 20:00:19]

I guess using "Unix-like" will avoid that confusion...

[mathias, 2015/03/04 12:32:37]

Done.

-    user@host:~/infrastructure$ ln -s private-stub modules/private

[Felix Dahlke, 2015/03/03 16:43:32]

I'd opt for just having `$ ln -s private-stub modules/private` - the rest of the
prompt doesn't matter, and it's quite customary to leave it out.

[mathias, 2015/03/04 12:32:37]

Done.

-
-    # Windows
-    C:\infrastructure\> MKLINK /D modules\private private-stub
+#### UNIX-like
+
+    ln -s private-stub modules/private
+
+#### Windows
+
+    MKLINK /D modules\private private-stub
 
 When creating a custom version, one may inspect the `modules/private-stub`
 directory to determine which resources have to be provided.
 
 ### `hiera/private`
 
 Analogous to `modules/private`, [Hiera](https://docs.puppetlabs.com/hiera/1/)
 configuration files specific to the current environment are expected to be
 found in `hiera/private`. Default resources for development (and testing)
-purpose are provided within `modules/private-stub/hiera`:
-
-    # UNIX-oid
-    user@host:~/infrastructure$ ln -s ../modules/private-stub/hiera hiera/private
-
-    # Windows
-    C:\infrastructure\> MKLINK /D ..\modules\private-stub\hiera hiera\private
+purposes are provided within `modules/private-stub/hiera`:
+
+#### UNIX-like
+
+    ln -s ../modules/private-stub/hiera hiera/private
+
+#### Windows
+
+    MKLINK /D ..\modules\private-stub\hiera hiera\private
 
 Note that custom versions are recommended to be tracked together with the
 custom `private` module, if any.
 
 Development environment
 -----------------------
 
 As with our other projects, all changes to our infrastructure should
 be made in a local development environment, and reviewed before
 deployment. Thanks to Puppet, we can easily set up local VMs that
 mirror our production environment.
 
 The most convenient way to do this is to use Vagrant, as described
 below.
 
 ### Requirements

[Felix Dahlke, 2015/03/03 16:43:32]

Shouldn't we also add hiera here? We should also describe the
PUPPET_HIERA_CONFIG and PUPPET_HOSTS_CONFIG variables.

[mathias, 2015/03/04 12:32:37]

Yes we should improve on documentation, yet this is not the right place do
document either Hiera or Puppet. These are the requirements you need to set up a
development environment. And in fact one does not need Puppet or Hiera for that
purpose - they are only used in the guest boxes.

[Felix Dahlke, 2015/03/04 14:27:53]

Oh, I actually presumed we need to have Hiera in the development environment -
nevermind then.

 
 * [VirtualBox](https://www.virtualbox.org/)
 * [Vagrant](http://vagrantup.com/)
 * Both `modules/private` and `hiera/private` exist (see above)
 
 ### Start a VM
 
 For each production server, we have a Vagrant VM with the same host
 name.
 
@@ skipping 27 matching lines @@
 To set up a new host, extend the custom `hiera/private/host.yaml` by another
 `servers:` item, e.g.:
 
     # ...
     custom1:
         ip: [10.8.0.254]
         dns: foobar.example.com
         ssh_public_key: AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAA...
         role: codereviewserver
 
-See `modules/base/manifests/init.pp`, especially the `explicit_host_record()`
-named type within class `base`, for more information on the possible option
-keys and values.
+See `modules/base/manifests/init.pp`, especially the definition of the named
+type `explicit_host_record()` within class `base`, for more information on the
+possible option keys and values.
+
+In development, this is all that needs to be done before the new box can be
+started using `vagrant up ...`. Production servers, however, need a working
+Puppet configuration first (see below).
 
 Configuring Puppet

[Felix Dahlke, 2015/03/03 16:43:32]

This part of "Adding a host", so it should rather be a ### headline I'd say.

[mathias, 2015/03/04 12:32:37]

Not exclusively, it also applies to the master. Nonetheless I've extended the
above section, and updated this one. That should improve it in the sense of your
comment.

 ------------------
 
-Below please find brief instructions for setting up Puppet on both master

[Felix Dahlke, 2015/03/03 16:43:32]

Sounds awfully formal and doesn't really add anything IMO, I'd just get rid of
this paragraph :)

[mathias, 2015/03/04 12:32:37]

Done.

-and agents:
-
 ### Prerequisites
 
 1. Install Ubuntu Server 12.04 LTS
-2. Perform an update and install Puppet
-
-        apt-get -y update && apt-get -y upgrade && apt-get -y install puppet
-
+2. Run `hiera/install_precise.py` as user `root` to install Puppet and Hiera
 3. Enable pluginsync (Add the following to the _main_ section in
    _/etc/puppet/puppet.conf_)
 
         pluginsync=true
 
 4. Configure the master address (Add the following to the bottom of
         _/etc/puppet/puppet.conf_)
 
         [agent]
         server = puppetmaster.adblockplus.org
 
 Now you can either set it up as a pure agent or as a master. The
 master provides the configuration, agents fetch it from the master and
 apply it locally. The master is also an agent, fetching configuration
 from itself.
 
-#### Puppet agent
+### Puppet agent
 
 1. Attempt an initial provisioning, this will fail
 
         puppet agent --test
 
 2. On the master: List the certificates to get the name of the new
    agent's certificate
 
         puppet cert list
 
 3. Still on the master: Sign the certificate, e.g. for serverx:
 
         puppet cert sign serverx
 
 4. Back on the agent: Attempt another provisioning, it should work now
 
         puppet agent --test
 
-#### Puppet master
+### Puppet master
 
 1. Configure the certificate name (Add the following to the _master_
    section in _/etc/puppet/puppet.conf_)
 
         certname = puppetmaster.adblockplus.org
 
 2. Install the required packages
 
         apt-get install puppetmaster mercurial
 
@@ skipping 31 matching lines @@
 Monitoring is fully functional in any environment, including development.
 Here, after bootstrapping the `server4` box, one can access the Nagios GUI
 from the host machine via <https://nagiosadmin:nagiosadmin@10.8.0.99/>.
 
 The monitoring service of our production environment, however, is accessible
 via <https://monitoring.adblockplus.org/>.
 Add yourself to _files/nagios-htpasswd_ in the _private_ module used on the
 server, or have someone add you if you don't have access.