Puppetmaster with nginx and Mongrel on Ubuntu

If you’ve not heard of Puppet, it is a
configuration management tool. You write descriptions of how you want
your systems to look and it checks the current setup and works out what
it needs to do to move your system so it matches your description.
The idea is to write how it should look, not how to change the system.

Puppet uses a client (puppetd) that talks to the central server
(puppetmaster) over HTTPS.The default puppetmaster HTTP server is
webbrick, which is a
lightweight Ruby HTTP server. While it’s simple and allows Puppetmaster
to work straight out the box, due to it’s pure Ruby structure and Ruby’s
green thread architecture, it doesn’t scale beyond a simple puppet
setup. After a while, every medium to large Puppet installation needs to
move to the other HTTP server that puppet supports: Mongrel. Mongrel is
a faster HTTP library, but supports a lot less features. In particular
it doesn’t support SSL, which is important with Puppet, as Puppet relies
heavily on client certificate verification for authentication. As a
result, we need to put another webserver in front that can handle the
SSL aspect. As a nice side effect of having to proxy to Puppetmaster is
that we can run several puppetmaster processes and improve on the green
threads problem that Ruby has. In this blog post, I’m going to describe
setting up nginx and mongrel.

The first thing to do is to install the mongrel and
nginx packages.

apt-get install mongrel nginx

We need to run nginx on port 8140 and proxy to
our mongrel servers on different ports, so lets move puppetmaster off
8140 and configure it to use mongrel while we’re at it. Edit
/etc/default/puppetmaster and set the following variables:

SERVERTYPE=mongrel
PUPPETMASTERS=4
PORT=18140
DAEMON_OPTS="--ssl_client_header=HTTP_X_SSL_SUBJECT"

This tells the init.d script to use the mongrel server type and to
run four of them. The init.d script is clever enough to start up the
right number of processes and will set them up to use a sequence of
ports for each one, starting at 18140 for the first process, up to 18143
for the last one. The DAEMON_OPTS option tells Puppetmaster how
we’re going to pass the SSL certificate information from nginx so it can
grant or refuse permission.

Now to set up nginx. Put the following in
/etc/nginx/conf.d/puppetmaster.conf:

ssl                     on;
ssl_certificate /var/lib/puppet/ssl/certs/puppetmaster.example.com.pem;
ssl_certificate_key /var/lib/puppet/ssl/private_keys/puppetmaster.example.com.pem;
ssl_client_certificate  /var/lib/puppet/ssl/certs/ca.pem;
ssl_ciphers             SSLv2:-LOW:-EXPORT:RC4+RSA;
ssl_session_cache       shared:SSL:8m;
ssl_session_timeout     5m;

upstream puppet-production {
   server 127.0.0.1:18140;
   server 127.0.0.1:18141;
   server 127.0.0.1:18142;
   server 127.0.0.1:18143;
}

In this file we tell nginx where to find the server certificates for
your puppetmaster, so your clients can authenticate your server. We also
tell nginx the CA certificate to authenticate clients with and set up
some SSL details required for Puppet. Finally we create a group of
remote servers for our pack of mongrel puppetmasters, so we can refer to
them later. If you added more or less servers earlier don’t forget to
add or remove them here. You also need to replace
puppetmaster.example.com with your FQDN. If at a later stage, you find
you need ever more performance, you can easily move some of your
puppetmaster processes to a separate box and update the upstream list to
refer to servers on the remote server.

Finally, we need to set up a couple of HTTP servers. Create
/etc/nginx/sites-enabled/puppetmaster with the following
contents:

server {
    listen                  8140;
    ssl_verify_client       on;
    root                    /var/empty;
    access_log              /var/log/nginx/access-8140.log;

    # Variables
    # $ssl_cipher returns the line of those utilized it is cipher for established SSL-connection
    # $ssl_client_serial returns the series number of client certificate for established SSL-connection
    # $ssl_client_s_dn returns line subject DN of client certificate for established SSL-connection
    # $ssl_client_i_dn returns line issuer DN of client certificate for established SSL-connection
    # $ssl_protocol returns the protocol of established SSL-connection

    location / {
        proxy_pass          http://puppet-production;
        proxy_redirect      off;
        proxy_set_header    Host             $host;
        proxy_set_header    X-Real-IP        $remote_addr;
        proxy_set_header    X-Forwarded-For  $proxy_add_x_forwarded_for;
        proxy_set_header    X-Client-Verify  SUCCESS;
        proxy_set_header    X-SSL-Subject    $ssl_client_s_dn;
        proxy_set_header    X-SSL-Issuer     $ssl_client_i_dn;
        proxy_read_timeout  65;
    }
}

server {
    listen                  8141;
    ssl_verify_client       off;
    root                    /var/empty;
    access_log              /var/log/nginx/access-8141.log;

    location / {
        proxy_pass  http://puppet-production;
        proxy_redirect     off;
        proxy_set_header   Host             $host;
        proxy_set_header   X-Real-IP        $remote_addr;
        proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
        proxy_set_header   X-Client-Verify  FAILURE;
        proxy_set_header   X-SSL-Subject    $ssl_client_s_dn;
        proxy_set_header   X-SSL-Issuer     $ssl_client_i_dn;
        proxy_read_timeout  65;
    }
}

This creates two servers on port 8140 and 8141 which both proxy all
requests to our group of mongrel servers, adding suitable headers to
pass on the SSL information. The only difference between them is the
X-Client-Verify header. This shows the one problem with using nginx with
puppet. Because the client verification success or failure is not
available as a variable before nginx 0.8.7, we can’t have a single port
for both the usual client connection and the initial unauthenticated
connection where the client requests a certificate to be signed. As a
result, with this setup, you are required to run puppet with --ca-port
8141 the first time you run puppet until the certificate has been
signed with puppetca.

Foruntately with versions of nginx later than 0.8.7, you can use a
simpler setup shown below. This replaces both files shown above with the single
server. Unfortunately, 0.8.7 is not available in any
version of Ubuntu yet, not even Karmic.

server {
  listen 8140;

  ssl                     on;
  ssl_session_timeout     5m;
  ssl_certificate         /var/lib/puppet/ssl/certs/puppetmaster.pem;
  ssl_certificate_key     /var/lib/puppet/ssl/private_keys/puppetmaster.pem;
  ssl_client_certificate  /var/lib/puppet/ssl/ca/ca_crt.pem;

  # choose any ciphers
  ssl_ciphers             SSLv2:-LOW:-EXPORT:RC4+RSA;

  # allow authenticated and client without certs
  ssl_verify_client       optional;

  # obey to the Puppet CRL
  ssl_crl /var/lib/puppet/ssl/ca/ca_crl.pem;

  root                    /var/tmp;

  location / {
    proxy_pass              http://puppet-production;
    proxy_redirect         off;
    proxy_set_header    Host             $host;
    proxy_set_header    X-Real-IP        $remote_addr;
    proxy_set_header    X-Forwarded-For  $proxy_add_x_forwarded_for;
    proxy_set_header    X-Client-Verify  $ssl_client_verify;
    proxy_set_header    X-SSL-Subject    $ssl_client_s_dn;
    proxy_set_header    X-SSL-Issuer     $ssl_client_i_dn;
    proxy_read_timeout  65;
  }
}

If you are running another webserver on the server, you may want to
delete /etc/nginx/sites-enabled/default which attempts to
create a server listening on port 80, which will conflict with your
existing HTTP server.

If you follow these instructions, you should find yourself with a
better performing puppetmaster and significantly few “connection reset
by peer” and other related error messages.