Ansible and Host Configuration

calendar Oct 23, 2021 · 8 min read · ansible lets encrypt

Overview

Backstory

I think just about everyone of my posts I start with some sort of justification why I'm writing the post. I don't really know why .. other than I think it serves as a good reminder of what I was doing when I wrote the darned post ...

As usual, these things start off at work.

We had a bit of a security audit on one of our 3rd party hosts (hosted off-domain in Azure), and it yielded some basic misconfigurations that should get fixed: Some old TLS settings that were no longer supported, updating PHP version, updates to the application to remove unsupported dependencies and so on.

The challenge is making these changes without breaking production.

Enter Ansible

I've been using Ansible at work for a little while. It started out as an "easy" way to run apt-get update; apt-get install -y across all the linux hosts we have (I think 6? oh so many..), and it's grown into a bit more - managing SSH keys, managing installation of monitoring software and so on. So.. why not use it to manage the installation of this quirky little application?

First thing, I would create a new Azure host to experiment on. I'll add this host to my inventory file, do my initial connect, then run my all-hosts-configure playbook. This installs a handful of default pacakges (aptitude, git, sudo, vim, etc..). It then runs some basic configurations against all hosts (adding SSH keys, loads in a root CA, installs monitoring agents).

With that out of the way, I have a host that is basically "ready to go" in terms of having the tools I need.

First playbook

Nothing too fancy here. The host is already in inventory, so a small playbook to call a role:

1# application.yml
2---
3- name: Configure a host for application.
4  hosts: application-test
5
6  roles:
7    - application

Then, create the role with ansible-galaxy role init application. Make the necessary basic changes to roles/application/meta/main.yml, then carry on to working on the meat of the role.

In this case, I'm trying to achieve a handful of items here:

  1. Install and configure nginx to be "default". This should mirror production.
  2. Install PHP to what's available in Debian's repository. This, again, should mirror production.
  3. Install mariaDB (server).
  4. Install certbot

I then manually load the application and database in. I'm not trying to handle full application deployment (yet) - just the service configuration.

So far, so good.

My playbook at this point looks something like:

1# /roles/application/tasks/main.yml
2- name: Ensure application software is installed
3  apt:
4    name: nginx,mariadb-server,python3-certbot-nginx,php-fpm,php-cli,php-mysql,php-opcache,php-readline
5    state: present

This gets me nginx, php and mariadb-server at default versions. Debian's repository is seemingly stuck at PHP version 7.3, so by installing the php-* pacakges, I get version 7.3 (which is out of support as of this posting).

Upgrading PHP

There's a manual way of upgrading PHP on a Debian host (found here), but this isn't the point of what we're trying to do here...

Instead, Jeff Geerling has a great PHP role for Ansible that handles this all so very nicely.

Minimally, we need to define what version of PHP we want to use. We'll set this in the role's vars file:

1# /roles/application/vars/main.yml
2php_version: '7.4'

Out of the gate, I actually set this to PHP 8, but found the application would not run under version - something for the software developers to fix. I just provide the infrastructure...

The role handled this generally quite well - by changing versions, the role would remove unnecessary packages and only install the ones that I need.

I then need to install the packages that the application needs. I'm updating my role's task's to:

 1# /roles/application/tasks/main.yml
 2- name: Ensure application software is installed
 3  apt:
 4    name: nginx,mariadb-server,python3-certbot-nginx,php{{ php_version }}-fpm,php{{ php_version }}-cli,php{{ php_version }}-mysql,php{{ php_version }}-opcache,php{{ php_version }}-readline
 5    state: present
 6
 7- name: Install php-json if we're < 8.0
 8  apt:
 9    name: php{{ php_version }}-json
10    state: present
11  when:
12    php_version is version('8.0', '<')

php-json is bundled as an extension in PHP versions before 8. After 8, it's part of the core installation - and no longer has a separate package. This causes apt to barf, so we split it out with the conditional.

Configuring nginx

Some of the things I needed to correct would require changes to the site's configuration in nginx. Ansible can handle that too, with the added benefit of being able to substitute variables into a template file. I add a few more tasks to the playbook:

 1# /roles/application/tasks/main.yml
 2- name: Ensure nginx configuration is loaded on server
 3  ansible.builtin.template:
 4    src: templates/application-nginx.conf.j2
 5    dest: /etc/nginx/sites-available/application.domain.com
 6    owner: root
 7    group: root
 8    mode: "0644"
 9  notify:
10    - Restart Nginx
11
12- name: Symlink configuration to enabled configuration
13  ansible.builtin.file:
14    state: link
15    src: /etc/nginx/sites-available/application.domain.com
16    dest: /etc/nginx/sites-enabled/application.domain.com
17    owner: root
18    group: root
19  notify:
20    - Restart Nginx

Notify allows me to trigger a restart for nginx after any of this changes. This only triggers once, if required, at the end of the play. Failures are reported back to the host calling the playbook and are a bit serialized (as in, all on one line). It can be a bit hard to read, but does the job.

1# /roles/application/handlers/main.yml
2---
3- name: Restart Nginx
4  ansible.builtin.service:
5    name: nginx
6    state: restarted

The template file, templates/application-nginx.conf.j2 is similar to the following:

 1# /roles/application/templates/application-nginx.conf.j2
 2
 3server {
 4    {% for local_server_name in hostvars[inventory_hostname].vars.server_names %}
 5    server_name {{ local_server_name }};
 6    {% endfor %}
 7    root /var/www/application;
 8    index index.php index.html;
 9
10    # Enforce secure cookies
11    add_header Set-Cookie "Path=/; HttpOnly; Secure";
12
13    location ~ /\.git {
14        deny all;
15    }
16
17    location / {
18        try_files $uri $uri/ /index.php;
19        location = /index.php {
20            include snippets/fastcgi-php.conf;
21            fastcgi_pass unix:/var/run/php/php{{ php_version }}-fpm.sock;
22        }
23    }
24
25    location ~ \.php$ {
26        return 444;
27    }
28
29    listen 443 ssl; # managed by Certbot
30    ssl_certificate /etc/letsencrypt/live/application-dev.domain.com/fullchain.pem; # managed by Certbot
31    ssl_certificate_key /etc/letsencrypt/live/application-dev.domain.com/privkey.pem; # managed by Certbot
32    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
33    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
34}
35
36server {
37    {% for local_server_name in hostvars[inventory_hostname].vars.server_names %}
38    if ($host = {{ local_server_name }}) {
39        return 301 https://$host$request_uri;
40    }
41    {% endfor %}
42    listen 80;
43    {% for local_server_name in hostvars[inventory_hostname].vars.server_names %}
44    server_name {{ local_server_name }};
45    {% endfor %}
46    return 404; # managed by Certbot
47}

At this point, I obviously already have Certbot doing it's thing. I thought it would be important to have the nginx configuration be as close to "correct" as possible. I later thought about automating Certbot (coming later...).

with a little bit of Jinja2 looping magic, I can get my configuration file to work the way I want it. Sweet!

I've had to tell Ansible what the server names will be. This becomes a bit of an issue as the Azure hostname doesn't necessarily match the server name that the application will be hosted at - otherwise I could just use ansible_hostname as a variable.

I'm providing this in my inventory file:

 1# /inventory
 2---
 3--- snip ---
 4hosts:
 5  application.domain.com:
 6    ansible_host: application-prod.canadacentral.cloudapp.azure.com
 7    vars:
 8      server_names:
 9      - application.domain.com
10      - applicationapi.domain.com
11  application-dev.domain.com:
12    ansible_host: application-test.canadacentral.cloudapp.azure.com
13    vars:
14      server_names:
15      - application-dev.domain.com
16      - applicationapi-dev.domain.com

LetsEncrypt options for Nginx

This could be in the last section, but.. eh.

Certbot for nginx installs a set of options that leave TLS 1.0 and 1.1 enabled, both are considered insecure. In this case, I want to leave the rest of the file intact, but just change the lines that I need to. Ansible's got a tool for that too - the lineinfile module.

I had a heck of a time getting this just right. It would often append the parameters to the end of the file, instead of replacing the line I was interested in. This turned out to be my judicious use of spaces.. so.. be mindful of blank spaces.

 1# /roles/application/tasks/main.yml
 2- name: Update LetsEncrypy options
 3  ansible.builtin.lineinfile:
 4    path: /etc/letsencrypt/options-ssl-nginx.conf
 5    owner: root
 6    group: root
 7    mode: "0644"
 8    state: present
 9    search_string: "{{ item.search_string }}"
10    line: "{{ item.line }}"
11  notify:
12    - Restart Nginx
13  loop:
14    - {
15        search_string: ssl_session_cache,
16        line: ssl_session_cache shared:le_nginx_SSL:40m;, # holds approx 40 x 4000 sessions,
17      }
18    - { search_string: ssl_protocols, line: ssl_protocols TLSv1.2 TLSv1.3; }
19    - { search_string: ssl_session_timeout, line: ssl_session_timeout 2h; }
20    - { search_string: ssl_session_tickets, line: ssl_session_tickets off; }
21    - {
22        search_string: ssl_ciphers,
23        line: ssl_ciphers "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384";,
24      }

Here, we're replacing options in /etc/letsencrypt/options-ssl-nginx.conf. The search_string parameter searches for strings that start with the supplied line. Optionally, we can do regex's, but an options file lends itself well to this kind of searching/replacing.

I'll then loop over a set of search_string and replace them with the corresponding line item. Just like changing the nginx configuration files, if this file changes it will trigger a restart of the service.

Application configurations

There are a handful of application configuration changes that need to get made as I take the source from source control. These are handled like the options-ssl-nginx.conf file: Using Ansible's lineinfile module, I am able to replace each of the parameters as I need to. This helps keep what's in source control decoupled from what winds up on the server.

LetsEncrypt

More on this later, but for now I am running certbot on the host manually.

Done-ish?

With the server and PHP being configured the way we want through Ansible, it puts me in a better position to do test deployments from the software group. It should enable me to test on the development server with new builds, and once we're happy with them, target the production server with the configuration.

Obviously there are still some gaps:

  • Automating SSL configuration
  • Automating renewal of SSL certificates
  • Automatic retrieval of the application files from source control (and installation on the host)
  • Decoupling important variables from the role out to the playbook (specifically PHP version, but also some DB connection information)

Posts in this Series