Improving Cert Handling in Nomad

When we first started using Nomad about 7 years ago, we quickly realized the need for Vault to handle secrets and certificates. Initially, we queried certificates from Vault during Docker image build time using a curl command.

The initial approach involved querying the certificate during Docker build time from Vault like this:

CERT=$(curl -s -H "X-Vault-Token:$VAULT_TOKEN" -XGET $VAULT_ADDR/v1/nomad/data/job.prod/certs | jq -r .data.data.CERTIFICATE)

However, this method had a significant downside: upon certificate renewal, we had to update all our Docker images and redeploy them. Over time, as we added this to more Docker images, we occasionally forgot to update an image.

To address this issue, we decided to change our procedure. Now, Nomad retrieves certificates from Vault, and the Docker entrypoint script processes the data if needed. Although this slightly increases startup time, the benefits outweigh the costs. As a bonus, whenever we change the certificate in Vault, the Nomad jobs are notified and re-rendered with the updated certificate.

In our Nomad job file, we populate a template block to fetch the certificate from Vault:

template {
  data = <<EOH
{{ with secret "job.prod/certs" }}
{{- .Data.data.CERTIFICATE -}}
{{ end }}
EOH
  destination = "secrets/cert.pem"
} 

Then, in the Docker configuration, we mount the /secrets directory to the container and read the certificate from there:

config {
  image      = "nexus.loc/app:1.0.1"
  command    = "/usr/local/bin/entrypoint.sh"
  volumes    = [
    "secrets:/secrets",
  ]
}

By leveraging Nomad and Vault, we've streamlined our certificate management process, eliminating the need for manual updates upon renewal. This approach has improved our deployment efficiency and reduced the likelihood of errors.