Using templates in Nomad job files

In a Nomad job file, template blocks are a convenient way to ship configuration files that are populated from environment variables, Vault secrets, or just general configurations.

For the past years, we've always embedded configuration files in the job file like this:

template {
    data = <<EOH
config:
  value1: "test"
  value2: "foo bar"
EOH
    destination = "local/config.yaml"
}

While this approach works well for small configuration files, it becomes more complicated when dealing with larger files. As your Nomad job file grows, it becomes harder to follow and understand. Additionally, you lose the benefits of your IDE's capabilities, such as error detection in the template file, since your IDE does not recognize the embedded piece as a YAML configuration or JSON file.

Furthermore, validating the content with an external tool is not possible because it is embedded directly in your Nomad job file.

During my research on resolving the issue, I discovered a crucial oversight: the ability to call HCL functions, such as readfile(), within the template block. That means, instead of embedding the configuration file directly in the job file, we can leverage the readfile() function to read an external file. For example:

template {
    data = file("config/custom-config.yaml")
    destination = "local/config.yaml"
}

Fortunately, Nomad still interprets the file content as a Go template, enabling the retrieval of data from Consul and Vault as before. A key requirement for us.

One notable drawback is that the file content must be readable by the Nomad CLI tool when running the job. However, this is typically not a significant issue since we run our jobs through a GitLab CI pipeline, and both the Nomad job file and the configuration file are versioned in Git.

Nevertheless, there is an important consideration when using the Nomad Web UI: the job definition will be displayed without the embedded file content. As a result, you will lose the convenience of easily restarting the job directly from the Nomad Web UI by copying the job definition and pasting it into the job editor.

In general, I prefer using external templates in Nomad job files because it enables us to lint these files in our CI pipeline. This approach allows us to catch and fail on any issues early on, rather than experiencing failures when the Nomad job file is actually running.