Dealing with special chars in a Nomad Template block

I recently encountered a puzzling issue with a Nomad job, where a crucial environment variable was missing a # character. After some investigation, I finally uncovered the root cause.

In a new Nomad job file I wrote to deploy an application in our cluster, I had a template block like this:

template {
    data        = <<EOH
ADMIN_PASS={{ with secret "nomad/data/myjob" }}{{ .Data.data.ADMIN_PASS }}{{ end }}
EOH
    destination = ".env"
    env         = true
  }

After deploying the job to our Nomad cluster, I attempted to log in to the application using the password stored in Vault, but was met with a password error.

Despite retrying several times, the issue persisted. To troubleshoot, I connected to the Nomad task via the Web UI and inspected the environment variables.

To my surprise, I found that the password was missing a trailing # character. This was weird as I had verified that the password was correctly stored in Vault with the # character intact.

Next, I checked the Nomad docs and found the source of the problem:

Likewise, when evaluating a password that may contain quotes or #, use the
toJSON function to ensure Nomad passes the password to the task unchanged.

It turns out that this behavior is intentional, not a bug. According to the Nomad documentation, secrets and certificates can contain special characters like newlines, quotes, and backslashes, which can be tricky to handle properly. This explanation makes sense, and it's reassuring to know that Nomad is designed to handle these complexities.

The solution to the issue was to modify the template definition by wrapping it in a toJSON function call, as shown below:

template {
    data        = <<EOH
ADMIN_PASS={{ with secret "nomad/data/myjob" }}{{ .Data.data.ADMIN_PASS | toJSON }}{{ end }}
EOH
    destination = ".env"
    env         = true
  }

Following the update, I restarted the job and was then able to successfully log in using the correct password.