Nomad Host Network Configuration

Recently, I encountered an issue with our new Nomad cluster. After adding a Nomad client node via a VPN connection, the node was successfully visible in Nomad and able to run workloads.

However, I experienced a problem with Nomad's built-in service discovery feature, which was unable to retrieve the correct IP address for a deployed service.

As part of a task in my Nomad job, I required the ability to look up the address of a service using Nomad's service discovery mechanism:

job "my-job" {
  datacenters = ["dc1"]
  type      = "service"

  group "my-job-group" {
    count = 1
    
    task "my-job-task" {
      driver = "docker"

      template {
        data        = <<EOH
MY_ENV="{{ range nomadService "service-on-vpn-node" }}http://{{ .Address }}:{{ .Port }}{{ end }}"
EOH
        destination = ".env"
        env = true
      }

      config {
        image = "bitexpert.loc/app:0.1.0"
      }
    }
  }
}

Upon starting the job, it failed to establish a connection to the service-on-vpn-node service. After conducting a debug session, I discovered that Nomad was attempting to use the public IP address of the Nomad client node instead of the VPN-internal IP address.

This behavior was unexpected, as I had explicitly configured the Nomad client to listen for connections and advertise its presence using the VPN-internal IP address:

data_dir  = "/var/nomad/data"
bind_addr = "10.10.22.250"

advertise {
  http = "10.10.22.250"
  rpc  = "10.10.22.250"
  serf = "10.10.22.250"
}

client {
  enabled   = true
  servers   = ["10.10.21.112:4647"]

}

As I was digging through the Nomad documentation, I discovered the host network configuration option for Nomad clients. This setting allows you to specify a host network that can be used by your job to expose services. To use this feature, add a host_network block to your Nomad client configuration, as shown in the following example:

data_dir  = "/var/nomad/data"
bind_addr = "10.10.22.250"

advertise {
  http = "10.10.22.250"
  rpc  = "10.10.22.250"
  serf = "10.10.22.250"
}

client {
  enabled   = true
  servers   = ["10.10.21.112:4647"]
  
  host_network "vpn" {
    cidr = "10.10.22.250/24"
  }
}

Once the host_network block is configured, you can reference the vpn host network in your job configuration when configuring the ports:

job "vpn-service" {
  datacenters = ["dc1"]
  type = "service"

  group "vpn-service-group" {
    count = 1

    network {
      port "app" {
        to           = 8080
        host_network = "vpn"
      }
    }

    task "vpn-service-task" {
      driver = "docker"

      config {
        image = "bitexpert.loc/vpn-service:0.5.1"
        ports = ["app"]
      }

      resources {
        cpu    = 125
        memory = 2048
      }

      service {
        name     = "service-on-vpn-node"
        provider = "nomad"
        port     = "app"
      }
    }
  }
}

Following a restart of the Nomad client and a redeployment of the vpn-service job, I successfully redeployed my my-job and verified that Nomad's service discovery returned the expected VPN IP address.