Running Qwen 3.8 on Hashicorp Nomad

While exploring ways to run Nvidia workloads on Nomad, I stumbled upon an opportunity to upgrade our vLLM setup to support the newly released Qwen 3.8 model. This involved reconfiguring our NVIDIA DGX Spark server.

By accident, I came across this GitHub repo that had already done the heavy lifting for fine-tuning Qwen 3.8 with vLLM on a NVIDIA DGX Spark. The task left to do was to convert the configuration from a shell script to our Nomad job file.

First, I launched a new Docker container and downloaded the Qwen 3.8 model from Hugging Face using the following commands:

docker run -it -v /var/nomad/data/host_volumes/da12a803-152e-83d4-049d-81e11d4adf5a/:/opt/vllm nvcr.io/nvidia/vllm:26.06-py3 bash

hf download unsloth--Qwen3.8-27B-NVFP4

After completing the model download, I submitted the Nomad job file to our cluster:

job "vllm.prod" {
  datacenters = ["dc1"]
  type        = "service"
  node_pool   = "gpu"

  group "vllm" {
    count = 1

    network {
      port "app" {
        to = 8000
        host_network = "vpn"
      }
    }

    volume "vllm.prod" {
      type            = "host"
      source          = "vllm.prod"
      access_mode     = "single-node-writer"
      attachment_mode = "file-system"
    }

    restart {
      attempts = 3
      interval = "30m"
      delay    = "45s"
      mode     = "fail"
    }

    task "server" {
      driver = "docker"

      vault {
        change_mode   = "signal"
        change_signal = "SIGUSR1"
      }

      template {
        data        = <<EOH
HUGGING_FACE_HUB_TOKEN=YOUR_TOKEN_HERE
HF_HOME=/opt/vllm/hf
HF_HUB_CACHE=/opt/vllm/hf/hub
VLLM_FLOAT32_MATMUL_PRECISION=high
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
UCX_RCACHE_MAX_UNRELEASED=1024
CUTE_DSL_ARCH=sm_121a
TRITON_CACHE_DIR=/opt/vllm/.triton
EOH
        destination = ".env"
        env         = true
      }

      config {
        image      = "nvcr.io/nvidia/vllm:26.06-py3"
        force_pull = true
        ports      = ["app"]
        args = [
          "vllm",
          "serve",
          "/opt/nim/.cache/hf/hub/models--unsloth--Qwen3.8-27B-NVFP4/snapshots/7d6f8d4d72f56b92b3cdbf22f156b90e1bab0108",
          "--served-model-name",
          "Qwen/Qwen3.8",
          "--tensor-parallel-size",
          "1",
          "--quantization",
          "compressed-tensors",
          "--attention-backend",
          "triton_attn",
          "--trust-remote-code",
          "--enable-auto-tool-choice",
          "--tool-call-parser",
          "qwen3_coder",
          "--reasoning-parser",
          "qwen3",
          "--skip-mm-profiling",
          "--default-chat-template-kwargs",
          jsonencode({
            enable_thinking = false
          }),
          "--enable-prefix-caching",
          "--enable-chunked-prefill",
          "--block-size",
          "64",
          "--gpu_memory_utilization",
          "0.84",
          "--max-num-seqs",
          "4",
          "--max-num-batched-tokens",
          "8192",
          "--max-model-len",
          "1000000",
          "--speculative-config",
          jsonencode({
            "method": "mtp",
            "num_speculative_tokens": 2
          }),
          "--hf-overrides",
          jsonencode({
            "text_config": {
              "rope_parameters": {
                "mrope_interleaved": true,
                "mrope_section": [11, 11, 10],
                "rope_type": "yarn",
                "rope_theta": 10000000,
                "partial_rotary_factor": 0.25,
                "factor": 4.0,
                "original_max_position_embeddings": 262144
              }
            }
          }),
          "--api-key",
          "some_api_key",
        ]

        runtime = "nvidia"
        ipc_mode = "host"
        ulimit {
          memlock = "-1:-1"
          stack   = "67108864:67108864"
        }
      }

      volume_mount {
        volume      = "vllm.prod"
        destination = "/opt/vllm"
        read_only   = false
      }

      resources {
        cpu    = 45000
        memory = 100000
        device "nvidia/gpu" {
          count = 1
        }
      }

      service {
        name     = "vllm-prod"
        provider = "nomad"
        port     = "app"

        check {
          name     = "alive"
          type     = "tcp"
          interval = "10s"
          timeout  = "2s"

          check_restart {
            limit           = 3
            grace           = "55m"
            ignore_warnings = false
          }
        }
      }
    }
  }
}

By leveraging existing configurations and converting them for our Nomad setup, we were able to efficiently deploy the Qwen 3.8 model on our NVIDIA DGX Spark device. This approach not only saved time but also ensured compatibility and optimal performance.