Deleting dynamic host volumes in Nomad

While doing a bit of spring cleanup in the various Nomad clusters we host for us and our customers, I wanted to delete a few dynamic host volumes that were no longer in use.

While running the following command to delete a dynamic host volume:

nomad volume delete --type=host 7f8d6ba2-992e-556a-5382-ba976c38febd

The Nomad server responded with:

Error deleting volume: Unexpected response code: 500 (could not delete volume
7f8d6ba2-992e-556a-5382-ba976c38febd in use by alloc b102ceb2-8fe3-9e8e-d465-d1c84f36502b)

That was a first, and the error seems weird to me, as the Nomad Web UI did not show that the volume is still used by a specific allocation. And even weirder, the error happened in two of our clusters at the same time.

After a bit of research and experimentation, I found 3 ways of fixing the issue:

1. As of Nomad 1.10.2, the --force flag can be used when deleting a volume. However, after upgrading our cluster and local CLI client to version 1.10.2, I found that the volume still could not be deleted, even with the --force flag.
2. To resolve the issue, first identify the relevant allocation by running a command like nomad alloc status b102ceb2-8fe3-9e8e-d465-d1c84f36502b. Next, stop and purge the associated Nomad job. Finally, attempt to delete the volume again. This approach resolved the issue for me.
3. As a more drastic measure, you can try the following steps: drain and stop the Nomad client that is hosting the allocation and volume. Then, run nomad system gc to trigger a garbage collection cycle. After that, restart the node and attempt to delete the volume again. I found this approach to be effective as well.