Autoheal your Docker containers

Sometimes one may face the need to restart a Docker container because it does not react anymore. While there are dozens of possible ways to accomplish this, I want to show the one we recently started to incorporate into our setups.

Some time ago we were in need to monitor and, if needed, automatically restarting a running Docker container based on some status of the application inside. This was also a good chance to re-think the approach we used in situations like this in the past.

The classic would be, to have an external script monitoring the application and restart the Container in case of some defined events. This script could, for example, run as a cronjob on the host. While this approach works, it feels clumsy and needs access to the server and some extra knowledge. Further, it is not nice to maintain and integrate into a Git repository.

Gladly, with Docker, there are better options today.

Docker provides the possibility to define healthchecks for your container. They are then labeled as healthy or unhealthy based on the result of that check. Those checks can be everything from simply calling a route and seeing if it is reachable to running a script in the container to perform more complex and customized checks to determine the status. The possibilities are wide and give the Developers control over how they determine a healthy service.

version: "3"
services:
  my-services:
    restart: unless-stopped
    ports:
      - "80:80"
    # ...
    healthcheck:
      test: [ "CMD", "curl", "-f", "-s", "http://192.168.0.10/healthcheck" ]
      interval: 30s
      timeout: 10s
      retries: 3

Most of the options are self-explaining, but the full documentation on this feature can be found here: https://docs.docker.com/compose/compose-file/compose-file-v3/#healthcheck

While Kubernetes or Nomad have the ability to restart unhealthy containers, Docker has not. Others then display the health status of a running container, it can not do anything to resolve this situation.

So let's add another service to this fictional setup:

version: "3"
services:
  my-services:
    restart: unless-stopped
    ports:
      - "80:80"
    # ...
    healthcheck:
      test: [ "CMD", "curl", "-f", "-s", "http://192.168.0.10/healthcheck" ]
      interval: 30s
      timeout: 10s
      retries: 3
    labels:
      - autoheal=true

  autoheal:
    image: willfarrell/autoheal:1.2.0
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    restart: unless-stopped

Autoheal acts as a stand-in until this functionality is supported by Docker natively at some point in the future. The full documentation can be found on GitHub: https://github.com/willfarrell/docker-autoheal

As soon as the healthcheck fails and the services become unhealthy, Autoheal will detect this and triggers a restart. With the label autoheal=true we can tell Autoheal which services it should monitor, but it's also possible to monitor all running containers on a host.

This solution is not only very developer-friendly and can be managed inside the Git Repository of your application, it also doesn't need any extra configuration or scripts on the server itself.