Auto-heal unhealthy containers

Restart the containers that Docker marks unhealthy with a small sidecar, without a yoink subcommand or a template.

Read as MarkdownEdit on GitHub

The Docker HEALTHCHECK directive only reports: docker ps shows (unhealthy), and nothing acts on it. The default yoink policy restart: unless-stopped restarts only a container that exits. A deadlocked process that stays alive stays unhealthy. A sidecar that monitors the docker socket restarts the unhealthy containers that have a label. The sidecar is a service fragment that you add, not code that yoink manages.

Deploy the sidecar

Add the service fragment

Save this fragment as services/autoheal.yaml:

services:
  - name: autoheal
    image: willfarrell/autoheal
    tag: 1.2.0
    description: Restart containers docker marks unhealthy.
    env:
      AUTOHEAL_CONTAINER_LABEL: autoheal
      AUTOHEAL_INTERVAL: "5"
      # Wait this many checks before acting after a fresh start —
      # avoids killing a container whose first probe is still in
      # the image's `--start-period`. Match this to the largest
      # `--start-period` across your healthchecked images.
      AUTOHEAL_START_PERIOD: "30"
      AUTOHEAL_DEFAULT_STOP_TIMEOUT: "10"
    run:
      binds:
        # Read-only socket bind. The `:ro` is cosmetic — anyone who
        # reaches /var/run/docker.sock can issue `restart` regardless
        # — but it documents intent.
        - /var/run/docker.sock:/var/run/docker.sock:ro
      options:
        restart: unless-stopped
        # autoheal needs root to read the docker socket; yoink's
        # default `nobody` user cannot. cap_drop=ALL stays — talking
        # to dockerd is privileged via socket access, not raw caps.
        user: "0:0"
        read_only: true
        tmpfs:
          /tmp: ""

The fragment has no hosts:, so yoink deploys the sidecar on every host.

Add the label to the services

Add the label autoheal: "true" to each service that auto-heal must restart:

services:
  - name: api
    # ...
    labels:
      autoheal: "true"

Auto-heal ignores a service without the label. The autoheal service is one of them.

Add a HEALTHCHECK to the images

Auto-heal reads State.Health.Status. That field is null unless the container has a HEALTHCHECK. Auto-heal ignores a container with no HEALTHCHECK, and shows no message.

If the image of a service declares no HEALTHCHECK, add one to its Dockerfile. yoink has no per-service config key that injects one. yoink sets a healthcheck only on the bundled yoink-proxy container.

# `apk add --no-cache curl` first if your base image (e.g. nginx:alpine)
# doesn't ship a HTTP client — busybox-wget is gated on the busybox
# config and isn't reliably present.
HEALTHCHECK --interval=10s --timeout=2s --start-period=30s --retries=3 \
  CMD curl -fsS -o /dev/null http://localhost:8080/health || exit 1

--start-period=30s gives time for a cold start before the first probe counts against --retries. Set AUTOHEAL_START_PERIOD on the sidecar to the same value. Then the sidecar also waits before it acts.

Make sure that auto-heal restarts a container

Deploy. Make sure that the sidecar runs on each host:

yoink up
yoink status        # autoheal running on each host

Freeze PID 1 of a labelled container. The process stays alive but does not answer its HEALTHCHECK.

ssh $HOST -- docker exec api-XXXX kill -SIGSTOP 1

The container status changes to (unhealthy) after interval × retries. Auto-heal restarts the container in AUTOHEAL_INTERVAL seconds or less. Then docker ps on the host shows a new uptime for the container.

See also

  • Security: the hardened yoink defaults that the fragment uses (cap-drop, read-only rootfs, tmpfs).
  • Templates: the accessories that yoink bundles (postgres, redis, restic, and more).
  • Audit log coverage: the operations that yoink records. A restart by the sidecar is not one of them.
  • Config reference: the fields of a service, which include labels:.

Type a flag, a config field, or a task.