Blue-green deployment with Docker and Caddy, by hand

Why a plain restart drops requests

Stopping the old container and starting the new one leaves a gap: the old one is gone and the new one is still booting. Requests that arrive in between get errors from the proxy. Blue-green closes the gap by running the new version next to the old one and moving traffic only once the new one is ready.

The sequence

1. 01Start the new colour next to the live one, on the same network, with no published port.
2. 02Wait until Docker reports it healthy. Nothing reaches it before that.
3. 03Write the new proxy configuration, validate it, then switch with a graceful reload.
4. 04Check the app through the proxy, the way a user arrives.
5. 05Only then remove the old colour.

A health check that tests the app

The whole method rests on one signal: Docker’s health status. A check that only proves the port is open will happily pass an app that returns errors on every request. Point it at an endpoint that does real work — one that queries the database, for example.

HEALTHCHECK --interval=5s --timeout=3s --start-period=20s --retries=3 \
  CMD wget -qO- http://127.0.0.1:3000/healthz || exit 1

wget is available in Alpine-based images. In a slim or distroless image, use what the image has, or build a small check command into the application.

The setup it assumes

Caddy runs as a container named caddy on a Docker network called web, with a configuration directory mounted at /etc/caddy. Mount the directory rather than the single file: the script replaces the file, and a single-file bind mount would keep showing Caddy the old one.

The live container is called app-blue or app-green, and the Caddyfile points at it by that name. The very first time, start app-blue by hand.

app.example.com {
	reverse_proxy app-blue:3000
}

The script

We ran it against a local Caddy and registry through four cases: a good release, one that never becomes healthy, one that is healthy but fails the smoke test, and another good release afterwards.

#!/usr/bin/env bash
# Usage: ./deploy.sh ghcr.io/example/app:1.5.0
set -Eeuo pipefail

IMAGE=${1:?usage: deploy.sh IMAGE}
NETWORK=web
PROXY=caddy                       # the Caddy container
CONF=/opt/caddy/conf              # mounted at /etc/caddy inside it
ENV_FILE=/opt/app/.env
SMOKE_URL=https://app.example.com/healthz

wait_healthy() {
  local state
  for _ in $(seq 1 60); do
    state=$(docker inspect -f '{{.State.Running}} {{.State.Health.Status}}' "$1")
    case "$state" in
      "true healthy") return 0 ;;
      false* | *unhealthy) return 1 ;;
    esac
    sleep 2
  done
  return 1
}

live=$(grep -m1 -oE 'app-(blue|green)' "$CONF/Caddyfile") ||
  { echo "The Caddyfile must point at app-blue or app-green." >&2; exit 1; }
next=$([ "$live" = app-blue ] && echo app-green || echo app-blue)
switched=0

rollback() {
  echo "Deploy failed. $live keeps serving." >&2
  if [ "$switched" = 1 ]; then
    cp "$CONF/Caddyfile.previous" "$CONF/Caddyfile"
    docker exec "$PROXY" caddy reload --config /etc/caddy/Caddyfile --adapter caddyfile || true
  fi
  rm -f "$CONF/Caddyfile.next"
  docker rm -f "$next" >/dev/null 2>&1 || true
}
trap rollback ERR

# 1. Start the new colour next to the live one.
docker pull "$IMAGE"
docker rm -f "$next" >/dev/null 2>&1 || true
docker run -d --name "$next" --network "$NETWORK" \
  --restart unless-stopped --env-file "$ENV_FILE" "$IMAGE" >/dev/null

# 2. Nothing reaches it until Docker reports it healthy.
wait_healthy "$next"

# 3. Validate the new proxy config before it replaces the running one.
cp "$CONF/Caddyfile" "$CONF/Caddyfile.previous"
sed "s/$live:/$next:/" "$CONF/Caddyfile.previous" > "$CONF/Caddyfile.next"
docker exec "$PROXY" caddy validate --config /etc/caddy/Caddyfile.next --adapter caddyfile

# 4. Switch. Caddy reloads gracefully; open connections finish.
switched=1
mv "$CONF/Caddyfile.next" "$CONF/Caddyfile"
docker exec "$PROXY" caddy reload --config /etc/caddy/Caddyfile --adapter caddyfile

# 5. Check through the proxy, the way a user arrives.
curl -fsS --retry 3 --retry-delay 1 "$SMOKE_URL" >/dev/null

# 6. Only now retire the old colour.
trap - ERR
docker rm -f "$live" >/dev/null
echo "Live: $next ($IMAGE)"

The failure path is the point

Most hand-written versions get the happy path right and leave a mess on failure. This one decides what to undo based on how far it got.

If the new container never becomes healthy, the proxy was never touched. The script removes the candidate and exits, and the live colour keeps serving as if nothing had happened.

If it fails after the switch — the reload errors, or the smoke test through the proxy fails — it puts the previous Caddyfile back, reloads, and removes the candidate. The old colour was never stopped, so it is serving again at once.

The old container is removed only after the smoke test passes. Until that line runs, a rollback is always possible.

Undoing a release that passed every check but turns out to be wrong is the same script with the previous image tag. That is one more blue-green switch, so it does not drop requests either.

Two things the script cannot do for you

For a few seconds both versions run against the same database. Schema migrations therefore have to be ones the old version survives: add a column before the code that needs it, and drop the old one a release later.

If you add volumes to the docker run line, both colours mount them at the same time. That is fine for most web apps and wrong for anything that assumes it is the only writer, such as a single-process job scheduler.

The automated version

MihraOps runs this sequence for applications published on a domain: it starts the candidate with the same configuration, waits for it to report healthy, moves the proxy, and undoes each step in reverse if the next one fails. A container that never becomes healthy fails the release before any traffic reaches it.

Blue-green releases are part of Pro. The docs page describes the exact sequence.