Skip to content
Hands-on IT IT Drills

Day 14: Deploying to a Server

Goal: Bring the stack to a second Docker host via SSH – transfer images without a registry, roll out with up --wait, update to 1.1 and roll back to 1.0 without losing data. · Duration: approx. 90 min · Prerequisite: Day 13

Briefing

  • The “server” is a container notes-server built from docker:29.9-dind with sshd: a Docker daemon of its own with its own images, volumes and networks, reachable only via SSH.
  • A Docker context notes-server sends every docker and docker compose command over SSH to this daemon. Compose itself keeps running on the Mac.
  • Images travel via docker save | docker --context notes-server load – no registry needed.
  • The daemon looks for bind mounts, and therefore also file: secrets and configs, on its own file system. That’s why compose.prod.yaml points to paths on the server.
 Mac                                    notes-server (dind, 127.0.0.1:2222)
 ┌─────────────────────────────────┐ SSH ┌──────────────────────────────────┐
 │ docker --context notes-server   ├────►│ sshd ─► dockerd (own daemon)     │
 │   compose -f compose.yaml       │     │  images notes-*:1.0, :1.1        │
 │           -f compose.prod.yaml  │     │  volume notes_pgdata             │
 │ docker save … | … load ─────────┼────►│  /home/deploy/notes/secrets/…    │
 └─────────────────────────────────┘     └──────────────────────────────────┘

Drill 1: Starting the server

source ~/compose-lab/setup/env.sh
docker compose -p notes down
mkdir -p ~/compose-lab/tag14 && cd ~/compose-lab/tag14
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag13/notes/." .
cp -R "$LAB/code/tag14/server" .

The folder server/ contains the simulated server. The Dockerfile installs sshd and creates the user deploy in the group docker:

cat server/Dockerfile

Expected:

# syntax=docker/dockerfile:1
# notes-server – zweiter Docker-Host mit SSH, nur für das Lab (Tag 14)
FROM docker:29.9-dind
# deploy: Login nur per Schlüssel ("*" statt "!" – Alpine sperrt sonst das Konto)
RUN apk add --no-cache openssh-server=~10.3 \
 && adduser -D -s /bin/sh deploy \
 && addgroup deploy docker \
 && sed -i 's/^deploy:!/deploy:*/' /etc/shadow \
 && printf '%s\n' 'PasswordAuthentication no' 'PermitRootLogin no' \
      'AllowUsers deploy' 'AllowTcpForwarding local' > /etc/ssh/sshd_config.d/lab.conf \
 && ssh-keygen -A
COPY --chmod=755 start.sh /usr/local/bin/start.sh
EXPOSE 22
ENTRYPOINT ["/usr/local/bin/start.sh"]

start.sh takes the public key from /keys, starts sshd and then the Docker daemon – only on the Unix socket, without TCP port 2375/2376:

grep -v '^#' server/start.sh

Expected:

set -eu
install -d -m 700 -o deploy -g deploy /home/deploy/.ssh
install -m 600 -o deploy -g deploy /keys/authorized_keys \
  /home/deploy/.ssh/authorized_keys
/usr/sbin/sshd -e
exec dockerd-entrypoint.sh dockerd --host=unix:///var/run/docker.sock

You generate the key yourself – a throwaway key just for the lab, never in the companion code and never your personal one. Before it is created, it goes into .gitignore, together with known_hosts and the root certificate you fetch in Drill 8:

printf '%s\n' '# Lab-Server: Schlüssel, known_hosts, Root-Zertifikat (Tag 14)' \
  'server/keys/' 'server/known_hosts' 'server/root.crt' >> .gitignore
mkdir -p server/keys
ssh-keygen -q -t ed25519 -N '' -C lab-deploy -f server/keys/id_ed25519
cp server/keys/id_ed25519.pub server/keys/authorized_keys
cat server/compose.yaml

Expected:

# compose.yaml – simulierter Server notes-server (Tag 14), nur für das Lab
name: notes-server

services:
  server:
    build: .
    hostname: notes-server
    privileged: true          # Docker im Container braucht das – nur im Lab!
    ports:
      - "127.0.0.1:${SSH_PORT:-2222}:22"
    volumes:
      - ./keys:/keys:ro
      - server_docker:/var/lib/docker
      - server_home:/home/deploy
    healthcheck:
      test: ["CMD", "docker", "info"]
      interval: 10s
      start_period: 30s
      start_interval: 1s
    restart: unless-stopped

volumes:
  server_docker: {}
  server_home: {}

Two volumes make the server long-lived: server_docker holds the images, containers and volumes of the inner daemon, server_home the files of deploy. The healthcheck only reports healthy once the inner daemon answers.

cd server && docker compose up -d --build --wait && cd ..
docker compose -p notes-server ps --format 'table {{.Service}}\t{{.Status}}\t{{.Ports}}'

Expected (shortened, IDs and times differ on your machine):

 Image notes-server-server Built
 Volume notes-server_server_docker Created
 Volume notes-server_server_home Created
 Container notes-server-server-1 Started
 Container notes-server-server-1 Healthy
SERVICE   STATUS                  PORTS
server    Up 1 second (healthy)   2375-2376/tcp, 127.0.0.1:2222->22/tcp

2375-2376/tcp is just the EXPOSE from the dind image; only SSH on 127.0.0.1:2222 is published.

Security: privileged: true gives the container practically root on the Colima VM. Acceptable for a lab, never on a real host.

Save your progress? Sign up for free

Drill 2: SSH access

The Docker context uses your normal ssh. So ssh has to know the server under a name. The finished ssh_config is in the companion code:

cat server/ssh_config

Expected:

# ssh_config – Lab-Server notes-server (Tag 14), per Include in ~/.ssh/config
Host notes-server
  HostName 127.0.0.1
  Port 2222
  User deploy
  IdentityFile ~/compose-lab/tag14/server/keys/id_ed25519
  IdentitiesOnly yes
  StrictHostKeyChecking accept-new
  UserKnownHostsFile ~/compose-lab/tag14/server/known_hosts

IdentitiesOnly prevents ssh from trying all your other keys, and the separate known_hosts keeps the lab server out of your real one. Include the file with Include at the beginning of ~/.ssh/config – back it up first:

mkdir -p -m 700 ~/.ssh && touch ~/.ssh/config
cp ~/.ssh/config ~/.ssh/config.vor-tag14
{ echo "Include ~/compose-lab/tag14/server/ssh_config"; cat ~/.ssh/config; } \
  > ~/.ssh/config.neu && mv ~/.ssh/config.neu ~/.ssh/config
chmod 600 ~/.ssh/config
head -1 ~/.ssh/config
ssh notes-server 'id; docker version --format "Docker {{.Server.Version}}"; hostname'

Expected:

Include ~/compose-lab/tag14/server/ssh_config
Warning: Permanently added '[127.0.0.1]:2222' (ED25519) to the list of known hosts.
uid=1000(deploy) gid=1000(deploy) groups=1000(deploy),2375(docker)
Docker 29.9.2
notes-server

deploy isn’t root, but may control the daemon via the group docker – on a real server that’s equivalent to root. The inner daemon is version 29.9.2, your Mac talks to 29.5.2 in Colima: two independent Docker hosts.

Save your progress? Sign up for free

Drill 3: The Docker context

docker context create notes-server --description "Lab-Server über SSH" \
  --docker host=ssh://notes-server
docker context ls
docker --context notes-server info --format \
  '{{.Name}} · {{.ServerVersion}} · {{.Containers}} Container · {{.Images}} Images'

Expected (shortened):

notes-server
Successfully created context "notes-server"
NAME           DESCRIPTION                               DOCKER ENDPOINT
colima *       colima                                    unix:///Users/du/.colima/…
default        Current DOCKER_HOST based configuration   unix:///var/run/docker.sock
notes-server   Lab-Server über SSH                       ssh://notes-server
notes-server · 29.9.2 · 0 Container · 0 Images

The asterisk stays on colima: --context only applies to one command. docker context use notes-server would switch over until you switch back – convenient, but that’s exactly how a down -v ends up on the wrong host. That’s why in this book the context is part of every command.

Save your progress? Sign up for free

Drill 4: Transferring images without a registry

The server has no images and can’t get at notes-api – it isn’t in any registry. Build the release locally and send it as a tar stream over SSH:

TAG=1.0 docker compose build api worker web
docker image ls --format '{{.Repository}}:{{.Tag}}  {{.Size}}' | grep ':1.0'
docker save notes-api:1.0 notes-worker:1.0 notes-web:1.0 \
  | docker --context notes-server load
docker --context notes-server image ls --format '{{.Repository}}:{{.Tag}}  {{.Size}}'

Expected (shortened, sizes differ on your machine):

 Image notes-worker:1.0 Built
 Image notes-web:1.0 Built
 Image notes-api:1.0 Built
notes-api:1.0  15.4MB
notes-web:1.0  266MB
notes-worker:1.0  214MB
Loaded image: notes-api:1.0
Loaded image: notes-worker:1.0
Loaded image: notes-web:1.0
notes-api:1.0  15.4MB
notes-web:1.0  266MB
notes-worker:1.0  214MB

docker save writes all layers into one stream, load reads it on the other side. In the lab this takes a few seconds. The server fetches postgres and caddy from Docker Hub itself on the first up.

Save your progress? Sign up for free

Drill 5: The bind mount trap

First attempt: the familiar compose.yaml, just with a different context.

TAG=1.0 docker --context notes-server compose up -d --no-build --wait 2>&1 \
  | grep -m1 'bind source'

Expected (shortened):

 service:db:1 Error response from daemon: invalid mount config for type "bind": bind
source path does not exist: /Users/du/compose-lab/tag14/secrets/db_password.txt

Compose runs on the Mac and resolves ./secrets/db_password.txt to a Mac path. But the daemon on the server creates the bind mount – and /Users/du doesn’t exist there. The same applies to configs: with file: and every ./ bind mount.

The obvious idea would be to deliver secrets via environment: and configs via content:; Compose copies such content into the container at startup. That fails because of the hardening from Day 9 (verified live): cannot create secret "notes_db_password" in read-only service db: file is the sole supported option, and likewise for configs. A read_only container doesn’t accept copied files. So the files go onto the server, and compose.prod.yaml points to the paths there with file::

cat > compose.prod.yaml <<'EOF'
# compose.prod.yaml – server deploy, only via -f together with compose.yaml (Day 14)
# file: sources are bind mounts: the path applies on the Docker host, not on
# the Mac. That's why secrets and configs live on the server under SERVER_DIR.
# (environment:/content: don't work: Compose copies them into the container, and
# that is read_only – verified live.)
secrets:
  api_key: !override
    file: ${SERVER_DIR:-/home/deploy/notes}/secrets/api_key.txt
  db_password: !override
    file: ${SERVER_DIR:-/home/deploy/notes}/secrets/db_password.txt
  db_url: !override
    file: ${SERVER_DIR:-/home/deploy/notes}/secrets/db_url.txt

configs:
  db_init: !override
    file: ${SERVER_DIR:-/home/deploy/notes}/db/init/10-readonly.sql
  caddyfile: !override
    file: ${SERVER_DIR:-/home/deploy/notes}/proxy/Caddyfile
EOF

!override replaces the whole entry from compose.yaml instead of merging it (Day 7). npmrc, grafana_admin and the Prometheus files stay untouched: remotely, nothing is built and the profile observability isn’t started, so nobody needs them there.

Save your progress? Sign up for free

Drill 6: Deploying with up --wait

The five files go to the server via tar over SSH, mode 644 as on Day 10:

tar -c --no-mac-metadata secrets/api_key.txt secrets/db_password.txt \
    secrets/db_url.txt db/init/10-readonly.sql proxy/Caddyfile \
  | ssh notes-server 'mkdir -p notes && tar -x -C notes \
      && find notes -type f -exec chmod 644 {} + && find notes -type f | sort'

Expected:

notes/db/init/10-readonly.sql
notes/proxy/Caddyfile
notes/secrets/api_key.txt
notes/secrets/db_password.txt
notes/secrets/db_url.txt

--no-mac-metadata leaves out the macOS extra data that would otherwise end up on the server as ._ files. Now the deploy:

export TAG=1.0
docker --context notes-server compose -f compose.yaml -f compose.prod.yaml \
  up -d --no-build --wait; echo "Exit: $?"
docker --context notes-server compose ps -a \
  --format 'table {{.Service}}\t{{.Image}}\t{{.Status}}'

Expected (shortened, times differ):

… level=warning msg="secret file notes_db_password does not exist"
… level=warning msg="secret file notes_db_url does not exist"
… level=warning msg="secret file notes_api_key does not exist"
 Container notes-db-1 Healthy
 Container notes-migrate-1 Exited
 Container notes-api-1 Healthy
 Container notes-web-1 Healthy
 Container notes-worker-1 Healthy
 Container notes-proxy-1 Healthy
Exit: 0
SERVICE   IMAGE                  STATUS
api       notes-api:1.0          Up 4 seconds (healthy)
db        postgres:18.6-alpine   Up 7 seconds (healthy)
migrate   notes-api:1.0          Exited (0) 4 seconds ago
proxy     caddy:2.11.7-alpine    Up Less than a second
web       notes-web:1.0          Up 2 seconds (healthy)
worker    notes-worker:1.0       Up 2 seconds

The warnings confirm the problem from before: Compose checks the paths on the Mac, doesn’t find /home/deploy/notes/… there – and carries on anyway, because the daemon on the server does find them. --no-build prevents Compose from wanting to build remotely; --wait is the gate: exit code 0 means “everything healthy”. The same order as locally: db, migrate, api, then worker, web and proxy (Day 5).

The ports are bound to 127.0.0.1 of the server. You can check them via SSH; in the container there’s only BusyBox wget (127.0.0.1 instead of localhost, otherwise wget tries IPv6 first):

ssh notes-server 'api=http://127.0.0.1:8080; wget -qO- $api/version
  wget -qO- --header "X-API-Key: $(cat notes/secrets/api_key.txt)" \
    --post-data "{\"text\":\"Version 1.0 auf dem Server\"}" $api/notes; echo
  wget -qO- $api/notes'; echo
docker compose ps -q | wc -l

Expected:

1.0
{"id":"15bba7fe-…","text":"Version 1.0 auf dem Server"}
[{"id":"15bba7fe-…","text":"Version 1.0 auf dem Server"}]
       0

Nothing runs locally: the whole stack lives on the server.

Tip: Interpolation happens on the Mac. TAG, API_PORT and all other variables come from your shell or the local .env – not from a file on the server.

Save your progress? Sign up for free

Drill 7: Update and rollback

Release 1.1: build, transfer, roll out.

export TAG=1.1
docker compose build api worker web
docker save notes-api:1.1 notes-worker:1.1 notes-web:1.1 \
  | docker --context notes-server load
docker --context notes-server compose -f compose.yaml -f compose.prod.yaml \
  up -d --no-build --wait; echo "Exit: $?"
ssh notes-server 'wget -qO- 127.0.0.1:8080/version; wget -qO- 127.0.0.1:8080/notes'
echo

Expected (shortened):

Loaded image: notes-api:1.1
Loaded image: notes-worker:1.1
Loaded image: notes-web:1.1
 Container notes-proxy-1 Running
 Container notes-db-1 Running
 Container notes-migrate-1 Recreated
 Container notes-api-1 Recreated
 Container notes-web-1 Recreated
 Container notes-worker-1 Recreated
 Container notes-db-1 Healthy
 Container notes-api-1 Healthy
 Container notes-web-1 Healthy
 Container notes-proxy-1 Healthy
 Container notes-worker-1 Healthy
Exit: 0
1.1
[{"id":"15bba7fe-…","text":"Version 1.0 auf dem Server"}]

Only the services with a new image are replaced; db and proxy keep running, and the note (“Version 1.0 on the server”) is still in the volume notes_pgdata on the server. The rollback is just a different TAG – the 1.0 image is still there:

SECONDS=0
TAG=1.0 docker --context notes-server compose -f compose.yaml -f compose.prod.yaml \
  up -d --no-build --wait >/dev/null 2>&1; echo "Exit: $? nach ${SECONDS}s"
ssh notes-server wget -qO- 127.0.0.1:8080/version

Expected (times differ):

Exit: 0 nach 6s
1.0

And what if you roll out a release you never transferred?

TAG=1.2 docker --context notes-server compose -f compose.yaml -f compose.prod.yaml \
  up -d --no-build --wait 2>&1 | grep -E 'denied|No such' | head -3
ssh notes-server wget -qO- 127.0.0.1:8080/version
TAG=1.0 docker --context notes-server compose -f compose.yaml -f compose.prod.yaml \
  up -d --no-build --wait >/dev/null 2>&1; echo "zurück: $?"

Expected (shortened):

 Image notes-api:1.2 pull access denied for notes-api, repository does not exist or …
 Image notes-web:1.2 pull access denied for notes-web, repository does not exist or …
 Image notes-worker:1.2 pull access denied for notes-worker, repository does not exis…
1.0
zurück: 0

Compose tries to pull the missing image, fails and aborts. api 1.0 keeps running; only migrate had already been removed, and the last command recreates it.

Save your progress? Sign up for free

Drill 8: Looking from the outside

The server’s proxy listens on its 127.0.0.1:8443. An SSH tunnel brings it to your Mac – locally as port 8444, so it doesn’t get in the way of Day 11. The root certificate is a different one than on Day 11: the server has its own Caddy CA in the volume notes_caddy_data.

docker --context notes-server compose cp \
  proxy:/data/caddy/pki/authorities/local/root.crt server/root.crt
ssh -f -N -o ExitOnForwardFailure=yes -L 8444:127.0.0.1:8443 notes-server
curl -s --resolve notes.localhost:8444:127.0.0.1 --cacert server/root.crt \
  https://notes.localhost:8444/api/notes; echo
pkill -f 'L 8444:127.0.0.1:8443'; echo "Tunnel zu: $?"

Expected (shortened):

 notes-proxy-1 Copied notes-proxy-1:/data/caddy/pki/authorities/local/root.crt to …
[{"id":"15bba7fe-…","text":"Version 1.0 auf dem Server"}]
Tunnel zu: 0

-f -N sends ssh into the background without a command; ExitOnForwardFailure makes it fail if the local port is taken. The server only allows local forwarding (AllowTcpForwarding local in the Dockerfile) – otherwise Alpine turns it off completely.

Save your progress? Sign up for free

Checkpoint

SSH with the lab key:

cd ~/compose-lab/tag14
ssh -o BatchMode=yes notes-server true; echo "Exit: $?"

Expected:

Exit: 0

The context points to the server:

docker context inspect notes-server --format '{{.Endpoints.docker.Host}}'

Expected:

ssh://notes-server

Both versions are on the server:

docker --context notes-server image ls notes-api

Expected (shortened, IDs and sizes differ):

IMAGE           ID             DISK USAGE   CONTENT SIZE   EXTRA
notes-api:1.0   cd602c157fa4       16.4MB          4.5MB   U
notes-api:1.1   fa99388fee06       16.4MB          4.5MB

Deploy version 1.0 on the server:

TAG=1.0 docker --context notes-server compose -f compose.yaml -f compose.prod.yaml \
  up -d --no-build --wait
ssh notes-server wget -qO- 127.0.0.1:8080/version; echo

Expected (shortened):

 Container notes-db-1 Healthy
…
 Container notes-proxy-1 Healthy
 Container notes-web-1 Healthy
1.0

The note survived the update and the rollback:

ssh notes-server wget -qO- 127.0.0.1:8080/notes | jq -r '.[].text'

Expected (shortened):

Version 1.0 auf dem Server

No notes stack runs locally:

docker compose ps -q | wc -l

Expected:

0

You’ll find the complete state in code/tag14/notes/ (compose.prod.yaml) and code/tag14/server/.

Challenge

Roll out version 1.1 on the server. Then try 1.2 with a --wait-timeout that’s too tight; if the stack doesn’t become healthy, roll back to 1.1 by hand.

Hint: DOCKER_CONTEXT and COMPOSE_FILE save you --context and the two -f on every command. Building happens locally, so build first, then switch.

Solution

Build locally and transfer to the server:

cd ~/compose-lab/tag14
unset TAG
TAG=1.1 docker compose build --quiet api worker web
docker save notes-api:1.1 notes-worker:1.1 notes-web:1.1 \
  | docker --context notes-server load

Expected:

 Image notes-api:1.1 Built
 Image notes-worker:1.1 Built
 Image notes-web:1.1 Built
Loaded image: notes-api:1.1
Loaded image: notes-worker:1.1
Loaded image: notes-web:1.1

From now on every docker command targets the server, with both Compose files:

export DOCKER_CONTEXT=notes-server COMPOSE_FILE=compose.yaml:compose.prod.yaml
TAG=1.1 docker compose up -d --no-build --wait
ssh notes-server wget -qO- 127.0.0.1:8080/version; echo

Expected (shortened):

 Container notes-migrate-1 Recreate
time="…" level=warning msg="secret file notes_db_url does not exist"
…
 Container notes-proxy-1 Healthy
1.1

The warnings secret file … does not exist come from your machine: the files aren’t there, but they are on the server under SERVER_DIR. Now 1.2, with one second of patience:

unset DOCKER_CONTEXT COMPOSE_FILE
TAG=1.2 docker compose build --quiet api worker web
docker save notes-api:1.2 notes-worker:1.2 notes-web:1.2 \
  | docker --context notes-server load
export DOCKER_CONTEXT=notes-server COMPOSE_FILE=compose.yaml:compose.prod.yaml
TAG=1.2 docker compose up -d --no-build --wait --wait-timeout 1; echo "Exit: $?"

Expected (shortened):

…
Loaded image: notes-web:1.2
…
 Container notes-api-1 Started
 Container notes-api-1 Error dependency api failed to start
timeout waiting for dependencies
Exit: 1

One second isn’t enough for api to become healthy. Back to 1.1:

TAG=1.1 docker compose up -d --no-build --wait
ssh notes-server wget -qO- 127.0.0.1:8080/version; echo
unset DOCKER_CONTEXT COMPOSE_FILE

Expected (shortened):

 Container notes-api-1 Healthy
…
 Container notes-proxy-1 Healthy
1.1

To wrap up the book, you tear down the server completely – context, server with both volumes, include line, local release images:

cd ~/compose-lab/tag14
docker context rm notes-server
(cd server && docker compose down -v)
sed -i '' '\#compose-lab/tag14/server/ssh_config#d' ~/.ssh/config
diff ~/.ssh/config ~/.ssh/config.vor-tag14
docker image rm notes-api:1.0 notes-worker:1.0 notes-web:1.0 notes-api:1.1 \
  notes-worker:1.1 notes-web:1.1 notes-api:1.2 notes-worker:1.2 notes-web:1.2 \
  notes-server-server

Expected (shortened): diff without output, so ~/.ssh/config is as before.

notes-server
…
Untagged: notes-api:1.0
…
Untagged: notes-server-server:latest
Deleted: sha256:…

Save your progress? Sign up for free

Enterprise Note

  • Instead of save | load, a registry (Harbor, GitLab, GHCR) with signed images and deploy by digest. save | load remains the way for air-gapped networks without a registry.
  • Whoever is in the group docker is root on the host. Restrict deploy users via ForceCommand or authorized_keys options, keys from a secret store, distribute host keys in advance instead of accept-new.
  • Secret files on the server with mode 600 and the matching UID, managed by configuration management (Ansible) or a secret manager (“OpenBao secrets hands-on”), not by a hand-rolled tar.
  • Compose on one host means: no rolling update, no failover. A short outage during recreate is built in; whoever needs more ends up with Swarm or Kubernetes.
  • Rollback requires backward-compatible migrations. A release that restructures the table no longer works with the old image – back up before every deploy (Day 3).