Day 14: Deploying to a Server
Goal: Bring the stack to a second Docker host via SSH – transfer images without a registry, roll out with
up --wait, update to 1.1 and roll back to 1.0 without losing data. · Duration: approx. 90 min · Prerequisite: Day 13
Briefing
- The “server” is a container
notes-serverbuilt fromdocker:29.9-dindwith sshd: a Docker daemon of its own with its own images, volumes and networks, reachable only via SSH. - A Docker context
notes-serversends everydockeranddocker composecommand over SSH to this daemon. Compose itself keeps running on the Mac. - Images travel via
docker save | docker --context notes-server load– no registry needed. - The daemon looks for bind mounts, and therefore also
file:secrets and configs, on its own file system. That’s whycompose.prod.yamlpoints to paths on the server.
Mac notes-server (dind, 127.0.0.1:2222)
┌─────────────────────────────────┐ SSH ┌──────────────────────────────────┐
│ docker --context notes-server ├────►│ sshd ─► dockerd (own daemon) │
│ compose -f compose.yaml │ │ images notes-*:1.0, :1.1 │
│ -f compose.prod.yaml │ │ volume notes_pgdata │
│ docker save … | … load ─────────┼────►│ /home/deploy/notes/secrets/… │
└─────────────────────────────────┘ └──────────────────────────────────┘
Drill 1: Starting the server
source ~/compose-lab/setup/env.sh
docker compose -p notes down
mkdir -p ~/compose-lab/tag14 && cd ~/compose-lab/tag14
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag13/notes/." .
cp -R "$LAB/code/tag14/server" .The folder server/ contains the simulated server. The
Dockerfile installs sshd and creates the user deploy in the
group docker:
cat server/DockerfileExpected:
# syntax=docker/dockerfile:1
# notes-server – zweiter Docker-Host mit SSH, nur für das Lab (Tag 14)
FROM docker:29.9-dind
# deploy: Login nur per Schlüssel ("*" statt "!" – Alpine sperrt sonst das Konto)
RUN apk add --no-cache openssh-server=~10.3 \
&& adduser -D -s /bin/sh deploy \
&& addgroup deploy docker \
&& sed -i 's/^deploy:!/deploy:*/' /etc/shadow \
&& printf '%s\n' 'PasswordAuthentication no' 'PermitRootLogin no' \
'AllowUsers deploy' 'AllowTcpForwarding local' > /etc/ssh/sshd_config.d/lab.conf \
&& ssh-keygen -A
COPY --chmod=755 start.sh /usr/local/bin/start.sh
EXPOSE 22
ENTRYPOINT ["/usr/local/bin/start.sh"]start.sh takes the public key from /keys,
starts sshd and then the Docker daemon – only on the Unix socket,
without TCP port 2375/2376:
grep -v '^#' server/start.shExpected:
set -eu
install -d -m 700 -o deploy -g deploy /home/deploy/.ssh
install -m 600 -o deploy -g deploy /keys/authorized_keys \
/home/deploy/.ssh/authorized_keys
/usr/sbin/sshd -e
exec dockerd-entrypoint.sh dockerd --host=unix:///var/run/docker.sock
You generate the key yourself – a throwaway key just for the lab,
never in the companion code and never your personal one. Before it is
created, it goes into .gitignore, together with
known_hosts and the root certificate you fetch in Drill
8:
printf '%s\n' '# Lab-Server: Schlüssel, known_hosts, Root-Zertifikat (Tag 14)' \
'server/keys/' 'server/known_hosts' 'server/root.crt' >> .gitignore
mkdir -p server/keys
ssh-keygen -q -t ed25519 -N '' -C lab-deploy -f server/keys/id_ed25519
cp server/keys/id_ed25519.pub server/keys/authorized_keys
cat server/compose.yamlExpected:
# compose.yaml – simulierter Server notes-server (Tag 14), nur für das Lab
name: notes-server
services:
server:
build: .
hostname: notes-server
privileged: true # Docker im Container braucht das – nur im Lab!
ports:
- "127.0.0.1:${SSH_PORT:-2222}:22"
volumes:
- ./keys:/keys:ro
- server_docker:/var/lib/docker
- server_home:/home/deploy
healthcheck:
test: ["CMD", "docker", "info"]
interval: 10s
start_period: 30s
start_interval: 1s
restart: unless-stopped
volumes:
server_docker: {}
server_home: {}Two volumes make the server long-lived: server_docker
holds the images, containers and volumes of the inner daemon,
server_home the files of deploy. The
healthcheck only reports healthy once the inner daemon
answers.
cd server && docker compose up -d --build --wait && cd ..
docker compose -p notes-server ps --format 'table {{.Service}}\t{{.Status}}\t{{.Ports}}'Expected (shortened, IDs and times differ on your machine):
Image notes-server-server Built
Volume notes-server_server_docker Created
Volume notes-server_server_home Created
Container notes-server-server-1 Started
Container notes-server-server-1 Healthy
SERVICE STATUS PORTS
server Up 1 second (healthy) 2375-2376/tcp, 127.0.0.1:2222->22/tcp
2375-2376/tcp is just the EXPOSE from the
dind image; only SSH on 127.0.0.1:2222 is published.
Security:
privileged: truegives the container practically root on the Colima VM. Acceptable for a lab, never on a real host.
Save your progress? Sign up for free
Drill 2: SSH access
The Docker context uses your normal ssh. So ssh has to
know the server under a name. The finished ssh_config is in
the companion code:
cat server/ssh_configExpected:
# ssh_config – Lab-Server notes-server (Tag 14), per Include in ~/.ssh/config
Host notes-server
HostName 127.0.0.1
Port 2222
User deploy
IdentityFile ~/compose-lab/tag14/server/keys/id_ed25519
IdentitiesOnly yes
StrictHostKeyChecking accept-new
UserKnownHostsFile ~/compose-lab/tag14/server/known_hosts
IdentitiesOnly prevents ssh from trying all your other
keys, and the separate known_hosts keeps the lab server out
of your real one. Include the file with Include at the
beginning of ~/.ssh/config – back it up
first:
mkdir -p -m 700 ~/.ssh && touch ~/.ssh/config
cp ~/.ssh/config ~/.ssh/config.vor-tag14
{ echo "Include ~/compose-lab/tag14/server/ssh_config"; cat ~/.ssh/config; } \
> ~/.ssh/config.neu && mv ~/.ssh/config.neu ~/.ssh/config
chmod 600 ~/.ssh/config
head -1 ~/.ssh/config
ssh notes-server 'id; docker version --format "Docker {{.Server.Version}}"; hostname'Expected:
Include ~/compose-lab/tag14/server/ssh_config
Warning: Permanently added '[127.0.0.1]:2222' (ED25519) to the list of known hosts.
uid=1000(deploy) gid=1000(deploy) groups=1000(deploy),2375(docker)
Docker 29.9.2
notes-server
deploy isn’t root, but may control the daemon via the
group docker – on a real server that’s equivalent to root.
The inner daemon is version 29.9.2, your Mac talks to 29.5.2 in Colima:
two independent Docker hosts.
Save your progress? Sign up for free
Drill 3: The Docker context
docker context create notes-server --description "Lab-Server über SSH" \
--docker host=ssh://notes-server
docker context ls
docker --context notes-server info --format \
'{{.Name}} · {{.ServerVersion}} · {{.Containers}} Container · {{.Images}} Images'Expected (shortened):
notes-server
Successfully created context "notes-server"
NAME DESCRIPTION DOCKER ENDPOINT
colima * colima unix:///Users/du/.colima/…
default Current DOCKER_HOST based configuration unix:///var/run/docker.sock
notes-server Lab-Server über SSH ssh://notes-server
notes-server · 29.9.2 · 0 Container · 0 Images
The asterisk stays on colima: --context
only applies to one command.
docker context use notes-server would switch over until you
switch back – convenient, but that’s exactly how a down -v
ends up on the wrong host. That’s why in this book the context is part
of every command.
Save your progress? Sign up for free
Drill 4: Transferring images without a registry
The server has no images and can’t get at notes-api – it
isn’t in any registry. Build the release locally and send it as a tar
stream over SSH:
TAG=1.0 docker compose build api worker web
docker image ls --format '{{.Repository}}:{{.Tag}} {{.Size}}' | grep ':1.0'
docker save notes-api:1.0 notes-worker:1.0 notes-web:1.0 \
| docker --context notes-server load
docker --context notes-server image ls --format '{{.Repository}}:{{.Tag}} {{.Size}}'Expected (shortened, sizes differ on your machine):
Image notes-worker:1.0 Built
Image notes-web:1.0 Built
Image notes-api:1.0 Built
notes-api:1.0 15.4MB
notes-web:1.0 266MB
notes-worker:1.0 214MB
Loaded image: notes-api:1.0
Loaded image: notes-worker:1.0
Loaded image: notes-web:1.0
notes-api:1.0 15.4MB
notes-web:1.0 266MB
notes-worker:1.0 214MB
docker save writes all layers into one stream,
load reads it on the other side. In the lab this takes a
few seconds. The server fetches postgres and
caddy from Docker Hub itself on the first
up.
Save your progress? Sign up for free
Drill 5: The bind mount trap
First attempt: the familiar compose.yaml, just with a
different context.
TAG=1.0 docker --context notes-server compose up -d --no-build --wait 2>&1 \
| grep -m1 'bind source'Expected (shortened):
service:db:1 Error response from daemon: invalid mount config for type "bind": bind
source path does not exist: /Users/du/compose-lab/tag14/secrets/db_password.txt
Compose runs on the Mac and resolves
./secrets/db_password.txt to a Mac path. But the daemon on
the server creates the bind mount – and /Users/du doesn’t
exist there. The same applies to configs: with
file: and every ./ bind mount.
The obvious idea would be to deliver secrets via
environment: and configs via content:; Compose
copies such content into the container at startup. That fails because of
the hardening from Day 9 (verified live):
cannot create secret "notes_db_password" in read-only service db: file is the sole supported option,
and likewise for configs. A read_only container doesn’t
accept copied files. So the files go onto the server, and
compose.prod.yaml points to the paths there with
file::
cat > compose.prod.yaml <<'EOF'
# compose.prod.yaml – server deploy, only via -f together with compose.yaml (Day 14)
# file: sources are bind mounts: the path applies on the Docker host, not on
# the Mac. That's why secrets and configs live on the server under SERVER_DIR.
# (environment:/content: don't work: Compose copies them into the container, and
# that is read_only – verified live.)
secrets:
api_key: !override
file: ${SERVER_DIR:-/home/deploy/notes}/secrets/api_key.txt
db_password: !override
file: ${SERVER_DIR:-/home/deploy/notes}/secrets/db_password.txt
db_url: !override
file: ${SERVER_DIR:-/home/deploy/notes}/secrets/db_url.txt
configs:
db_init: !override
file: ${SERVER_DIR:-/home/deploy/notes}/db/init/10-readonly.sql
caddyfile: !override
file: ${SERVER_DIR:-/home/deploy/notes}/proxy/Caddyfile
EOF!override replaces the whole entry from
compose.yaml instead of merging it (Day 7).
npmrc, grafana_admin and the Prometheus files
stay untouched: remotely, nothing is built and the profile
observability isn’t started, so nobody needs them
there.
Save your progress? Sign up for free
Drill 6: Deploying with
up --wait
The five files go to the server via tar over SSH, mode
644 as on Day 10:
tar -c --no-mac-metadata secrets/api_key.txt secrets/db_password.txt \
secrets/db_url.txt db/init/10-readonly.sql proxy/Caddyfile \
| ssh notes-server 'mkdir -p notes && tar -x -C notes \
&& find notes -type f -exec chmod 644 {} + && find notes -type f | sort'Expected:
notes/db/init/10-readonly.sql
notes/proxy/Caddyfile
notes/secrets/api_key.txt
notes/secrets/db_password.txt
notes/secrets/db_url.txt
--no-mac-metadata leaves out the macOS extra data that
would otherwise end up on the server as ._ files. Now the
deploy:
export TAG=1.0
docker --context notes-server compose -f compose.yaml -f compose.prod.yaml \
up -d --no-build --wait; echo "Exit: $?"
docker --context notes-server compose ps -a \
--format 'table {{.Service}}\t{{.Image}}\t{{.Status}}'Expected (shortened, times differ):
… level=warning msg="secret file notes_db_password does not exist"
… level=warning msg="secret file notes_db_url does not exist"
… level=warning msg="secret file notes_api_key does not exist"
Container notes-db-1 Healthy
Container notes-migrate-1 Exited
Container notes-api-1 Healthy
Container notes-web-1 Healthy
Container notes-worker-1 Healthy
Container notes-proxy-1 Healthy
Exit: 0
SERVICE IMAGE STATUS
api notes-api:1.0 Up 4 seconds (healthy)
db postgres:18.6-alpine Up 7 seconds (healthy)
migrate notes-api:1.0 Exited (0) 4 seconds ago
proxy caddy:2.11.7-alpine Up Less than a second
web notes-web:1.0 Up 2 seconds (healthy)
worker notes-worker:1.0 Up 2 seconds
The warnings confirm the problem from before: Compose checks the
paths on the Mac, doesn’t find /home/deploy/notes/… there –
and carries on anyway, because the daemon on the server does find them.
--no-build prevents Compose from wanting to build remotely;
--wait is the gate: exit code 0 means “everything healthy”.
The same order as locally: db, migrate, api, then worker, web and proxy
(Day 5).
The ports are bound to 127.0.0.1 of the
server. You can check them via SSH; in the container there’s
only BusyBox wget (127.0.0.1 instead of
localhost, otherwise wget tries IPv6 first):
ssh notes-server 'api=http://127.0.0.1:8080; wget -qO- $api/version
wget -qO- --header "X-API-Key: $(cat notes/secrets/api_key.txt)" \
--post-data "{\"text\":\"Version 1.0 auf dem Server\"}" $api/notes; echo
wget -qO- $api/notes'; echo
docker compose ps -q | wc -lExpected:
1.0
{"id":"15bba7fe-…","text":"Version 1.0 auf dem Server"}
[{"id":"15bba7fe-…","text":"Version 1.0 auf dem Server"}]
0
Nothing runs locally: the whole stack lives on the server.
Tip: Interpolation happens on the Mac.
TAG,API_PORTand all other variables come from your shell or the local.env– not from a file on the server.
Save your progress? Sign up for free
Drill 7: Update and rollback
Release 1.1: build, transfer, roll out.
export TAG=1.1
docker compose build api worker web
docker save notes-api:1.1 notes-worker:1.1 notes-web:1.1 \
| docker --context notes-server load
docker --context notes-server compose -f compose.yaml -f compose.prod.yaml \
up -d --no-build --wait; echo "Exit: $?"
ssh notes-server 'wget -qO- 127.0.0.1:8080/version; wget -qO- 127.0.0.1:8080/notes'
echoExpected (shortened):
Loaded image: notes-api:1.1
Loaded image: notes-worker:1.1
Loaded image: notes-web:1.1
Container notes-proxy-1 Running
Container notes-db-1 Running
Container notes-migrate-1 Recreated
Container notes-api-1 Recreated
Container notes-web-1 Recreated
Container notes-worker-1 Recreated
Container notes-db-1 Healthy
Container notes-api-1 Healthy
Container notes-web-1 Healthy
Container notes-proxy-1 Healthy
Container notes-worker-1 Healthy
Exit: 0
1.1
[{"id":"15bba7fe-…","text":"Version 1.0 auf dem Server"}]
Only the services with a new image are replaced; db and proxy keep
running, and the note (“Version 1.0 on the server”) is still in the
volume notes_pgdata on the server. The rollback is just a
different TAG – the 1.0 image is still there:
SECONDS=0
TAG=1.0 docker --context notes-server compose -f compose.yaml -f compose.prod.yaml \
up -d --no-build --wait >/dev/null 2>&1; echo "Exit: $? nach ${SECONDS}s"
ssh notes-server wget -qO- 127.0.0.1:8080/versionExpected (times differ):
Exit: 0 nach 6s
1.0
And what if you roll out a release you never transferred?
TAG=1.2 docker --context notes-server compose -f compose.yaml -f compose.prod.yaml \
up -d --no-build --wait 2>&1 | grep -E 'denied|No such' | head -3
ssh notes-server wget -qO- 127.0.0.1:8080/version
TAG=1.0 docker --context notes-server compose -f compose.yaml -f compose.prod.yaml \
up -d --no-build --wait >/dev/null 2>&1; echo "zurück: $?"Expected (shortened):
Image notes-api:1.2 pull access denied for notes-api, repository does not exist or …
Image notes-web:1.2 pull access denied for notes-web, repository does not exist or …
Image notes-worker:1.2 pull access denied for notes-worker, repository does not exis…
1.0
zurück: 0
Compose tries to pull the missing image, fails and aborts. api 1.0 keeps running; only migrate had already been removed, and the last command recreates it.
Save your progress? Sign up for free
Drill 8: Looking from the outside
The server’s proxy listens on its 127.0.0.1:8443. An SSH
tunnel brings it to your Mac – locally as port 8444, so it doesn’t get
in the way of Day 11. The root certificate is a different one than on
Day 11: the server has its own Caddy CA in the volume
notes_caddy_data.
docker --context notes-server compose cp \
proxy:/data/caddy/pki/authorities/local/root.crt server/root.crt
ssh -f -N -o ExitOnForwardFailure=yes -L 8444:127.0.0.1:8443 notes-server
curl -s --resolve notes.localhost:8444:127.0.0.1 --cacert server/root.crt \
https://notes.localhost:8444/api/notes; echo
pkill -f 'L 8444:127.0.0.1:8443'; echo "Tunnel zu: $?"Expected (shortened):
notes-proxy-1 Copied notes-proxy-1:/data/caddy/pki/authorities/local/root.crt to …
[{"id":"15bba7fe-…","text":"Version 1.0 auf dem Server"}]
Tunnel zu: 0
-f -N sends ssh into the background without a command;
ExitOnForwardFailure makes it fail if the local port is
taken. The server only allows local forwarding
(AllowTcpForwarding local in the Dockerfile) – otherwise
Alpine turns it off completely.
Save your progress? Sign up for free
Checkpoint
SSH with the lab key:
cd ~/compose-lab/tag14
ssh -o BatchMode=yes notes-server true; echo "Exit: $?"Expected:
Exit: 0
The context points to the server:
docker context inspect notes-server --format '{{.Endpoints.docker.Host}}'Expected:
ssh://notes-server
Both versions are on the server:
docker --context notes-server image ls notes-apiExpected (shortened, IDs and sizes differ):
IMAGE ID DISK USAGE CONTENT SIZE EXTRA
notes-api:1.0 cd602c157fa4 16.4MB 4.5MB U
notes-api:1.1 fa99388fee06 16.4MB 4.5MB
Deploy version 1.0 on the server:
TAG=1.0 docker --context notes-server compose -f compose.yaml -f compose.prod.yaml \
up -d --no-build --wait
ssh notes-server wget -qO- 127.0.0.1:8080/version; echoExpected (shortened):
Container notes-db-1 Healthy
…
Container notes-proxy-1 Healthy
Container notes-web-1 Healthy
1.0
The note survived the update and the rollback:
ssh notes-server wget -qO- 127.0.0.1:8080/notes | jq -r '.[].text'Expected (shortened):
Version 1.0 auf dem Server
No notes stack runs locally:
docker compose ps -q | wc -lExpected:
0
You’ll find the complete state in code/tag14/notes/
(compose.prod.yaml) and
code/tag14/server/.
Challenge
Roll out version 1.1 on the server. Then try 1.2 with a
--wait-timeout that’s too tight; if the stack doesn’t
become healthy, roll back to 1.1 by hand.
Hint: DOCKER_CONTEXT and
COMPOSE_FILE save you --context and the two
-f on every command. Building happens locally, so build
first, then switch.
Solution
Build locally and transfer to the server:
cd ~/compose-lab/tag14
unset TAG
TAG=1.1 docker compose build --quiet api worker web
docker save notes-api:1.1 notes-worker:1.1 notes-web:1.1 \
| docker --context notes-server loadExpected:
Image notes-api:1.1 Built
Image notes-worker:1.1 Built
Image notes-web:1.1 Built
Loaded image: notes-api:1.1
Loaded image: notes-worker:1.1
Loaded image: notes-web:1.1
From now on every docker command targets the server,
with both Compose files:
export DOCKER_CONTEXT=notes-server COMPOSE_FILE=compose.yaml:compose.prod.yaml
TAG=1.1 docker compose up -d --no-build --wait
ssh notes-server wget -qO- 127.0.0.1:8080/version; echoExpected (shortened):
Container notes-migrate-1 Recreate
time="…" level=warning msg="secret file notes_db_url does not exist"
…
Container notes-proxy-1 Healthy
1.1
The warnings secret file … does not exist come from your
machine: the files aren’t there, but they are on the server under
SERVER_DIR. Now 1.2, with one second of patience:
unset DOCKER_CONTEXT COMPOSE_FILE
TAG=1.2 docker compose build --quiet api worker web
docker save notes-api:1.2 notes-worker:1.2 notes-web:1.2 \
| docker --context notes-server load
export DOCKER_CONTEXT=notes-server COMPOSE_FILE=compose.yaml:compose.prod.yaml
TAG=1.2 docker compose up -d --no-build --wait --wait-timeout 1; echo "Exit: $?"Expected (shortened):
…
Loaded image: notes-web:1.2
…
Container notes-api-1 Started
Container notes-api-1 Error dependency api failed to start
timeout waiting for dependencies
Exit: 1
One second isn’t enough for api to become healthy. Back to 1.1:
TAG=1.1 docker compose up -d --no-build --wait
ssh notes-server wget -qO- 127.0.0.1:8080/version; echo
unset DOCKER_CONTEXT COMPOSE_FILEExpected (shortened):
Container notes-api-1 Healthy
…
Container notes-proxy-1 Healthy
1.1
To wrap up the book, you tear down the server completely – context, server with both volumes, include line, local release images:
cd ~/compose-lab/tag14
docker context rm notes-server
(cd server && docker compose down -v)
sed -i '' '\#compose-lab/tag14/server/ssh_config#d' ~/.ssh/config
diff ~/.ssh/config ~/.ssh/config.vor-tag14
docker image rm notes-api:1.0 notes-worker:1.0 notes-web:1.0 notes-api:1.1 \
notes-worker:1.1 notes-web:1.1 notes-api:1.2 notes-worker:1.2 notes-web:1.2 \
notes-server-serverExpected (shortened): diff without output, so
~/.ssh/config is as before.
notes-server
…
Untagged: notes-api:1.0
…
Untagged: notes-server-server:latest
Deleted: sha256:…
Save your progress? Sign up for free
Enterprise Note
- Instead of
save | load, a registry (Harbor, GitLab, GHCR) with signed images and deploy by digest.save | loadremains the way for air-gapped networks without a registry. - Whoever is in the group
dockeris root on the host. Restrict deploy users viaForceCommandorauthorized_keysoptions, keys from a secret store, distribute host keys in advance instead ofaccept-new. - Secret files on the server with mode 600 and the matching UID,
managed by configuration management (Ansible) or a secret manager
(“OpenBao secrets hands-on”), not by a hand-rolled
tar. - Compose on one host means: no rolling update, no failover. A short outage during recreate is built in; whoever needs more ends up with Swarm or Kubernetes.
- Rollback requires backward-compatible migrations. A release that restructures the table no longer works with the old image – back up before every deploy (Day 3).