Day 11: Reverse Proxy and TLS
Goal: A Caddy proxy is the only service that accepts HTTPS connections, with its own local CA, which you export and verify with
curl --cacert. · Duration: approx. 75 min · Prerequisite: Day 10
Briefing
- New service
proxywith Caddy: no Docker socket needed, a small configuration file, TLS without a public domain. tls internalmakes Caddy create its own CA (root and intermediate) and use it to issue certificates fornotes.localhost.- The
Caddyfilegoes into the container asconfigs:(Day 4); the CA and certificates live in the volumecaddy_data. - web loses its port; only proxy and (for tests) api are still reachable from the outside.
Mac network frontend network backend
:8443 ─HTTPS─┐
├─► proxy ──► web:3000 ──► api:8080 ──────► db:5432
:8000 ─HTTP──┘ (308) ▲
:8080 ─────────────────────────────────────┘ (only for drills and tests)
Drill 1: The Caddyfile
Start of the day: tear down the Day 10 stack, fetch the Day 10 state:
source ~/compose-lab/setup/env.sh
docker compose -p notes down
mkdir -p ~/compose-lab/tag11 && cd ~/compose-lab/tag11
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag10/notes/." .
docker pull -q caddy:2.11.7-alpineThe Caddyfile describes one block per address (“site”). Caddy expects
tabs for indentation; caddy fmt would complain about
spaces.
mkdir -p proxy
cat > proxy/Caddyfile <<'EOF'
{
skip_install_trust
}
notes.localhost {
tls internal
encode gzip
header {
Strict-Transport-Security "max-age=31536000"
X-Content-Type-Options "nosniff"
-Server
}
reverse_proxy web:3000
}
EOF| Line | Effect |
|---|---|
skip_install_trust |
no installation of the root into a trust store |
tls internal |
certificate from its own CA instead of via ACME (Let’s Encrypt) |
encode gzip |
compress responses if the client offers it |
header { … } |
set security headers; -Server removes the Server
header |
reverse_proxy web:3000 |
forward everything to web (DNS in the network
frontend) |
The unnamed block at the very top contains global options. Without
skip_install_trust, Caddy tries to install the root into
the system on the first certificate – pointless in a container.
Save your progress? Sign up for free
Drill 2: The proxy service
Two changes in compose.yaml: web loses its
ports:, and the new service proxy is added.
The complete file is in code/tag11/notes/compose.yaml.
web:
# … as on Day 10, but without ports: …
networks: [frontend]
proxy:
<<: *hardening
image: caddy:2.11.7-alpine
cap_add: [NET_BIND_SERVICE]
tmpfs: [/tmp]
ports:
- "127.0.0.1:${HTTPS_PORT:-8443}:443"
- "127.0.0.1:${HTTP_PORT:-8000}:80"
configs:
- source: caddyfile
target: /etc/caddy/Caddyfile
volumes:
- caddy_data:/data
- caddy_config:/config
networks: [frontend]
depends_on:
web:
condition: service_healthy
restart: unless-stoppedAnd down with the top-level entries:
volumes:
pgdata: {}
caddy_data: {}
caddy_config: {}
configs:
db_init:
file: ./db/init/10-readonly.sql
caddyfile:
file: ./proxy/CaddyfileBefore starting, Caddy checks the file itself – run
publishes no ports and, with --no-deps, starts nothing else
along with it:
docker compose run --rm --no-deps proxy caddy validate --config /etc/caddy/Caddyfile \
2>&1 | tail -1
docker compose up -d --wait
docker compose ps --format 'table {{.Service}}\t{{.Status}}\t{{.Ports}}'Expected (shortened, IDs and times differ on your machine):
Valid configuration
Container notes-web-1 Healthy
Container notes-proxy-1 Starting
Container notes-proxy-1 Started
…
Container notes-proxy-1 Healthy
SERVICE STATUS PORTS
api Up 3 seconds (healthy) 127.0.0.1:8080->8080/tcp
db Up 6 seconds (healthy) 5432/tcp
proxy Up Less than a second 443/udp, 2019/tcp, 127.0.0.1:8000->80/tcp, …
web Up 2 seconds (healthy) 3000/tcp
worker Up 2 seconds
proxy only starts once web is healthy. It has no healthcheck, so for
--wait “running” counts. 443/udp (HTTP/3) and
2019/tcp (admin API) are only EXPOSE entries
in the image; only the two ports on 127.0.0.1 are
published.
Save your progress? Sign up for free
Drill 3: Hardening and one capability
Via <<: *hardening, proxy inherits
cap_drop: [ALL], read_only and
no-new-privileges (Day 9). So why does it still contain
cap_add: [NET_BIND_SERVICE]? Take the line out
temporarily:
sed -i.bak 's/^ cap_add: \[NET_BIND_SERVICE\]/ # &/' compose.yaml
docker compose up -d proxy >/dev/null 2>&1
sleep 20
docker compose ps -a proxy --format 'table {{.Service}}\t{{.Status}}'
docker compose logs --no-log-prefix proxy | uniq -c
docker compose run --rm --no-deps --entrypoint getcap proxy /usr/bin/caddyExpected (shortened, numbers differ):
SERVICE STATUS
proxy Restarting (255) 6 seconds ago
8 exec /usr/bin/caddy: operation not permitted
/usr/bin/caddy cap_net_bind_service=ep
The program itself carries a file capability so that it may open ports 80 and 443 even without root. If this capability is missing from the container’s bounding set, the kernel refuses to even start the program – Caddy doesn’t even get as far as its first log line. Back to the original:
mv compose.yaml.bak compose.yaml
docker compose up -d --wait >/dev/null 2>&1
docker compose exec proxy grep CapEff /proc/1/statusExpected:
CapEff: 0000000000000400
0x400 is bit 10, CAP_NET_BIND_SERVICE – the
only capability Caddy has.
Warning: Without a healthcheck,
up --waitmay briefly report a container that is stuck in a restart loop as “Healthy”. For services you sign off via--wait, a real healthcheck or a look atps -ais worthwhile.
Save your progress? Sign up for free
Drill 4: Verifying TLS with your own CA
According to RFC 6761, *.localhost always points to your
own machine; curl resolves the name itself. --resolve still
makes the mapping explicit and works with any name.
curl -sS --resolve notes.localhost:8443:127.0.0.1 https://notes.localhost:8443/healthz
echo "Exit: $?"Expected:
curl: (60) SSL certificate problem: unable to get local issuer certificate
…
Exit: 60
That’s correct: curl doesn’t know Caddy’s CA. It lives in the volume
under /data/caddy/pki/authorities/local/.
docker compose cp fetches the public root – only that; the
private key root.key stays in the volume:
docker compose exec -T proxy ls /data/caddy/pki/authorities/local/
printf '%s\n' '# Lokale CA von Caddy (Tag 11)' 'proxy/root.crt' >> .gitignore
docker compose cp proxy:/data/caddy/pki/authorities/local/root.crt proxy/root.crt
openssl x509 -in proxy/root.crt -noout -subject -enddate
curl -sS --resolve notes.localhost:8443:127.0.0.1 --cacert proxy/root.crt \
https://notes.localhost:8443/healthz; echoExpected (shortened, times differ):
intermediate.crt
intermediate.key
root.crt
root.key
notes-proxy-1 Copied notes-proxy-1:/data/caddy/pki/authorities/local/root.crt to …
subject=CN=Caddy Local Authority - 2026 ECC Root
notAfter=Sep 6 22:26:30 2036 GMT
ok
Real TLS, real verification, no exception with -k.
proxy/root.crt goes into .gitignore beforehand
(printf in the block): the root belongs to exactly this
volume. Now the site’s certificate:
echo | openssl s_client -connect 127.0.0.1:8443 -servername notes.localhost \
2>/dev/null | openssl x509 -noout -issuer -ext subjectAltName -dates
curl -sS --cacert proxy/root.crt https://localhost:8443/healthz; echo "Exit: $?"Expected (shortened):
issuer=CN=Caddy Local Authority - ECC Intermediate
X509v3 Subject Alternative Name: critical
DNS:notes.localhost
notBefore=Oct 29 22:26:30 2026 GMT
notAfter=Oct 30 10:26:30 2026 GMT
curl: (35) LibreSSL/3.3.6: error:1404B438:SSL routines:…:tlsv1 alert internal error
Exit: 35
- The leaf certificate is issued by the intermediate,
not the root. It is valid for 12 hours (the
tls internaldefault); Caddy renews it by itself. - For a name without a site block (here
localhost), Caddy has no certificate and aborts the handshake.
Save your progress? Sign up for free
Drill 5: HTTP redirects, headers and gzip
curl -sS -i --resolve notes.localhost:8000:127.0.0.1 http://notes.localhost:8000/healthzExpected (shortened):
HTTP/1.1 308 Permanent Redirect
Connection: close
Location: https://notes.localhost/healthz
Server: Caddy
Caddy automatically redirects to HTTPS with a 308,
but to port 443: Caddy doesn’t know the published port 8443. In
production with 80/443 that fits; in the lab the redirect points into
the void – the challenge solves that. The redirect also still answers
with Server: Caddy, because header only
applies in the HTTPS site block.
For the next commands, a small shell function:
c() { curl -sS --resolve notes.localhost:8443:127.0.0.1 --cacert proxy/root.crt "$@"; }
c -I https://notes.localhost:8443/Expected (shortened):
HTTP/2 200
alt-svc: h3=":443"; ma=2592000
content-type: text/html; charset=utf-8
strict-transport-security: max-age=31536000
via: 1.1 Caddy
x-content-type-options: nosniff
x-powered-by: Express
content-length: 273
- HTTP/2 is on automatically with TLS. The
serverheader is missing; instead,vianames the proxy. x-powered-by: Expressstill reveals the technology behind it. With-X-Powered-Byin theheaderblock, that would be gone too.
Whether encode gzip kicks in shows with a larger
response. Create a few notes through the proxy – web adds the key
itself:
for i in 1 2 3 4 5 6 7 8; do
c -o /dev/null -w '%{http_code} ' -X POST https://notes.localhost:8443/api/notes \
-H 'Content-Type: application/json' -d "{\"text\":\"Notiz $i über HTTPS\"}"
done; echo
for enc in identity gzip; do
c -o /dev/null -D - -w "$enc: %{size_download} Bytes\n" -H "Accept-Encoding: $enc" \
https://notes.localhost:8443/api/notes | grep -i -E 'content-encoding|Bytes'
doneExpected (sizes differ):
201 201 201 201 201 201 201 201
identity: 766 Bytes
content-encoding: gzip
gzip: 341 Bytes
Small responses like index.html (273 bytes) stay
uncompressed; Caddy skips them below a minimum size.
Save your progress? Sign up for free
Drill 6: Path routing for web and api
A proxy can distribute by path. Change the Caddyfile
temporarily: everything under /v1/ goes to
the API, /v1/metrics stays blocked, the rest goes to
web.
cp proxy/Caddyfile proxy/Caddyfile.tag11
cat > proxy/Caddyfile <<'EOF'
{
skip_install_trust
}
notes.localhost {
tls internal
encode gzip
header {
Strict-Transport-Security "max-age=31536000"
X-Content-Type-Options "nosniff"
-Server
}
handle /v1/metrics {
respond "forbidden" 403
}
handle_path /v1/* {
reverse_proxy api:8080
}
handle {
reverse_proxy web:3000
}
}
EOF
docker compose exec proxy caddy reload --config /etc/caddy/Caddyfile 2>/dev/null
echo "Exit: $?"Expected:
Exit: 0
handle blocks are mutually exclusive; the more specific
path wins. handle_path strips the prefix:
/v1/version becomes /version at the API.
caddy reload reloads the file via the admin API inside the
container – without a restart and without dropped connections.
for p in /v1/version /v1/notes /v1/metrics /api/notes; do
printf '%-12s %s\n' "$p" "$(c -o /dev/null -w '%{http_code}' \
"https://notes.localhost:8443$p")"
done
c -X POST https://notes.localhost:8443/v1/notes -H 'Content-Type: application/json' \
-d '{"text":"direkt ans API"}' -w '%{http_code}\n'Expected:
/v1/version 200
/v1/notes 200
/v1/metrics 403
/api/notes 200
{"error":"invalid api key"}
401
Via /v1/, the API’s own rule applies: POST only with
X-API-Key. The metrics stay internal – Prometheus fetches
them tomorrow directly within the network (Day 12). Now back to today’s
state, and pay attention to what up says about it:
cp proxy/Caddyfile.tag11 proxy/Caddyfile && rm proxy/Caddyfile.tag11
docker compose up -d --wait 2>&1 | grep proxy
c -o /dev/null -w '%{http_code}\n' https://notes.localhost:8443/v1/metrics
docker compose exec proxy caddy reload --config /etc/caddy/Caddyfile 2>/dev/null
c -o /dev/null -w '%{http_code}\n' https://notes.localhost:8443/v1/metricsExpected:
Container notes-proxy-1 Running
Container notes-proxy-1 Waiting
Container notes-proxy-1 Healthy
403
404
Compose sees no change to the service and leaves the container
running. The file in the container is already new (configs:
with file: is a bind mount), but Caddy keeps working with
the loaded version until you call reload. After that,
/v1/metrics lands at web again, which doesn’t know the
path.
Warning: A bind mount of a single file is tied to the inode. Editors that create a new file and rename it when saving do not change the file in the container. Then only
docker compose up -d --force-recreate proxyhelps.
Save your progress? Sign up for free
Drill 7: The CA lives in the volume
docker compose up -d --wait --force-recreate proxy >/dev/null 2>&1
c https://notes.localhost:8443/healthz; echo
docker compose rm -sf proxy >/dev/null 2>&1
docker volume rm notes_caddy_data
docker compose up -d --wait proxy >/dev/null 2>&1
c https://notes.localhost:8443/healthz; echo "Exit: $?"Expected:
ok
notes_caddy_data
curl: (60) SSL certificate problem: unable to get local issuer certificate
…
Exit: 60
A new container with the same volume keeps the CA; a new volume
brings a new CA – and every distributed root.crt becomes
worthless. The same happens with docker compose down -v.
Fetch the root again:
docker compose cp proxy:/data/caddy/pki/authorities/local/root.crt proxy/root.crt \
>/dev/null 2>&1
c https://notes.localhost:8443/healthz; echoExpected:
ok
Save your progress? Sign up for free
Checkpoint
The stack is running (docker compose up -d --wait).
You’ll find the complete state in code/tag11/notes/. proxy
is running, web publishes no port:
cd ~/compose-lab/tag11
docker compose ps --format '{{.Service}} {{.Ports}}'Expected (shortened):
api 127.0.0.1:8080->8080/tcp
db 5432/tcp
proxy 443/udp, 2019/tcp, 127.0.0.1:8000->80/tcp, 127.0.0.1:8443->443/tcp
web 3000/tcp
worker
HTTPS with your own CA:
curl -s --resolve notes.localhost:8443:127.0.0.1 --cacert proxy/root.crt \
https://notes.localhost:8443/healthz; echoExpected:
ok
HTTP redirects:
curl -s -o /dev/null -w '%{http_code}\n' --resolve notes.localhost:8000:127.0.0.1 \
http://notes.localhost:8000/Expected:
308
HSTS set, no Server header (only one line):
curl -s -o /dev/null -D - --resolve notes.localhost:8443:127.0.0.1 \
--cacert proxy/root.crt https://notes.localhost:8443/ | grep -iE '^(strict|server)'Expected:
strict-transport-security: max-age=31536000
The root CA is in .gitignore:
grep -x 'proxy/root.crt' .gitignoreExpected:
proxy/root.crt
Challenge
Make sure that
curl -L http://notes.localhost:8000/healthz gets all the
way through to the response ok, without changing the ports
on the Mac. compose.yaml and proxy/Caddyfile
stay unchanged; write an additional Compose file.
Hint: Caddy appends a port to the redirect if the
site address itself names a port, and the Caddyfile supports environment
variables as {$NAME:default}.
Solution
Caddy listens inside the container on the same port that the Mac
publishes. A separate Caddyfile and an override file with
!override (Day 7):
cat > proxy/Caddyfile.ports <<'EOF'
{
skip_install_trust
}
notes.localhost:{$HTTPS_PORT:8443} {
tls internal
encode gzip
header {
Strict-Transport-Security "max-age=31536000"
X-Content-Type-Options "nosniff"
-Server
}
reverse_proxy web:3000
}
EOF
cat > compose.ports.yaml <<'EOF'
# compose.ports.yaml – challenge Day 11: redirect to the published HTTPS port
services:
proxy:
environment:
HTTPS_PORT: ${HTTPS_PORT:-8443}
ports: !override
- "127.0.0.1:${HTTPS_PORT:-8443}:${HTTPS_PORT:-8443}"
- "127.0.0.1:${HTTP_PORT:-8000}:80"
configs: !override
- source: caddyfile_ports
target: /etc/caddy/Caddyfile
configs:
caddyfile_ports:
file: ./proxy/Caddyfile.ports
EOF
docker compose -f compose.yaml -f compose.ports.yaml up -d --wait proxy >/dev/null 2>&1
docker compose ps proxy --format '{{.Ports}}'
curl -sS -i --resolve notes.localhost:8000:127.0.0.1 \
http://notes.localhost:8000/healthz | grep -i location
curl -sS -L --resolve notes.localhost:8000:127.0.0.1 \
--resolve notes.localhost:8443:127.0.0.1 --cacert proxy/root.crt \
http://notes.localhost:8000/healthz; echoExpected:
443/tcp, 2019/tcp, 443/udp, 127.0.0.1:8443->8443/tcp, 127.0.0.1:8000->80/tcp
Location: https://notes.localhost:8443/healthz
ok
By the way, a port alone in https_port (global option)
doesn’t help: Caddy omits the port in the redirect if it matches the
configured HTTPS port, because it assumes that a forward from 443 sits
in front of it (verified live). cap_add is still needed
despite the high port – the file capability from Drill 3 is attached to
the program, not to the port. The files are under
code/tag11/challenge/. Back to the state of the day:
docker compose up -d --wait >/dev/null 2>&1
docker compose ps proxy --format '{{.Ports}}'
docker compose downExpected:
443/udp, 2019/tcp, 127.0.0.1:8000->80/tcp, 127.0.0.1:8443->443/tcp
Save your progress? Sign up for free
Enterprise Note
- In production, certificates come via ACME (Let’s Encrypt, or an
internal ACME CA such as step-ca, Vault/OpenBao PKI) – Caddy can do
both;
tls internalis for the lab and internal test environments. - The platform team distributes the root of an internal CA via
configuration management into the trust stores (operating system, Java,
container images), not via
curl --cacert. caddy_datacontains private keys: back it up encrypted, treat access like secrets. If the volume is lost, every distributed root is worthless (Drill 7).- Traefik is the common alternative with Docker labels as configuration. Its Docker provider needs the Docker socket – only through a read-only socket proxy, never directly (Day 9).
- The proxy is the place for rate limits, access protection for
/metricsand admin paths, and access logs; roll out HSTS with a longmax-ageandincludeSubDomainsonly once all subdomains support HTTPS.