Skip to content
Hands-on IT IT Drills

Day 11: Reverse Proxy and TLS

Goal: A Caddy proxy is the only service that accepts HTTPS connections, with its own local CA, which you export and verify with curl --cacert. · Duration: approx. 75 min · Prerequisite: Day 10

Briefing

  • New service proxy with Caddy: no Docker socket needed, a small configuration file, TLS without a public domain.
  • tls internal makes Caddy create its own CA (root and intermediate) and use it to issue certificates for notes.localhost.
  • The Caddyfile goes into the container as configs: (Day 4); the CA and certificates live in the volume caddy_data.
  • web loses its port; only proxy and (for tests) api are still reachable from the outside.
 Mac                       network frontend               network backend
 :8443 ─HTTPS─┐
              ├─► proxy ──► web:3000 ──► api:8080 ──────► db:5432
 :8000 ─HTTP──┘   (308)                     ▲
 :8080 ─────────────────────────────────────┘ (only for drills and tests)

Drill 1: The Caddyfile

Start of the day: tear down the Day 10 stack, fetch the Day 10 state:

source ~/compose-lab/setup/env.sh
docker compose -p notes down
mkdir -p ~/compose-lab/tag11 && cd ~/compose-lab/tag11
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag10/notes/." .
docker pull -q caddy:2.11.7-alpine

The Caddyfile describes one block per address (“site”). Caddy expects tabs for indentation; caddy fmt would complain about spaces.

mkdir -p proxy
cat > proxy/Caddyfile <<'EOF'
{
    skip_install_trust
}

notes.localhost {
    tls internal
    encode gzip
    header {
        Strict-Transport-Security "max-age=31536000"
        X-Content-Type-Options "nosniff"
        -Server
    }
    reverse_proxy web:3000
}
EOF
Line Effect
skip_install_trust no installation of the root into a trust store
tls internal certificate from its own CA instead of via ACME (Let’s Encrypt)
encode gzip compress responses if the client offers it
header { … } set security headers; -Server removes the Server header
reverse_proxy web:3000 forward everything to web (DNS in the network frontend)

The unnamed block at the very top contains global options. Without skip_install_trust, Caddy tries to install the root into the system on the first certificate – pointless in a container.

Save your progress? Sign up for free

Drill 2: The proxy service

Two changes in compose.yaml: web loses its ports:, and the new service proxy is added. The complete file is in code/tag11/notes/compose.yaml.

  web:
    # … as on Day 10, but without ports: …
    networks: [frontend]

  proxy:
    <<: *hardening
    image: caddy:2.11.7-alpine
    cap_add: [NET_BIND_SERVICE]
    tmpfs: [/tmp]
    ports:
      - "127.0.0.1:${HTTPS_PORT:-8443}:443"
      - "127.0.0.1:${HTTP_PORT:-8000}:80"
    configs:
      - source: caddyfile
        target: /etc/caddy/Caddyfile
    volumes:
      - caddy_data:/data
      - caddy_config:/config
    networks: [frontend]
    depends_on:
      web:
        condition: service_healthy
    restart: unless-stopped

And down with the top-level entries:

volumes:
  pgdata: {}
  caddy_data: {}
  caddy_config: {}

configs:
  db_init:
    file: ./db/init/10-readonly.sql
  caddyfile:
    file: ./proxy/Caddyfile

Before starting, Caddy checks the file itself – run publishes no ports and, with --no-deps, starts nothing else along with it:

docker compose run --rm --no-deps proxy caddy validate --config /etc/caddy/Caddyfile \
  2>&1 | tail -1
docker compose up -d --wait
docker compose ps --format 'table {{.Service}}\t{{.Status}}\t{{.Ports}}'

Expected (shortened, IDs and times differ on your machine):

Valid configuration
 Container notes-web-1 Healthy
 Container notes-proxy-1 Starting
 Container notes-proxy-1 Started
 …
 Container notes-proxy-1 Healthy
SERVICE   STATUS                   PORTS
api       Up 3 seconds (healthy)   127.0.0.1:8080->8080/tcp
db        Up 6 seconds (healthy)   5432/tcp
proxy     Up Less than a second    443/udp, 2019/tcp, 127.0.0.1:8000->80/tcp, …
web       Up 2 seconds (healthy)   3000/tcp
worker    Up 2 seconds

proxy only starts once web is healthy. It has no healthcheck, so for --wait “running” counts. 443/udp (HTTP/3) and 2019/tcp (admin API) are only EXPOSE entries in the image; only the two ports on 127.0.0.1 are published.

Save your progress? Sign up for free

Drill 3: Hardening and one capability

Via <<: *hardening, proxy inherits cap_drop: [ALL], read_only and no-new-privileges (Day 9). So why does it still contain cap_add: [NET_BIND_SERVICE]? Take the line out temporarily:

sed -i.bak 's/^    cap_add: \[NET_BIND_SERVICE\]/    # &/' compose.yaml
docker compose up -d proxy >/dev/null 2>&1
sleep 20
docker compose ps -a proxy --format 'table {{.Service}}\t{{.Status}}'
docker compose logs --no-log-prefix proxy | uniq -c
docker compose run --rm --no-deps --entrypoint getcap proxy /usr/bin/caddy

Expected (shortened, numbers differ):

SERVICE   STATUS
proxy     Restarting (255) 6 seconds ago
      8 exec /usr/bin/caddy: operation not permitted
/usr/bin/caddy cap_net_bind_service=ep

The program itself carries a file capability so that it may open ports 80 and 443 even without root. If this capability is missing from the container’s bounding set, the kernel refuses to even start the program – Caddy doesn’t even get as far as its first log line. Back to the original:

mv compose.yaml.bak compose.yaml
docker compose up -d --wait >/dev/null 2>&1
docker compose exec proxy grep CapEff /proc/1/status

Expected:

CapEff: 0000000000000400

0x400 is bit 10, CAP_NET_BIND_SERVICE – the only capability Caddy has.

Warning: Without a healthcheck, up --wait may briefly report a container that is stuck in a restart loop as “Healthy”. For services you sign off via --wait, a real healthcheck or a look at ps -a is worthwhile.

Save your progress? Sign up for free

Drill 4: Verifying TLS with your own CA

According to RFC 6761, *.localhost always points to your own machine; curl resolves the name itself. --resolve still makes the mapping explicit and works with any name.

curl -sS --resolve notes.localhost:8443:127.0.0.1 https://notes.localhost:8443/healthz
echo "Exit: $?"

Expected:

curl: (60) SSL certificate problem: unable to get local issuer certificate
…
Exit: 60

That’s correct: curl doesn’t know Caddy’s CA. It lives in the volume under /data/caddy/pki/authorities/local/. docker compose cp fetches the public root – only that; the private key root.key stays in the volume:

docker compose exec -T proxy ls /data/caddy/pki/authorities/local/
printf '%s\n' '# Lokale CA von Caddy (Tag 11)' 'proxy/root.crt' >> .gitignore
docker compose cp proxy:/data/caddy/pki/authorities/local/root.crt proxy/root.crt
openssl x509 -in proxy/root.crt -noout -subject -enddate
curl -sS --resolve notes.localhost:8443:127.0.0.1 --cacert proxy/root.crt \
  https://notes.localhost:8443/healthz; echo

Expected (shortened, times differ):

intermediate.crt
intermediate.key
root.crt
root.key
 notes-proxy-1 Copied notes-proxy-1:/data/caddy/pki/authorities/local/root.crt to …
subject=CN=Caddy Local Authority - 2026 ECC Root
notAfter=Sep  6 22:26:30 2036 GMT
ok

Real TLS, real verification, no exception with -k. proxy/root.crt goes into .gitignore beforehand (printf in the block): the root belongs to exactly this volume. Now the site’s certificate:

echo | openssl s_client -connect 127.0.0.1:8443 -servername notes.localhost \
  2>/dev/null | openssl x509 -noout -issuer -ext subjectAltName -dates
curl -sS --cacert proxy/root.crt https://localhost:8443/healthz; echo "Exit: $?"

Expected (shortened):

issuer=CN=Caddy Local Authority - ECC Intermediate
X509v3 Subject Alternative Name: critical
    DNS:notes.localhost
notBefore=Oct 29 22:26:30 2026 GMT
notAfter=Oct 30 10:26:30 2026 GMT
curl: (35) LibreSSL/3.3.6: error:1404B438:SSL routines:…:tlsv1 alert internal error
Exit: 35
  • The leaf certificate is issued by the intermediate, not the root. It is valid for 12 hours (the tls internal default); Caddy renews it by itself.
  • For a name without a site block (here localhost), Caddy has no certificate and aborts the handshake.

Save your progress? Sign up for free

Drill 5: HTTP redirects, headers and gzip

curl -sS -i --resolve notes.localhost:8000:127.0.0.1 http://notes.localhost:8000/healthz

Expected (shortened):

HTTP/1.1 308 Permanent Redirect
Connection: close
Location: https://notes.localhost/healthz
Server: Caddy

Caddy automatically redirects to HTTPS with a 308, but to port 443: Caddy doesn’t know the published port 8443. In production with 80/443 that fits; in the lab the redirect points into the void – the challenge solves that. The redirect also still answers with Server: Caddy, because header only applies in the HTTPS site block.

For the next commands, a small shell function:

c() { curl -sS --resolve notes.localhost:8443:127.0.0.1 --cacert proxy/root.crt "$@"; }
c -I https://notes.localhost:8443/

Expected (shortened):

HTTP/2 200
alt-svc: h3=":443"; ma=2592000
content-type: text/html; charset=utf-8
strict-transport-security: max-age=31536000
via: 1.1 Caddy
x-content-type-options: nosniff
x-powered-by: Express
content-length: 273
  • HTTP/2 is on automatically with TLS. The server header is missing; instead, via names the proxy.
  • x-powered-by: Express still reveals the technology behind it. With -X-Powered-By in the header block, that would be gone too.

Whether encode gzip kicks in shows with a larger response. Create a few notes through the proxy – web adds the key itself:

for i in 1 2 3 4 5 6 7 8; do
  c -o /dev/null -w '%{http_code} ' -X POST https://notes.localhost:8443/api/notes \
    -H 'Content-Type: application/json' -d "{\"text\":\"Notiz $i über HTTPS\"}"
done; echo
for enc in identity gzip; do
  c -o /dev/null -D - -w "$enc: %{size_download} Bytes\n" -H "Accept-Encoding: $enc" \
    https://notes.localhost:8443/api/notes | grep -i -E 'content-encoding|Bytes'
done

Expected (sizes differ):

201 201 201 201 201 201 201 201
identity: 766 Bytes
content-encoding: gzip
gzip: 341 Bytes

Small responses like index.html (273 bytes) stay uncompressed; Caddy skips them below a minimum size.

Save your progress? Sign up for free

Drill 6: Path routing for web and api

A proxy can distribute by path. Change the Caddyfile temporarily: everything under /v1/ goes to the API, /v1/metrics stays blocked, the rest goes to web.

cp proxy/Caddyfile proxy/Caddyfile.tag11
cat > proxy/Caddyfile <<'EOF'
{
    skip_install_trust
}

notes.localhost {
    tls internal
    encode gzip
    header {
        Strict-Transport-Security "max-age=31536000"
        X-Content-Type-Options "nosniff"
        -Server
    }
    handle /v1/metrics {
        respond "forbidden" 403
    }
    handle_path /v1/* {
        reverse_proxy api:8080
    }
    handle {
        reverse_proxy web:3000
    }
}
EOF
docker compose exec proxy caddy reload --config /etc/caddy/Caddyfile 2>/dev/null
echo "Exit: $?"

Expected:

Exit: 0

handle blocks are mutually exclusive; the more specific path wins. handle_path strips the prefix: /v1/version becomes /version at the API. caddy reload reloads the file via the admin API inside the container – without a restart and without dropped connections.

for p in /v1/version /v1/notes /v1/metrics /api/notes; do
  printf '%-12s %s\n' "$p" "$(c -o /dev/null -w '%{http_code}' \
    "https://notes.localhost:8443$p")"
done
c -X POST https://notes.localhost:8443/v1/notes -H 'Content-Type: application/json' \
  -d '{"text":"direkt ans API"}' -w '%{http_code}\n'

Expected:

/v1/version  200
/v1/notes    200
/v1/metrics  403
/api/notes   200
{"error":"invalid api key"}
401

Via /v1/, the API’s own rule applies: POST only with X-API-Key. The metrics stay internal – Prometheus fetches them tomorrow directly within the network (Day 12). Now back to today’s state, and pay attention to what up says about it:

cp proxy/Caddyfile.tag11 proxy/Caddyfile && rm proxy/Caddyfile.tag11
docker compose up -d --wait 2>&1 | grep proxy
c -o /dev/null -w '%{http_code}\n' https://notes.localhost:8443/v1/metrics
docker compose exec proxy caddy reload --config /etc/caddy/Caddyfile 2>/dev/null
c -o /dev/null -w '%{http_code}\n' https://notes.localhost:8443/v1/metrics

Expected:

 Container notes-proxy-1 Running
 Container notes-proxy-1 Waiting
 Container notes-proxy-1 Healthy
403
404

Compose sees no change to the service and leaves the container running. The file in the container is already new (configs: with file: is a bind mount), but Caddy keeps working with the loaded version until you call reload. After that, /v1/metrics lands at web again, which doesn’t know the path.

Warning: A bind mount of a single file is tied to the inode. Editors that create a new file and rename it when saving do not change the file in the container. Then only docker compose up -d --force-recreate proxy helps.

Save your progress? Sign up for free

Drill 7: The CA lives in the volume

docker compose up -d --wait --force-recreate proxy >/dev/null 2>&1
c https://notes.localhost:8443/healthz; echo
docker compose rm -sf proxy >/dev/null 2>&1
docker volume rm notes_caddy_data
docker compose up -d --wait proxy >/dev/null 2>&1
c https://notes.localhost:8443/healthz; echo "Exit: $?"

Expected:

ok
notes_caddy_data
curl: (60) SSL certificate problem: unable to get local issuer certificate
…
Exit: 60

A new container with the same volume keeps the CA; a new volume brings a new CA – and every distributed root.crt becomes worthless. The same happens with docker compose down -v. Fetch the root again:

docker compose cp proxy:/data/caddy/pki/authorities/local/root.crt proxy/root.crt \
  >/dev/null 2>&1
c https://notes.localhost:8443/healthz; echo

Expected:

ok

Save your progress? Sign up for free

Checkpoint

The stack is running (docker compose up -d --wait). You’ll find the complete state in code/tag11/notes/. proxy is running, web publishes no port:

cd ~/compose-lab/tag11
docker compose ps --format '{{.Service}} {{.Ports}}'

Expected (shortened):

api 127.0.0.1:8080->8080/tcp
db 5432/tcp
proxy 443/udp, 2019/tcp, 127.0.0.1:8000->80/tcp, 127.0.0.1:8443->443/tcp
web 3000/tcp
worker

HTTPS with your own CA:

curl -s --resolve notes.localhost:8443:127.0.0.1 --cacert proxy/root.crt \
  https://notes.localhost:8443/healthz; echo

Expected:

ok

HTTP redirects:

curl -s -o /dev/null -w '%{http_code}\n' --resolve notes.localhost:8000:127.0.0.1 \
  http://notes.localhost:8000/

Expected:

308

HSTS set, no Server header (only one line):

curl -s -o /dev/null -D - --resolve notes.localhost:8443:127.0.0.1 \
  --cacert proxy/root.crt https://notes.localhost:8443/ | grep -iE '^(strict|server)'

Expected:

strict-transport-security: max-age=31536000

The root CA is in .gitignore:

grep -x 'proxy/root.crt' .gitignore

Expected:

proxy/root.crt

Challenge

Make sure that curl -L http://notes.localhost:8000/healthz gets all the way through to the response ok, without changing the ports on the Mac. compose.yaml and proxy/Caddyfile stay unchanged; write an additional Compose file.

Hint: Caddy appends a port to the redirect if the site address itself names a port, and the Caddyfile supports environment variables as {$NAME:default}.

Solution

Caddy listens inside the container on the same port that the Mac publishes. A separate Caddyfile and an override file with !override (Day 7):

cat > proxy/Caddyfile.ports <<'EOF'
{
    skip_install_trust
}

notes.localhost:{$HTTPS_PORT:8443} {
    tls internal
    encode gzip
    header {
        Strict-Transport-Security "max-age=31536000"
        X-Content-Type-Options "nosniff"
        -Server
    }
    reverse_proxy web:3000
}
EOF
cat > compose.ports.yaml <<'EOF'
# compose.ports.yaml – challenge Day 11: redirect to the published HTTPS port
services:
  proxy:
    environment:
      HTTPS_PORT: ${HTTPS_PORT:-8443}
    ports: !override
      - "127.0.0.1:${HTTPS_PORT:-8443}:${HTTPS_PORT:-8443}"
      - "127.0.0.1:${HTTP_PORT:-8000}:80"
    configs: !override
      - source: caddyfile_ports
        target: /etc/caddy/Caddyfile

configs:
  caddyfile_ports:
    file: ./proxy/Caddyfile.ports
EOF
docker compose -f compose.yaml -f compose.ports.yaml up -d --wait proxy >/dev/null 2>&1
docker compose ps proxy --format '{{.Ports}}'
curl -sS -i --resolve notes.localhost:8000:127.0.0.1 \
  http://notes.localhost:8000/healthz | grep -i location
curl -sS -L --resolve notes.localhost:8000:127.0.0.1 \
  --resolve notes.localhost:8443:127.0.0.1 --cacert proxy/root.crt \
  http://notes.localhost:8000/healthz; echo

Expected:

443/tcp, 2019/tcp, 443/udp, 127.0.0.1:8443->8443/tcp, 127.0.0.1:8000->80/tcp
Location: https://notes.localhost:8443/healthz
ok

By the way, a port alone in https_port (global option) doesn’t help: Caddy omits the port in the redirect if it matches the configured HTTPS port, because it assumes that a forward from 443 sits in front of it (verified live). cap_add is still needed despite the high port – the file capability from Drill 3 is attached to the program, not to the port. The files are under code/tag11/challenge/. Back to the state of the day:

docker compose up -d --wait >/dev/null 2>&1
docker compose ps proxy --format '{{.Ports}}'
docker compose down

Expected:

443/udp, 2019/tcp, 127.0.0.1:8000->80/tcp, 127.0.0.1:8443->443/tcp

Save your progress? Sign up for free

Enterprise Note

  • In production, certificates come via ACME (Let’s Encrypt, or an internal ACME CA such as step-ca, Vault/OpenBao PKI) – Caddy can do both; tls internal is for the lab and internal test environments.
  • The platform team distributes the root of an internal CA via configuration management into the trust stores (operating system, Java, container images), not via curl --cacert.
  • caddy_data contains private keys: back it up encrypted, treat access like secrets. If the volume is lost, every distributed root is worthless (Drill 7).
  • Traefik is the common alternative with Docker labels as configuration. Its Docker provider needs the Docker socket – only through a read-only socket proxy, never directly (Day 9).
  • The proxy is the place for rate limits, access protection for /metrics and admin paths, and access logs; roll out HSTS with a long max-age and includeSubDomains only once all subdomains support HTTPS.