Skip to content
Hands-on IT IT Drills

Day 9: Resources and Hardening

Goal: Every service gets limits for CPU, memory and processes, a read-only file system, no capabilities and rotating logs – defined together in an extension field. · Duration: approx. 75 min · Prerequisite: Day 8

Briefing

  • Without settings, a container may use all the memory and all CPUs of the Docker host, start any number of processes and write to its file system.
  • deploy.resources.limits sets limits for cpus, memory and pids. Whoever wants too much memory gets killed by the kernel (OOM kill, exit code 137).
  • read_only, cap_drop: [ALL], no-new-privileges and a non-root user take away almost everything an attacker inside the container would need.
  • The local driver rotates logs with max-size/max-file.
  • An extension field x-hardening with a YAML anchor keeps the hardening in one place.
 x-hardening: &hardening ──<<:──► api, migrate, worker, web, db
   read_only · cap_drop [ALL] · no-new-privileges · logging local 10m×3
 per service: deploy.resources.limits { cpus, memory, pids } · tmpfs · user · init

Drill 1: What a container may do today

source ~/compose-lab/setup/env.sh
docker compose -p notes down
mkdir -p ~/compose-lab/tag09 && cd ~/compose-lab/tag09
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag08/notes/." .
docker compose up -d --build --wait >/dev/null 2>&1; echo "Exit: $?"
docker stats --no-stream --format \
  'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.PIDs}}' \
  notes-api-1 notes-worker-1 notes-web-1 notes-db-1

Expected (shortened, values differ on your machine):

Exit: 0
NAME             CPU %     MEM USAGE / LIMIT     PIDS
notes-api-1      0.00%     2.625MiB / 7.737GiB   8
notes-worker-1   0.08%     23.58MiB / 7.737GiB   1
notes-web-1      0.00%     19.98MiB / 7.737GiB   11
notes-db-1       0.04%     29.52MiB / 7.737GiB   10

The “limit” is the entire memory of the Colima VM. A memory leak in one service takes the space away from all the others. Now the permissions, seen from the inside:

docker compose exec web sh -c 'id; grep -E "^(CapEff|CapBnd|NoNewPrivs)" /proc/1/status
  touch /tmp/x && echo "/tmp beschreibbar"'
docker compose exec -u 0 web sh -c 'grep CapEff /proc/self/status
  chown node /app && echo "chown als root ok"'

Expected:

uid=1000(node) gid=1000(node) groups=1000(node)
CapEff: 0000000000000000
CapBnd: 00000000a80425fb
NoNewPrivs: 0
/tmp beschreibbar
CapEff: 00000000a80425fb
chown als root ok

CapBnd is the upper bound of the capabilities (Docker default: 14 of them, such as CHOWN, NET_RAW, SETUID). User node has no effective ones, but whoever becomes root inside the container – via exec -u 0 or through a vulnerability – gets all of them.

notes-api runs in scratch without a shell. A helper container that shares its PID namespace can still read its process data:

docker run --rm --label lab=compose-lab --pid container:notes-api-1 alpine:3.24 \
  sh -c 'grep -E "^(Uid|CapBnd|NoNewPrivs)" /proc/1/status
         awk "\$5==\"/\" {print \$5, \$6}" /proc/1/mountinfo'

Expected:

Uid:    65532   65532   65532   65532
CapBnd: 00000000a80425fb
NoNewPrivs: 0
/ rw,relatime

Non-root, but capabilities in the bounding set and a writable root file system (rw). You’ll need this helper container often today.

Save your progress? Sign up for free

Drill 2: Limits for CPU, memory and processes

Add the limits to api, and, as the training plan says, pids_limit too:

  api:
    # … as on Day 8 …
    deploy:
      resources:
        limits:
          cpus: "0.50"
          memory: 64M
    pids_limit: 100

The same for worker ("0.25"/128M), web ("0.50"/128M), db ("1.0"/512M, pids_limit: 200) and migrate (like api, without pids_limit).

docker compose up -d --wait; echo "Exit: $?"

Expected (shortened, the service named may differ):

services.worker: can't set distinct values on 'pids_limit' and
'deploy.resources.limits.pids': invalid compose project
Exit: 1

Compose 5.6 treats pids_limit and deploy.resources.limits.pids as the same field. If limits is present, pids counts as 0 there – and that contradicts the 100 (verified live; without deploy:, pids_limit alone works). So the process limit belongs in the limits block:

    deploy:
      resources:
        limits:
          cpus: "0.50"
          memory: 64M
          pids: 100

Replace pids_limit accordingly in all four services (db: pids: 200).

docker compose up -d --wait >/dev/null 2>&1; echo "Exit: $?"
docker stats --no-stream --format 'table {{.Name}}\t{{.MemUsage}}\t{{.PIDs}}' \
  notes-api-1 notes-worker-1 notes-web-1 notes-db-1
docker compose exec web cat /sys/fs/cgroup/cpu.max /sys/fs/cgroup/memory.max \
  /sys/fs/cgroup/pids.max

Expected (shortened):

Exit: 0
NAME             MEM USAGE / LIMIT   PIDS
notes-api-1      2.188MiB / 64MiB    7
notes-worker-1   23.59MiB / 128MiB   2
notes-web-1      24.17MiB / 128MiB   11
notes-db-1       32.29MiB / 512MiB   10
50000 100000
134217728
100

The limits end up as cgroup values in the container: cpu.max means “50 ms of CPU time per 100 ms”, i.e. half a CPU. Try it with an endless loop in Node:

docker compose exec -d web \
  node -e 'const t = Date.now(); while (Date.now() - t < 15000) {}'
sleep 6
docker stats --no-stream --format 'table {{.Name}}\t{{.CPUPerc}}' notes-web-1
docker compose exec web grep -E 'nr_throttled|throttled_usec' /sys/fs/cgroup/cpu.stat

Expected (shortened, numbers differ):

NAME          CPU %
notes-web-1   51.34%
nr_throttled 75
throttled_usec 3694773

There’s no more than 50 %; the rest of the time is throttled. Besides limits, deploy.resources also supports reservations. In the test, a memory reservation of 256M ended up as memory.low (a soft guarantee under memory pressure); a CPU reservation changed nothing visible in the cgroup. The final state sticks with limits.

Tip: The reference also knows mem_limit and cpus directly on the service. According to the docs, they must match deploy.resources.limits if both are set – in this book everything lives uniformly under deploy.

Save your progress? Sign up for free

Drill 3: When the limit kicks in

What happens with 200 MB in a container with a 128 MB limit? docker compose events shows what Docker reports:

docker compose events --json worker > events.json &
docker compose exec worker python -c 'b = bytearray(200 * 1024 * 1024); print("ok")'
echo "Exit: $?"
sleep 2; kill %1
jq -c '{action, service}' events.json
docker compose ps worker --format 'table {{.Service}}\t{{.Status}}'
docker compose exec worker cat /sys/fs/cgroup/memory.events
rm events.json

Expected (shortened):

Exit: 137
{"action":"exec_create: python -c b = bytearray(200 * 1024 * 1024); …",…}
{"action":"exec_start: python -c b = bytearray(200 * 1024 * 1024); …",…}
{"action":"oom","service":"worker"}
{"action":"exec_die","service":"worker"}
SERVICE   STATUS
worker    Up About a minute
low 0
high 0
max 35
oom 1
oom_kill 1
oom_group_kill 0

Exit 137 = 128 + 9 (SIGKILL). The kernel killed the largest process in the cgroup – here the exec process; the worker itself kept running. If it hits the main process, it looks different. An additional file temporarily sets worker’s limit to 16 MB:

cat > compose.oom.yaml <<'EOF'
# compose.oom.yaml – only for Day 9, Drill 3
services:
  worker:
    deploy:
      resources:
        limits:
          memory: 16M
EOF
docker compose -f compose.yaml -f compose.oom.yaml up -d --no-deps worker
sleep 10
docker compose ps -a worker --format 'table {{.Service}}\t{{.Status}}'
docker compose logs worker | wc -l
docker compose events --json --since 15s --until 1s worker 2>/dev/null \
  | jq -r 'select(.action == "oom" or .action == "die")
           | "\(.time[11:19]) \(.action) \(.attributes.exitCode // "")"' | tail -4

Expected (shortened, times differ):

 Container notes-worker-1 Recreate
 Container notes-worker-1 Recreated
 Container notes-worker-1 Started
SERVICE   STATUS
worker    Restarting (137) 3 seconds ago
       0
23:33:17 oom
23:33:17 die 137
23:33:20 oom
23:33:20 die 137

That’s the typical picture of a limit that’s too tight: Restarting (137), not a single log line – Python dies while loading its modules – and only the events name the reason. --until ends the event stream; Compose 5.6 then reports an EOF on stderr, hence 2>/dev/null. Back to the correct limit:

rm compose.oom.yaml
docker compose up -d --wait worker 2>&1 | grep -E 'Recreated|Healthy'

Expected:

 Container notes-worker-1 Recreated
 Container notes-worker-1 Healthy

The process limit is just as hard, for web with pids: 100:

docker compose exec web sh -c 'for i in $(seq 1 120); do sleep 5 & done; wait'
docker compose exec web cat /sys/fs/cgroup/pids.events

Expected:

sh: can't fork: Resource temporarily unavailable
max 1

That way, a fork bomb or a thread leak only cripples its own container, not the host.

Save your progress? Sign up for free

Drill 4: Read-only with tmpfs

From now on you collect the hardening in an extension field. Add below x-api-build:

x-hardening: &hardening
  read_only: true

and in api, migrate, worker and web put <<: *hardening as the first line:

  api:
    <<: *hardening
    build: *api-build
    # …

<< is YAML’s merge key: it copies all keys of the anchor into the service. Whatever the service sets itself wins.

docker compose up -d --wait 2>&1 | grep -E 'Recreated' | sort -u
docker compose exec web sh -c 'touch /tmp/x; touch /app/x'
docker compose exec worker touch /tmp/x
docker run --rm --label lab=compose-lab --pid container:notes-api-1 alpine:3.24 \
  sh -c 'awk "\$5==\"/\" {print \$5, \$6}" /proc/1/mountinfo'
curl -s -X POST localhost:3000/api/notes -H 'Content-Type: application/json' \
  -d '{"text":"read only"}'; echo

Expected (shortened, IDs differ):

 Container notes-api-1 Recreated
 Container notes-migrate-1 Recreated
 Container notes-web-1 Recreated
 Container notes-worker-1 Recreated
touch: /tmp/x: Read-only file system
touch: /app/x: Read-only file system
touch: cannot touch '/tmp/x': Read-only file system
/ ro,relatime
{"id":"b720e8f5-…","text":"read only"}

All three keep running – they don’t write anything to disk. But now an attacker can’t drop any tools there either. Many programs still need a /tmp; for web you provide it as tmpfs (in memory, disappears with the container):

  web:
    <<: *hardening
    # …
    tmpfs: [/tmp]
docker compose up -d --wait web >/dev/null 2>&1
docker compose exec web sh -c 'touch /tmp/x && echo "/tmp beschreibbar"
  grep " /tmp " /proc/mounts'

Expected:

/tmp beschreibbar
tmpfs /tmp tmpfs rw,nosuid,nodev,noexec,relatime,inode64 0 0

Docker mounts tmpfs with noexec: whatever lands there can’t be executed.

Save your progress? Sign up for free

Drill 5: Capabilities and no-new-privileges

Extend the anchor:

x-hardening: &hardening
  read_only: true
  cap_drop: [ALL]
  security_opt: ["no-new-privileges:true"]
docker compose up -d --wait >/dev/null 2>&1
docker compose exec web grep -E '^(CapEff|CapBnd|NoNewPrivs)' /proc/1/status
docker compose exec -u 0 web sh -c 'touch /tmp/f && chown node /tmp/f
  cat /etc/shadow >/dev/null && echo "shadow lesbar"'

Expected:

CapEff: 0000000000000000
CapBnd: 0000000000000000
NoNewPrivs: 1
chown: /tmp/f: Operation not permitted
shadow lesbar
  • cap_drop: [ALL] empties the bounding set. Even root may no longer take over other people’s files (CHOWN), open raw sockets (NET_RAW) or change the UID (SETUID).
  • /etc/shadow belongs to root – owner permissions don’t need a capability (“shadow lesbar” = “shadow readable”). That’s why a non-root user (from the Dockerfile or via user:) remains mandatory.
  • no-new-privileges prevents a process from gaining more privileges than it had at startup via setuid programs or file capabilities.

If a service does need a capability, you give back exactly that one with cap_add: (Day 11: the proxy for port 443).

Save your progress? Sign up for free

Drill 6: Postgres hardened

Now db. First just the anchor:

  db:
    <<: *hardening
    image: postgres:18.6-alpine
    # …
docker compose up -d --wait --wait-timeout 20 db 2>&1 | tail -1
docker compose ps -a db --format 'table {{.Service}}\t{{.Status}}'
docker compose logs db | tail -4

Expected (shortened):

container notes-db-1 is unhealthy
SERVICE   STATUS
db        Restarting (1) Less than a second ago
db-1  | find: /var/lib/postgresql/18/docker: Permission denied
db-1  | chmod: /var/lib/postgresql/18/docker: Operation not permitted
db-1  | chmod: /var/run/postgresql: Read-only file system
db-1  | find: /var/lib/postgresql/18/docker: Permission denied

The entrypoint script starts as root, wants to fix permissions in the data folder and then switch to postgres via su-exec. Without capabilities, root isn’t even allowed into the folder owned by UID 70. So start as UID 70 right away (the Alpine image has postgres with UID/GID 70):

    user: "70:70"
docker compose up -d --wait --wait-timeout 20 db 2>&1 | tail -1
docker compose logs db | grep FATAL | tail -1

Expected (shortened):

container notes-db-1 is unhealthy
db-1  | … FATAL:  could not create lock file "/var/run/postgresql/.s.PGSQL.5432.lock":
Read-only file system

Postgres puts its socket and lock file in /var/run/postgresql. That and /tmp get tmpfs, plus more time to shut down:

    user: "70:70"
    tmpfs: [/var/run/postgresql, /tmp]
    stop_grace_period: 30s
docker compose up -d --wait db 2>&1 | grep -E 'db-1 (Recreated|Healthy)'
docker compose exec db sh -c 'id; grep -E "^(CapEff|NoNewPrivs)" /proc/1/status'
docker compose stop db
docker compose logs db | grep -E 'shutdown request|shut down$' | tail -2
docker compose up -d --wait >/dev/null 2>&1; echo "Exit: $?"

Expected (shortened, times differ):

 Container notes-db-1 Recreated
 Container notes-db-1 Healthy
uid=70(postgres) gid=70(postgres) groups=70(postgres)
CapEff: 0000000000000000
NoNewPrivs: 1
 Container notes-db-1 Stopping
 Container notes-db-1 Stopped
db-1  | … [1] LOG:  received fast shutdown request
db-1  | … [1] LOG:  database system is shut down
Exit: 0

docker compose stop sends the image’s stop signal and waits stop_grace_period (default according to the reference: 10 s), then SIGKILL follows. Under load, Postgres needs longer for a clean checkpoint – a kill would mean crash recovery on the next start.

The volume from Day 8 works without changes: the entrypoint script had already handed the files over to UID 70 back then. Postgres also initializes an empty volume as UID 70 (verified live with down -v), including the init script from Day 4.

Save your progress? Sign up for free

Drill 7: init and logging

worker gets an init process:

  worker:
    <<: *hardening
    # …
    init: true
docker compose up -d --wait worker >/dev/null 2>&1
docker compose exec worker sh -c 'for p in /proc/[0-9]*; do
  printf "%s " "${p#/proc/}"; tr "\0" " " < "$p/cmdline"; echo; done'

Expected (shortened):

1 /sbin/docker-init -- notes-worker
7 /app/.venv/bin/python /app/.venv/bin/notes-worker
8 sh -c for p in /proc/[0-9]*; do …

docker-init is PID 1, forwards signals and reaps orphaned child processes. The Python process does catch SIGTERM itself, but without init nobody collects zombies if the worker ever starts subprocesses.

Now the logs. Without settings, Docker uses the daemon’s default driver:

docker info --format '{{.LoggingDriver}}'

Expected:

json-file

Without options, json-file never rotates – a chatty service fills up the VM’s disk. The local driver rotates on its own (default according to the docs: 5 files of 20 MB each, compressed). You can see the difference with two helper containers that each write 100,000 lines:

docker run -d --name notes-chatter --label lab=compose-lab --log-driver local \
  --log-opt max-size=100k --log-opt max-file=2 alpine:3.24 \
  sh -c 'seq 1 100000; sleep 600' >/dev/null
docker run -d --name notes-chatter2 --label lab=compose-lab alpine:3.24 \
  sh -c 'seq 1 100000; sleep 600' >/dev/null
sleep 3
for c in notes-chatter notes-chatter2; do
  echo "$c: $(docker logs "$c" | wc -l) Zeilen, erste: $(docker logs "$c" | head -1)"
done
docker rm -f notes-chatter notes-chatter2 >/dev/null

Expected (shortened, numbers differ):

notes-chatter:     5704 Zeilen, erste: 94297
notes-chatter2:   100000 Zeilen, erste: 1

With local and max-size=100k/max-file=2, only the last 6,000 or so lines remain; the rest has been rotated away. For the stack, this goes into the anchor:

x-hardening: &hardening
  read_only: true
  cap_drop: [ALL]
  security_opt: ["no-new-privileges:true"]
  logging:
    driver: local
    options:
      max-size: "10m"
      max-file: "3"
docker compose up -d --wait >/dev/null 2>&1; echo "Exit: $?"
docker compose logs --tail 1 api
docker inspect -f '{{.HostConfig.LogConfig}}' notes-api-1

Expected (shortened):

Exit: 0
api-1  | time=… level=INFO msg=listening addr=:8080 version=dev pid=1

docker inspect additionally shows the logging driver local with the options from the file, max-size 10m and max-file 3.

docker compose logs works with local as usual. At most 30 MB of logs per container; after that the oldest lines are dropped – whoever needs them longer sends them to a central system (Day 12, Enterprise Note).

Save your progress? Sign up for free

Drill 8: Development, tools and the limits of the anchor

The complete file is in code/tag09/notes/compose.yaml. Five services use the anchor:

grep -n '<<: \*hardening' compose.yaml

Expected:

25:    <<: *hardening
53:    <<: *hardening
74:    <<: *hardening
99:    <<: *hardening
148:    <<: *hardening

In development mode (Day 7), watch syncs files to /app – that fails because of read_only. compose.dev.yaml turns it off for web:

# compose.dev.yaml – development, only via -f together with compose.yaml (as of Day 9)
services:
  web:
    build:
      target: build
    command: ["node", "--watch", "server.js"]
    environment:
      NODE_ENV: development
    read_only: false
TAG=watch docker compose -f compose.yaml -f compose.dev.yaml \
  up -d --build --wait --no-deps web 2>&1 | grep -E 'Built|Healthy'
docker compose exec web sh -c 'id -u; touch /app/public/x && echo "/app beschreibbar"
  rm -f /app/public/x'
docker compose up -d --wait >/dev/null 2>&1
docker image rm notes-web:watch >/dev/null

Expected (shortened):

 Image notes-web:watch Built
 Container notes-web-1 Healthy
0
/app beschreibbar

Without the line read_only: false, the same touch reports Read-only file system (verified live). The build stage runs as root; it may write despite cap_drop: [ALL] because /app belongs to root. TAG=watch keeps the dev image away from notes-web:dev (Day 8).

psql and backup in compose.tools.yaml stay unhardened: they run for seconds, have no ports, and backup writes to ./backup. The anchor wouldn’t even work there:

cp compose.tools.yaml tools.bak
sed -i.tmp 's/^    profiles: \[tools\]/    <<: *hardening\n    profiles: [tools]/' \
  compose.tools.yaml
docker compose up --dry-run -d 2>&1 | tail -1
mv tools.bak compose.tools.yaml && rm -f compose.tools.yaml.tmp

Expected (shortened):

failed to parse /Users/du/compose-lab/tag09/compose.tools.yaml: go-yaml load error in
composer at L5.C9: unknown anchor 'hardening' referenced

YAML anchors only apply in the file in which they are defined – even with include. Whoever needs the hardening in multiple files uses extends (Day 7) on a hardened base service.

Save your progress? Sign up for free

Checkpoint

Five services with the anchor:

cd ~/compose-lab/tag09
grep -c '<<: \*hardening' compose.yaml

Expected:

5

The stack is healthy, the memory limits apply:

docker compose up -d --wait
docker stats --no-stream --format '{{.Name}} {{.MemUsage}}'

Expected (shortened, usage differs):

 Container notes-db-1 Healthy
…
 Container notes-web-1 Healthy
notes-web-1 20.17MiB / 128MiB
notes-worker-1 22.7MiB / 128MiB
notes-api-1 2.082MiB / 64MiB
notes-db-1 29.23MiB / 512MiB

web: / read-only, /tmp writable:

docker compose exec web touch /x
docker compose exec web touch /tmp/x; echo "Exit: $?"

Expected:

touch: /x: Read-only file system
Exit: 0

db runs as UID 70 with no-new-privileges:

docker compose exec db id -u
docker compose exec db grep NoNewPrivs /proc/1/status

Expected:

70
NoNewPrivs: 1

worker has init as PID 1:

docker compose exec worker cat /proc/1/cmdline | tr '\0' ' '; echo

Expected:

/sbin/docker-init -- notes-worker

POST via web:

curl -s -o /dev/null -w '%{http_code}\n' -X POST localhost:3000/api/notes \
  -H 'Content-Type: application/json' -d '{"text":"Kontrollpunkt 9"}'

Expected:

201

Challenge

Check from the inside whether notes-api-1 runs hardened: root file system read-only, no-new-privileges active, no capabilities, memory limit set. Prove with a psql container that the same commands reveal what is unhardened.

Hint: The helper container with --pid container:<name> from Drill 1 reads /proc/1/status and /proc/1/mountinfo of any container, even with scratch.

Solution

NoNewPrivs and the capability mask CapBnd from /proc/1/status:

docker run --rm --pid container:notes-api-1 alpine:3.24 \
  grep -E '^(NoNewPrivs|CapBnd)' /proc/1/status

Expected:

CapBnd: 0000000000000000
NoNewPrivs: 1

1 means active, all zeros means no capabilities. The line for the root file system in /proc/1/mountinfo has / / in the middle:

docker run --rm --pid container:notes-api-1 alpine:3.24 \
  grep ' / / ' /proc/1/mountinfo

Expected (shortened):

495 466 0:48 / / ro,relatime - overlay overlay rw,lowerdir=/var/lib/containerd/…

ro comes first in the options. docker stats shows the memory limit:

docker stats --no-stream --format '{{.Name}} {{.MemUsage}}' notes-api-1

Expected (usage differs):

notes-api-1 2.227MiB / 64MiB

Now a tool container next to it that sleeps for a minute, and the same commands:

docker compose run -d --rm --name notes-psql-probe --entrypoint sleep psql 60
docker run --rm --pid container:notes-psql-probe alpine:3.24 \
  grep -E '^(NoNewPrivs|CapBnd)' /proc/1/status
docker run --rm --pid container:notes-psql-probe alpine:3.24 \
  grep ' / / ' /proc/1/mountinfo
docker stats --no-stream --format '{{.Name}} {{.MemUsage}}' notes-psql-probe

Expected (shortened, IDs differ):

 Container notes-db-1 Healthy
9ab57f918e86…
CapBnd: 00000000a80425fb
NoNewPrivs: 0
665 630 0:63 / / rw,relatime - overlay overlay rw,lowerdir=/var/lib/containerd/…
notes-psql-probe 348KiB / 7.755GiB

NoNewPrivs: 0, a full capability mask, rw, and the entire VM memory as the limit: the commands find all four gaps. Because they look from the inside, they show what really applies, not what is written in the YAML. To finish the day: docker compose down (without -v).

Save your progress? Sign up for free

Enterprise Note

  • Limits without measurement are guesswork. First observe docker stats or metrics (Day 12) under load, then limit = peak plus headroom; OOM events belong in alerting.
  • Kubernetes separates requests (scheduling) and limits (cap) more cleanly; the Pod Security Standards “restricted” profile requires exactly what you did here: non-root, drop: [ALL], allowPrivilegeEscalation: false, plus readOnlyRootFilesystem.
  • security_opt also supports seccomp and AppArmor profiles. Docker’s default profile is active as long as nobody sets seccomp=unconfined – that belongs in every review.
  • Harden Docker itself: rootless mode or user namespaces (userns-remap), no privileged, never mount the Docker socket into containers.
  • Logs with local stay on the host; in operation, a driver (fluentd, gelf, journald) or an agent sends them to a central system with retention rules.