Day 9: Resources and Hardening
Goal: Every service gets limits for CPU, memory and processes, a read-only file system, no capabilities and rotating logs – defined together in an extension field. · Duration: approx. 75 min · Prerequisite: Day 8
Briefing
- Without settings, a container may use all the memory and all CPUs of the Docker host, start any number of processes and write to its file system.
deploy.resources.limitssets limits forcpus,memoryandpids. Whoever wants too much memory gets killed by the kernel (OOM kill, exit code 137).read_only,cap_drop: [ALL],no-new-privilegesand a non-rootusertake away almost everything an attacker inside the container would need.- The
localdriver rotates logs withmax-size/max-file. - An extension field
x-hardeningwith a YAML anchor keeps the hardening in one place.
x-hardening: &hardening ──<<:──► api, migrate, worker, web, db
read_only · cap_drop [ALL] · no-new-privileges · logging local 10m×3
per service: deploy.resources.limits { cpus, memory, pids } · tmpfs · user · init
Drill 1: What a container may do today
source ~/compose-lab/setup/env.sh
docker compose -p notes down
mkdir -p ~/compose-lab/tag09 && cd ~/compose-lab/tag09
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag08/notes/." .
docker compose up -d --build --wait >/dev/null 2>&1; echo "Exit: $?"
docker stats --no-stream --format \
'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.PIDs}}' \
notes-api-1 notes-worker-1 notes-web-1 notes-db-1Expected (shortened, values differ on your machine):
Exit: 0
NAME CPU % MEM USAGE / LIMIT PIDS
notes-api-1 0.00% 2.625MiB / 7.737GiB 8
notes-worker-1 0.08% 23.58MiB / 7.737GiB 1
notes-web-1 0.00% 19.98MiB / 7.737GiB 11
notes-db-1 0.04% 29.52MiB / 7.737GiB 10
The “limit” is the entire memory of the Colima VM. A memory leak in one service takes the space away from all the others. Now the permissions, seen from the inside:
docker compose exec web sh -c 'id; grep -E "^(CapEff|CapBnd|NoNewPrivs)" /proc/1/status
touch /tmp/x && echo "/tmp beschreibbar"'
docker compose exec -u 0 web sh -c 'grep CapEff /proc/self/status
chown node /app && echo "chown als root ok"'Expected:
uid=1000(node) gid=1000(node) groups=1000(node)
CapEff: 0000000000000000
CapBnd: 00000000a80425fb
NoNewPrivs: 0
/tmp beschreibbar
CapEff: 00000000a80425fb
chown als root ok
CapBnd is the upper bound of the capabilities (Docker
default: 14 of them, such as CHOWN, NET_RAW,
SETUID). User node has no effective ones, but
whoever becomes root inside the container – via exec -u 0
or through a vulnerability – gets all of them.
notes-api runs in scratch without a shell. A helper
container that shares its PID namespace can still read its process
data:
docker run --rm --label lab=compose-lab --pid container:notes-api-1 alpine:3.24 \
sh -c 'grep -E "^(Uid|CapBnd|NoNewPrivs)" /proc/1/status
awk "\$5==\"/\" {print \$5, \$6}" /proc/1/mountinfo'Expected:
Uid: 65532 65532 65532 65532
CapBnd: 00000000a80425fb
NoNewPrivs: 0
/ rw,relatime
Non-root, but capabilities in the bounding set and a writable root
file system (rw). You’ll need this helper container often
today.
Save your progress? Sign up for free
Drill 2: Limits for CPU, memory and processes
Add the limits to api, and, as the training plan says,
pids_limit too:
api:
# … as on Day 8 …
deploy:
resources:
limits:
cpus: "0.50"
memory: 64M
pids_limit: 100The same for worker ("0.25"/128M), web
("0.50"/128M), db
("1.0"/512M, pids_limit: 200) and
migrate (like api, without pids_limit).
docker compose up -d --wait; echo "Exit: $?"Expected (shortened, the service named may differ):
services.worker: can't set distinct values on 'pids_limit' and
'deploy.resources.limits.pids': invalid compose project
Exit: 1
Compose 5.6 treats pids_limit and
deploy.resources.limits.pids as the same field. If
limits is present, pids counts as 0 there –
and that contradicts the 100 (verified live; without
deploy:, pids_limit alone works). So the
process limit belongs in the limits block:
deploy:
resources:
limits:
cpus: "0.50"
memory: 64M
pids: 100Replace pids_limit accordingly in all four services (db:
pids: 200).
docker compose up -d --wait >/dev/null 2>&1; echo "Exit: $?"
docker stats --no-stream --format 'table {{.Name}}\t{{.MemUsage}}\t{{.PIDs}}' \
notes-api-1 notes-worker-1 notes-web-1 notes-db-1
docker compose exec web cat /sys/fs/cgroup/cpu.max /sys/fs/cgroup/memory.max \
/sys/fs/cgroup/pids.maxExpected (shortened):
Exit: 0
NAME MEM USAGE / LIMIT PIDS
notes-api-1 2.188MiB / 64MiB 7
notes-worker-1 23.59MiB / 128MiB 2
notes-web-1 24.17MiB / 128MiB 11
notes-db-1 32.29MiB / 512MiB 10
50000 100000
134217728
100
The limits end up as cgroup values in the container:
cpu.max means “50 ms of CPU time per 100 ms”, i.e. half a
CPU. Try it with an endless loop in Node:
docker compose exec -d web \
node -e 'const t = Date.now(); while (Date.now() - t < 15000) {}'
sleep 6
docker stats --no-stream --format 'table {{.Name}}\t{{.CPUPerc}}' notes-web-1
docker compose exec web grep -E 'nr_throttled|throttled_usec' /sys/fs/cgroup/cpu.statExpected (shortened, numbers differ):
NAME CPU %
notes-web-1 51.34%
nr_throttled 75
throttled_usec 3694773
There’s no more than 50 %; the rest of the time is throttled. Besides
limits, deploy.resources also supports
reservations. In the test, a memory reservation of
256M ended up as memory.low (a soft guarantee
under memory pressure); a CPU reservation changed nothing visible in the
cgroup. The final state sticks with limits.
Tip: The reference also knows
mem_limitandcpusdirectly on the service. According to the docs, they must matchdeploy.resources.limitsif both are set – in this book everything lives uniformly underdeploy.
Save your progress? Sign up for free
Drill 3: When the limit kicks in
What happens with 200 MB in a container with a 128 MB limit?
docker compose events shows what Docker reports:
docker compose events --json worker > events.json &
docker compose exec worker python -c 'b = bytearray(200 * 1024 * 1024); print("ok")'
echo "Exit: $?"
sleep 2; kill %1
jq -c '{action, service}' events.json
docker compose ps worker --format 'table {{.Service}}\t{{.Status}}'
docker compose exec worker cat /sys/fs/cgroup/memory.events
rm events.jsonExpected (shortened):
Exit: 137
{"action":"exec_create: python -c b = bytearray(200 * 1024 * 1024); …",…}
{"action":"exec_start: python -c b = bytearray(200 * 1024 * 1024); …",…}
{"action":"oom","service":"worker"}
{"action":"exec_die","service":"worker"}
SERVICE STATUS
worker Up About a minute
low 0
high 0
max 35
oom 1
oom_kill 1
oom_group_kill 0
Exit 137 = 128 + 9 (SIGKILL). The kernel killed the largest process
in the cgroup – here the exec process; the worker itself
kept running. If it hits the main process, it looks different. An
additional file temporarily sets worker’s limit to 16 MB:
cat > compose.oom.yaml <<'EOF'
# compose.oom.yaml – only for Day 9, Drill 3
services:
worker:
deploy:
resources:
limits:
memory: 16M
EOF
docker compose -f compose.yaml -f compose.oom.yaml up -d --no-deps worker
sleep 10
docker compose ps -a worker --format 'table {{.Service}}\t{{.Status}}'
docker compose logs worker | wc -l
docker compose events --json --since 15s --until 1s worker 2>/dev/null \
| jq -r 'select(.action == "oom" or .action == "die")
| "\(.time[11:19]) \(.action) \(.attributes.exitCode // "")"' | tail -4Expected (shortened, times differ):
Container notes-worker-1 Recreate
Container notes-worker-1 Recreated
Container notes-worker-1 Started
SERVICE STATUS
worker Restarting (137) 3 seconds ago
0
23:33:17 oom
23:33:17 die 137
23:33:20 oom
23:33:20 die 137
That’s the typical picture of a limit that’s too tight:
Restarting (137), not a single log line –
Python dies while loading its modules – and only the events name the
reason. --until ends the event stream; Compose 5.6 then
reports an EOF on stderr, hence
2>/dev/null. Back to the correct limit:
rm compose.oom.yaml
docker compose up -d --wait worker 2>&1 | grep -E 'Recreated|Healthy'Expected:
Container notes-worker-1 Recreated
Container notes-worker-1 Healthy
The process limit is just as hard, for web with
pids: 100:
docker compose exec web sh -c 'for i in $(seq 1 120); do sleep 5 & done; wait'
docker compose exec web cat /sys/fs/cgroup/pids.eventsExpected:
sh: can't fork: Resource temporarily unavailable
max 1
That way, a fork bomb or a thread leak only cripples its own container, not the host.
Save your progress? Sign up for free
Drill 4: Read-only with tmpfs
From now on you collect the hardening in an extension field. Add
below x-api-build:
x-hardening: &hardening
read_only: trueand in api, migrate, worker and web put
<<: *hardening as the first line:
api:
<<: *hardening
build: *api-build
# …<< is YAML’s merge key: it copies all keys of the
anchor into the service. Whatever the service sets itself wins.
docker compose up -d --wait 2>&1 | grep -E 'Recreated' | sort -u
docker compose exec web sh -c 'touch /tmp/x; touch /app/x'
docker compose exec worker touch /tmp/x
docker run --rm --label lab=compose-lab --pid container:notes-api-1 alpine:3.24 \
sh -c 'awk "\$5==\"/\" {print \$5, \$6}" /proc/1/mountinfo'
curl -s -X POST localhost:3000/api/notes -H 'Content-Type: application/json' \
-d '{"text":"read only"}'; echoExpected (shortened, IDs differ):
Container notes-api-1 Recreated
Container notes-migrate-1 Recreated
Container notes-web-1 Recreated
Container notes-worker-1 Recreated
touch: /tmp/x: Read-only file system
touch: /app/x: Read-only file system
touch: cannot touch '/tmp/x': Read-only file system
/ ro,relatime
{"id":"b720e8f5-…","text":"read only"}
All three keep running – they don’t write anything to disk. But now
an attacker can’t drop any tools there either. Many programs still need
a /tmp; for web you provide it as tmpfs (in memory,
disappears with the container):
web:
<<: *hardening
# …
tmpfs: [/tmp]docker compose up -d --wait web >/dev/null 2>&1
docker compose exec web sh -c 'touch /tmp/x && echo "/tmp beschreibbar"
grep " /tmp " /proc/mounts'Expected:
/tmp beschreibbar
tmpfs /tmp tmpfs rw,nosuid,nodev,noexec,relatime,inode64 0 0
Docker mounts tmpfs with noexec: whatever lands there
can’t be executed.
Save your progress? Sign up for free
Drill 5: Capabilities and no-new-privileges
Extend the anchor:
x-hardening: &hardening
read_only: true
cap_drop: [ALL]
security_opt: ["no-new-privileges:true"]docker compose up -d --wait >/dev/null 2>&1
docker compose exec web grep -E '^(CapEff|CapBnd|NoNewPrivs)' /proc/1/status
docker compose exec -u 0 web sh -c 'touch /tmp/f && chown node /tmp/f
cat /etc/shadow >/dev/null && echo "shadow lesbar"'Expected:
CapEff: 0000000000000000
CapBnd: 0000000000000000
NoNewPrivs: 1
chown: /tmp/f: Operation not permitted
shadow lesbar
cap_drop: [ALL]empties the bounding set. Even root may no longer take over other people’s files (CHOWN), open raw sockets (NET_RAW) or change the UID (SETUID)./etc/shadowbelongs to root – owner permissions don’t need a capability (“shadow lesbar” = “shadow readable”). That’s why a non-root user (from the Dockerfile or viauser:) remains mandatory.no-new-privilegesprevents a process from gaining more privileges than it had at startup via setuid programs or file capabilities.
If a service does need a capability, you give back exactly that one
with cap_add: (Day 11: the proxy for port 443).
Save your progress? Sign up for free
Drill 6: Postgres hardened
Now db. First just the anchor:
db:
<<: *hardening
image: postgres:18.6-alpine
# …docker compose up -d --wait --wait-timeout 20 db 2>&1 | tail -1
docker compose ps -a db --format 'table {{.Service}}\t{{.Status}}'
docker compose logs db | tail -4Expected (shortened):
container notes-db-1 is unhealthy
SERVICE STATUS
db Restarting (1) Less than a second ago
db-1 | find: /var/lib/postgresql/18/docker: Permission denied
db-1 | chmod: /var/lib/postgresql/18/docker: Operation not permitted
db-1 | chmod: /var/run/postgresql: Read-only file system
db-1 | find: /var/lib/postgresql/18/docker: Permission denied
The entrypoint script starts as root, wants to fix permissions in the
data folder and then switch to postgres via
su-exec. Without capabilities, root isn’t even allowed into
the folder owned by UID 70. So start as UID 70 right away (the Alpine
image has postgres with UID/GID 70):
user: "70:70"docker compose up -d --wait --wait-timeout 20 db 2>&1 | tail -1
docker compose logs db | grep FATAL | tail -1Expected (shortened):
container notes-db-1 is unhealthy
db-1 | … FATAL: could not create lock file "/var/run/postgresql/.s.PGSQL.5432.lock":
Read-only file system
Postgres puts its socket and lock file in
/var/run/postgresql. That and /tmp get tmpfs,
plus more time to shut down:
user: "70:70"
tmpfs: [/var/run/postgresql, /tmp]
stop_grace_period: 30sdocker compose up -d --wait db 2>&1 | grep -E 'db-1 (Recreated|Healthy)'
docker compose exec db sh -c 'id; grep -E "^(CapEff|NoNewPrivs)" /proc/1/status'
docker compose stop db
docker compose logs db | grep -E 'shutdown request|shut down$' | tail -2
docker compose up -d --wait >/dev/null 2>&1; echo "Exit: $?"Expected (shortened, times differ):
Container notes-db-1 Recreated
Container notes-db-1 Healthy
uid=70(postgres) gid=70(postgres) groups=70(postgres)
CapEff: 0000000000000000
NoNewPrivs: 1
Container notes-db-1 Stopping
Container notes-db-1 Stopped
db-1 | … [1] LOG: received fast shutdown request
db-1 | … [1] LOG: database system is shut down
Exit: 0
docker compose stop sends the image’s stop signal and
waits stop_grace_period (default according to the
reference: 10 s), then SIGKILL follows. Under load, Postgres needs
longer for a clean checkpoint – a kill would mean crash recovery on the
next start.
The volume from Day 8 works without changes: the entrypoint script
had already handed the files over to UID 70 back then. Postgres also
initializes an empty volume as UID 70 (verified live with
down -v), including the init script from Day 4.
Save your progress? Sign up for free
Drill 7: init and logging
worker gets an init process:
worker:
<<: *hardening
# …
init: truedocker compose up -d --wait worker >/dev/null 2>&1
docker compose exec worker sh -c 'for p in /proc/[0-9]*; do
printf "%s " "${p#/proc/}"; tr "\0" " " < "$p/cmdline"; echo; done'Expected (shortened):
1 /sbin/docker-init -- notes-worker
7 /app/.venv/bin/python /app/.venv/bin/notes-worker
8 sh -c for p in /proc/[0-9]*; do …
docker-init is PID 1, forwards signals and reaps
orphaned child processes. The Python process does catch SIGTERM itself,
but without init nobody collects zombies if the worker ever starts
subprocesses.
Now the logs. Without settings, Docker uses the daemon’s default driver:
docker info --format '{{.LoggingDriver}}'Expected:
json-file
Without options, json-file never rotates – a chatty
service fills up the VM’s disk. The local driver rotates on
its own (default according to the docs: 5 files of 20 MB each,
compressed). You can see the difference with two helper containers that
each write 100,000 lines:
docker run -d --name notes-chatter --label lab=compose-lab --log-driver local \
--log-opt max-size=100k --log-opt max-file=2 alpine:3.24 \
sh -c 'seq 1 100000; sleep 600' >/dev/null
docker run -d --name notes-chatter2 --label lab=compose-lab alpine:3.24 \
sh -c 'seq 1 100000; sleep 600' >/dev/null
sleep 3
for c in notes-chatter notes-chatter2; do
echo "$c: $(docker logs "$c" | wc -l) Zeilen, erste: $(docker logs "$c" | head -1)"
done
docker rm -f notes-chatter notes-chatter2 >/dev/nullExpected (shortened, numbers differ):
notes-chatter: 5704 Zeilen, erste: 94297
notes-chatter2: 100000 Zeilen, erste: 1
With local and
max-size=100k/max-file=2, only the last 6,000
or so lines remain; the rest has been rotated away. For the stack, this
goes into the anchor:
x-hardening: &hardening
read_only: true
cap_drop: [ALL]
security_opt: ["no-new-privileges:true"]
logging:
driver: local
options:
max-size: "10m"
max-file: "3"docker compose up -d --wait >/dev/null 2>&1; echo "Exit: $?"
docker compose logs --tail 1 api
docker inspect -f '{{.HostConfig.LogConfig}}' notes-api-1Expected (shortened):
Exit: 0
api-1 | time=… level=INFO msg=listening addr=:8080 version=dev pid=1
docker inspect additionally shows the logging driver
local with the options from the file, max-size
10m and max-file 3.
docker compose logs works with local as
usual. At most 30 MB of logs per container; after that the oldest lines
are dropped – whoever needs them longer sends them to a central system
(Day 12, Enterprise Note).
Save your progress? Sign up for free
Drill 8: Development, tools and the limits of the anchor
The complete file is in code/tag09/notes/compose.yaml.
Five services use the anchor:
grep -n '<<: \*hardening' compose.yamlExpected:
25: <<: *hardening
53: <<: *hardening
74: <<: *hardening
99: <<: *hardening
148: <<: *hardening
In development mode (Day 7), watch syncs files to
/app – that fails because of read_only.
compose.dev.yaml turns it off for web:
# compose.dev.yaml – development, only via -f together with compose.yaml (as of Day 9)
services:
web:
build:
target: build
command: ["node", "--watch", "server.js"]
environment:
NODE_ENV: development
read_only: falseTAG=watch docker compose -f compose.yaml -f compose.dev.yaml \
up -d --build --wait --no-deps web 2>&1 | grep -E 'Built|Healthy'
docker compose exec web sh -c 'id -u; touch /app/public/x && echo "/app beschreibbar"
rm -f /app/public/x'
docker compose up -d --wait >/dev/null 2>&1
docker image rm notes-web:watch >/dev/nullExpected (shortened):
Image notes-web:watch Built
Container notes-web-1 Healthy
0
/app beschreibbar
Without the line read_only: false, the same
touch reports Read-only file system (verified
live). The build stage runs as root; it may write despite
cap_drop: [ALL] because /app belongs to root.
TAG=watch keeps the dev image away from
notes-web:dev (Day 8).
psql and backup in
compose.tools.yaml stay unhardened: they run for seconds,
have no ports, and backup writes to ./backup.
The anchor wouldn’t even work there:
cp compose.tools.yaml tools.bak
sed -i.tmp 's/^ profiles: \[tools\]/ <<: *hardening\n profiles: [tools]/' \
compose.tools.yaml
docker compose up --dry-run -d 2>&1 | tail -1
mv tools.bak compose.tools.yaml && rm -f compose.tools.yaml.tmpExpected (shortened):
failed to parse /Users/du/compose-lab/tag09/compose.tools.yaml: go-yaml load error in
composer at L5.C9: unknown anchor 'hardening' referenced
YAML anchors only apply in the file in which they are defined – even
with include. Whoever needs the hardening in multiple files
uses extends (Day 7) on a hardened base service.
Save your progress? Sign up for free
Checkpoint
Five services with the anchor:
cd ~/compose-lab/tag09
grep -c '<<: \*hardening' compose.yamlExpected:
5
The stack is healthy, the memory limits apply:
docker compose up -d --wait
docker stats --no-stream --format '{{.Name}} {{.MemUsage}}'Expected (shortened, usage differs):
Container notes-db-1 Healthy
…
Container notes-web-1 Healthy
notes-web-1 20.17MiB / 128MiB
notes-worker-1 22.7MiB / 128MiB
notes-api-1 2.082MiB / 64MiB
notes-db-1 29.23MiB / 512MiB
web: / read-only, /tmp writable:
docker compose exec web touch /x
docker compose exec web touch /tmp/x; echo "Exit: $?"Expected:
touch: /x: Read-only file system
Exit: 0
db runs as UID 70 with no-new-privileges:
docker compose exec db id -u
docker compose exec db grep NoNewPrivs /proc/1/statusExpected:
70
NoNewPrivs: 1
worker has init as PID 1:
docker compose exec worker cat /proc/1/cmdline | tr '\0' ' '; echoExpected:
/sbin/docker-init -- notes-worker
POST via web:
curl -s -o /dev/null -w '%{http_code}\n' -X POST localhost:3000/api/notes \
-H 'Content-Type: application/json' -d '{"text":"Kontrollpunkt 9"}'Expected:
201
Challenge
Check from the inside whether notes-api-1 runs hardened:
root file system read-only, no-new-privileges active, no
capabilities, memory limit set. Prove with a psql container
that the same commands reveal what is unhardened.
Hint: The helper container with
--pid container:<name> from Drill 1 reads
/proc/1/status and /proc/1/mountinfo of any
container, even with scratch.
Solution
NoNewPrivs and the capability mask CapBnd
from /proc/1/status:
docker run --rm --pid container:notes-api-1 alpine:3.24 \
grep -E '^(NoNewPrivs|CapBnd)' /proc/1/statusExpected:
CapBnd: 0000000000000000
NoNewPrivs: 1
1 means active, all zeros means no capabilities. The
line for the root file system in /proc/1/mountinfo has
/ / in the middle:
docker run --rm --pid container:notes-api-1 alpine:3.24 \
grep ' / / ' /proc/1/mountinfoExpected (shortened):
495 466 0:48 / / ro,relatime - overlay overlay rw,lowerdir=/var/lib/containerd/…
ro comes first in the options. docker stats
shows the memory limit:
docker stats --no-stream --format '{{.Name}} {{.MemUsage}}' notes-api-1Expected (usage differs):
notes-api-1 2.227MiB / 64MiB
Now a tool container next to it that sleeps for a minute, and the same commands:
docker compose run -d --rm --name notes-psql-probe --entrypoint sleep psql 60
docker run --rm --pid container:notes-psql-probe alpine:3.24 \
grep -E '^(NoNewPrivs|CapBnd)' /proc/1/status
docker run --rm --pid container:notes-psql-probe alpine:3.24 \
grep ' / / ' /proc/1/mountinfo
docker stats --no-stream --format '{{.Name}} {{.MemUsage}}' notes-psql-probeExpected (shortened, IDs differ):
Container notes-db-1 Healthy
9ab57f918e86…
CapBnd: 00000000a80425fb
NoNewPrivs: 0
665 630 0:63 / / rw,relatime - overlay overlay rw,lowerdir=/var/lib/containerd/…
notes-psql-probe 348KiB / 7.755GiB
NoNewPrivs: 0, a full capability mask, rw,
and the entire VM memory as the limit: the commands find all four gaps.
Because they look from the inside, they show what really applies, not
what is written in the YAML. To finish the day:
docker compose down (without -v).
Save your progress? Sign up for free
Enterprise Note
- Limits without measurement are guesswork. First observe
docker statsor metrics (Day 12) under load, then limit = peak plus headroom; OOM events belong in alerting. - Kubernetes separates
requests(scheduling) andlimits(cap) more cleanly; the Pod Security Standards “restricted” profile requires exactly what you did here: non-root,drop: [ALL],allowPrivilegeEscalation: false, plusreadOnlyRootFilesystem. security_optalso supports seccomp and AppArmor profiles. Docker’s default profile is active as long as nobody setsseccomp=unconfined– that belongs in every review.- Harden Docker itself: rootless mode or user namespaces
(
userns-remap), noprivileged, never mount the Docker socket into containers. - Logs with
localstay on the host; in operation, a driver (fluentd,gelf,journald) or an agent sends them to a central system with retention rules.