Day 1: Linux internals
Goal: Fully inspect and constrain a model server process using /proc, signals, strace, cgroups v2, namespaces and systemd · Duration: approx. 120 min · Prerequisite: Day 0
Briefing
- To Linux, an LLM server is just a process: PID, file descriptors, memory, signals.
- cgroups v2 limit resources
(
memory.max,cpu.max) and count events (memory.events). - Namespaces isolate the view (PID, network, hostname, mounts). Container = namespaces + cgroups.
- systemd writes your limits into cgroup files and logs OOM kills in the journal.
- PSI (
/proc/pressure/*) shows how long processes wait for CPU, RAM or I/O.
systemd ── fake-llm.service ── cgroup /system.slice/fake-llm.service
memory.max = 300M cpu.max = 50000 100000
└── PID 132 python3 (ns: pid, net, mnt, uts)
All drills run on the VM (multipass shell llm-vm). A
Python HTTP server on port 11434 stands in for Ollama – from Day 5 on
you replace it with the real model server.
Drill 1: A process under the microscope
mkdir -p ~/tag01 && cd ~/tag01
python3 -m http.server 11434 >/dev/null 2>&1 &
PID=$!; sleep 2
grep -E '^(Name|State|PPid|Threads|VmRSS)' /proc/$PID/status
ls -l /proc/$PID/fd
ss -tlnp | grep 11434Expected (PIDs and inodes vary):
Name: python3
State: S (sleeping)
PPid: 5019
VmRSS: 18520 kB
Threads: 1
lr-x------ 1 root root 64 Oct 9 16:50 0 -> /dev/null
l-wx------ 1 root root 64 Oct 9 16:50 1 -> /dev/null
l-wx------ 1 root root 64 Oct 9 16:50 2 -> /dev/null
lrwx------ 1 root root 64 Oct 9 16:50 3 -> socket:[812183]
LISTEN 0 5 0.0.0.0:11434 0.0.0.0:* users:(("python3",pid=5025,fd=3))
Why? fd 3 is the listening socket. Send-Q 5 for
LISTEN is the backlog length – Python’s default is tiny,
real servers use 4096 and more.
Read limits and change them at runtime:
grep -E 'open files|processes' /proc/$PID/limits
sudo prlimit --pid $PID --nofile=4096:4096
grep 'open files' /proc/$PID/limits
tr '\0' ' ' < /proc/$PID/cmdline; echo
readlink /proc/$PID/exeExpected (abbreviated, values may differ):
Max processes 31398 31398 processes
Max open files 1024 524288 files
Max open files 4096 4096 files
python3 -m http.server 11434
/usr/bin/python3.12
Tip:
Too many open fileswith many concurrent streams? First check/proc/<pid>/limits, thenls /proc/<pid>/fd | wc -l– the cause is almost always a soft limit that is too small.
Save your progress? Sign up for free
Drill 2: Signals
kill -STOP $PID
grep State /proc/$PID/status
for i in 1 2 3; do curl -s -m 1 localhost:11434 >/dev/null & done; sleep 0.5
ss -tln sport = :11434Expected:
State: T (stopped)
State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN 3 5 0.0.0.0:11434 0.0.0.0:*
Why? The kernel handles the TCP handshake. The three connections sit
in the accept queue (Recv-Q 3), but nobody picks them up.
An open port does not mean a healthy service – that is
why health checks verify a real response (Day 4:
/readyz).
kill -CONT $PID; kill -TERM $PID; wait $PID; echo "exit=$?"Expected:
exit=143
Exit code 128 + signal number: 143 = SIGTERM (15), 137 = SIGKILL (9).
You will see these numbers again later as exitCode in
Kubernetes.
A server that handles SIGTERM cleanly:
cat > server.sh <<'EOF'
#!/usr/bin/env bash
graceful() {
echo "SIGTERM: laufende Requests beenden ..."
sleep 1
echo "sauber beendet"
exit 0
}
trap graceful TERM
echo "Server läuft als PID $$"
while true; do sleep 1; done
EOF
chmod +x server.sh
./server.sh & P=$!; sleep 1.5; kill -TERM $P; wait $P; echo "exit=$?"
./server.sh & P=$!; sleep 1.5; kill -KILL $P; wait $P; echo "exit=$?"Expected:
Server läuft als PID 5075
SIGTERM: laufende Requests beenden ...
sauber beendet
exit=0
Server läuft als PID 5080
bash: line 12: 5080 Killed ./server.sh
exit=137
Why? No process can catch SIGKILL. On SIGTERM, an LLM server needs time to finish running streams (drain) – otherwise every response breaks off mid-sentence.
Save your progress? Sign up for free
Drill 3: strace – what is the process really doing?
python3 -m http.server 11434 >/dev/null 2>&1 &
PID=$!; sleep 2
(sleep 1; curl -s localhost:11434/ >/dev/null) &
sudo timeout 3 strace -f -p $PID -e trace=accept4,recvfrom,sendto,openat
kill $PIDExpected (abbreviated):
strace: Process 5127 attached
accept4(3, {sa_family=AF_INET, sin_port=htons(60510), ...}, [16], SOCK_CLOEXEC) = 4
strace: Process 5137 attached
[pid 5137] recvfrom(4, "GET / HTTP/1.1\r\nHost: localhost:"..., 8192, ...) = 78
[pid 5137] openat(AT_FDCWD, "//", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 5
[pid 5137] sendto(4, "HTTP/1.0 200 OK\r\nServer: SimpleH"..., 156, 0, NULL, 0) = 156
[pid 5137] +++ exited with 0 +++
Why -f? The server starts a thread per request
(Process 5137 attached). Without -f you only
see the accept4.
Where is a client hanging? -T shows the duration of each
syscall:
strace -T -e trace=connect curl -s -m 2 http://10.255.255.1:80Expected (abbreviated):
connect(5, {sa_family=AF_INET, sin_port=htons(80), ...}, 16) = -1 EINPROGRESS
(Operation now in progress) <0.000452>
+++ exited with 28 +++
A non-blocking connect, followed by a timeout (curl exit
28). Summary of all syscalls:
strace -c -f <befehl>.
Warning: strace slows the process down massively. In production use it only briefly and selectively (
-e trace=...), never permanently on a model server under load.
Save your progress? Sign up for free
Drill 4: cgroups v2 – memory limit and OOM kill
The memory hog simulates a model that needs more RAM than allowed while loading:
cat > hog.py <<'EOF'
import sys
import time
step = int(sys.argv[1]) if len(sys.argv) > 1 else 50
buf = []
while True:
buf.append(bytearray(step * 1024 * 1024))
print(f"{len(buf) * step} MB belegt", flush=True)
time.sleep(0.2)
EOF
cat /sys/fs/cgroup/cgroup.subtree_control
sudo mkdir /sys/fs/cgroup/llm-demo
echo 200M | sudo tee /sys/fs/cgroup/llm-demo/memory.max
echo 0 | sudo tee /sys/fs/cgroup/llm-demo/memory.swap.max
sudo bash -c 'echo $$ > /sys/fs/cgroup/llm-demo/cgroup.procs; exec python3 hog.py 50'
echo "exit=$?"Expected (abbreviated, controller list varies):
cpu memory pids
200M
0
50 MB belegt
100 MB belegt
150 MB belegt
Killed
exit=137
Why echo $$ > cgroup.procs? A process ends up in a
cgroup when you write its PID into cgroup.procs; through
exec, Python inherits it.
cat /sys/fs/cgroup/llm-demo/memory.events
sudo dmesg | grep -i 'memory cgroup out of memory' | tail -1Expected (abbreviated):
low 0
high 0
max 35
oom 1
oom_kill 1
oom_group_kill 0
Memory cgroup out of memory: Killed process 60519 (python3) total-vm:219196kB,
anon-rss:204160kB, file-rss:5632kB, shmem-rss:0kB, UID:0 pgtables:464kB
Why? oom_kill counts kills in exactly this cgroup,
max counts how often the limit was hit. Later, cAdvisor
exports these counters to Prometheus (Day 8).
Save your progress? Sign up for free
Drill 5: cgroups v2 – CPU limit and throttling
sudo mkdir /sys/fs/cgroup/llm-cpu
echo "50000 100000" | sudo tee /sys/fs/cgroup/llm-cpu/cpu.max
sudo bash -c 'echo $$ > /sys/fs/cgroup/llm-cpu/cgroup.procs; \
exec stress-ng --cpu 2 --timeout 10s'
grep -E 'usage_usec|nr_periods|nr_throttled|throttled_usec' \
/sys/fs/cgroup/llm-cpu/cpu.statExpected (abbreviated, values vary):
50000 100000
stress-ng: info: [5200] successful run completed in 10.35 secs
usage_usec 5222961
nr_periods 104
nr_throttled 104
throttled_usec 14699286
Why? cpu.max = 50 ms quota per 100 ms period = 0.5 CPU.
10 s of runtime yield 5.2 s of CPU time; every period was throttled. On
a model server this shows up as slow tokens, not as errors – that is why
you monitor nr_throttled.
Save your progress? Sign up for free
Drill 6: Namespaces – and why containers are just processes
sudo unshare --pid --fork --mount-proc --uts --net \
bash -c 'hostname llm-box; hostname; ps -e -o pid,comm; ip -br link'Expected:
llm-box
PID COMMAND
1 bash
4 ps
lo DOWN 00:00:00:00:00:00 <LOOPBACK>
Its own hostname, PID 1, only a loopback with no connection to the outside. Now isolate a long-lived process and enter it from outside:
sudo unshare --pid --fork --mount-proc --uts --net \
bash -c 'hostname llm-box; exec sleep 300' &
sleep 1
sudo lsns -t net -t uts | tail -2
P=$(pgrep -n -x sleep)
sudo nsenter -t $P --uts --net bash -c 'hostname; ip -br link'
sudo readlink /proc/1/ns/net /proc/$P/ns/netExpected (inode numbers vary):
4026534205 uts 2 5218 root unshare --pid --fork --mount-proc ...
4026534267 net 2 5218 root unassigned unshare --pid --fork --mount-proc ...
llm-box
lo DOWN 00:00:00:00:00:00 <LOOPBACK>
net:[4026533204]
net:[4026534267]
Why? nsenter is the tool you use on Kubernetes nodes to
debug a pod’s network when the image contains no tools.
Now both combined – a container without Docker:
sudo mkdir /sys/fs/cgroup/mini-ctr
echo 100M | sudo tee /sys/fs/cgroup/mini-ctr/memory.max
echo 0 | sudo tee /sys/fs/cgroup/mini-ctr/memory.swap.max
cat > mini-ctr.sh <<'EOF'
#!/usr/bin/env bash
# already running in the cgroup mini-ctr – now with its own namespaces
hostname mini-ctr
echo "Hostname: $(hostname), meine PID: $$"
cat /proc/self/cgroup
ip -br link
exec python3 hog.py 20
EOF
sudo bash -c 'echo $$ > /sys/fs/cgroup/mini-ctr/cgroup.procs; \
exec unshare --pid --fork --mount-proc --uts --net bash ./mini-ctr.sh'Expected:
100M
0
Hostname: mini-ctr, meine PID: 1
0::/mini-ctr
lo DOWN 00:00:00:00:00:00 <LOOPBACK>
20 MB belegt
40 MB belegt
60 MB belegt
80 MB belegt
unshare: sigprocmask unblock failed: Invalid argument
Cross-check on the laptop – Docker writes the same files:
docker run --rm --memory 100m --cpus 0.5 ubuntu:24.04 \
cat /sys/fs/cgroup/memory.max /sys/fs/cgroup/cpu.maxExpected:
104857600
50000 100000
Save your progress? Sign up for free
Drill 7: systemd unit with resource limits
sudo mkdir -p /opt/fake-llm && sudo cp ~/tag01/hog.py /opt/fake-llm/
sudo tee /etc/systemd/system/fake-llm.service >/dev/null <<'EOF'
[Unit]
Description=Fake-LLM-Server (Labor Tag 1)
After=network-online.target
[Service]
ExecStart=/usr/bin/python3 -m http.server 11434 --bind 127.0.0.1
DynamicUser=yes
MemoryMax=300M
MemoryHigh=250M
CPUQuota=50%
TasksMax=64
LimitNOFILE=65536
Restart=on-failure
RestartSec=2s
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now fake-llm.service
systemctl status fake-llm --no-pager | head -9Expected (abbreviated):
● fake-llm.service - Fake-LLM-Server (Labor Tag 1)
Loaded: loaded (/etc/systemd/system/fake-llm.service; enabled; ...)
Active: active (running) since Fri 2026-10-09 17:00:29 UTC; 2s ago
Main PID: 132 (python3)
Tasks: 1 (limit: 64)
Memory: 3.6M (high: 250.0M max: 300.0M available: 246.3M peak: 3.9M)
CPU: 1.370s
CGroup: /system.slice/fake-llm.service
└─132 /usr/bin/python3 -m http.server 11434 --bind 127.0.0.1
The bridge to Drills 4 and 5:
CG=$(systemctl show -p ControlGroup --value fake-llm)
cat /sys/fs/cgroup$CG/memory.max /sys/fs/cgroup$CG/cpu.maxExpected:
314572800
50000 100000
MemoryHigh throttles (reclaim, slower),
MemoryMax kills. DynamicUser=yes starts the
service under an ephemeral UID without root privileges.
Now a service that blows its limit:
sudo tee /etc/systemd/system/llm-leak.service >/dev/null <<'EOF'
[Unit]
Description=Speicherleck-Simulation (Labor Tag 1)
[Service]
ExecStart=/usr/bin/python3 /opt/fake-llm/hog.py 50
DynamicUser=yes
MemoryMax=200M
MemorySwapMax=0
Restart=on-failure
RestartSec=5s
EOF
sudo systemctl daemon-reload
sudo systemctl start llm-leak; sleep 8
sudo journalctl -u llm-leak --no-pager -o cat | head -11
systemctl show llm-leak -p NRestarts
sudo systemctl stop llm-leakExpected:
Started llm-leak.service - Speicherleck-Simulation (Labor Tag 1).
50 MB belegt
100 MB belegt
150 MB belegt
llm-leak.service: A process of this unit has been killed by the OOM killer.
llm-leak.service: Main process exited, code=killed, status=9/KILL
llm-leak.service: Failed with result 'oom-kill'.
llm-leak.service: Consumed 1.506s CPU time.
llm-leak.service: Scheduled restart job, restart counter is at 1.
Started llm-leak.service - Speicherleck-Simulation (Labor Tag 1).
50 MB belegt
NRestarts=1
Why? Exactly this pattern – start, load, OOM kill, restart – is what
a model server shows when its model does not fit into the allowed RAM.
In Kubernetes it is called OOMKilled and
CrashLoopBackOff (Day 7).
Tip: Ad hoc without a unit file:
sudo systemd-run --unit=adhoc -p MemoryMax=200M python3 hog.py(not tested live, needs D-Bus).
Save your progress? Sign up for free
Drill 8: Load, PSI and top consumers
cat /proc/loadavg
for r in cpu memory io; do echo "$r: $(head -1 /proc/pressure/$r)"; done
stress-ng --cpu 8 --timeout 20s >/dev/null &
sleep 15
cat /proc/loadavg
head -1 /proc/pressure/cpu
vmstat 1 3Expected (abbreviated, sample values, may differ):
0.08 0.03 0.01 1/180 2210
cpu: some avg10=0.00 avg60=0.00 avg300=0.00 total=1998352
memory: some avg10=0.00 avg60=0.00 avg300=0.00 total=35116
io: some avg10=0.00 avg60=0.00 avg300=0.00 total=352737
6.38 2.59 1.11 9/190 2238
some avg10=46.78 avg60=16.83 avg300=4.66 total=2006413580
procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
r b swpd free buff cache si so bi bo in cs us sy id wa st gu
9 0 0 1327512 103800 2185652 0 0 136 735 4180 18 5 3 92 0 0 0
6 0 0 1327512 103808 2185712 0 0 0 72 7616 10237 98 2 0 0 0 0
Why PSI instead of load? A load of 6 on 4 CPUs only says “queue”.
cpu some avg10=46 means: in 46 % of the last 10 s at least
one task was waiting for CPU – directly usable as a latency signal. The
same files exist per cgroup (cpu.pressure,
memory.pressure).
Who is eating the CPU?
top -b -n 1 -o %CPU | head -12
sudo perf top -p $(pgrep -n stress-ng) --stdio 2>/dev/null | head -15Expected (abbreviated, may differ depending on the version;
perf not tested live):
top - 17:01:17 up 9:36, 0 user, load average: 6.38, 2.59, 1.11
Tasks: 25 total, 9 running, 16 sleeping, 0 stopped, 0 zombie
%Cpu(s): 95.8 us, 4.2 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2234 ubuntu 20 0 47380 5512 2428 R 90.9 0.1 0:07.19 stress-ng-cpu
Tip: On Ubuntu,
perfneeds the package matching the kernel:sudo apt-get install linux-tools-$(uname -r). For Python processes,py-spy top --pid <pid>gives more readable stacks.
Save your progress? Sign up for free
Checkpoint
cat > ~/tag01/check.sh <<'EOF'
#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "strace vorhanden" "command -v strace"
check "cgroup v2 aktiv" "[ -f /sys/fs/cgroup/cgroup.controllers ]"
check "OOM-Kill in llm-demo gezählt" \
"grep -Eq '^oom_kill [1-9]' /sys/fs/cgroup/llm-demo/memory.events"
check "llm-cpu wurde gedrosselt" \
"grep -Eq '^nr_throttled [1-9]' /sys/fs/cgroup/llm-cpu/cpu.stat"
check "fake-llm aktiv" "systemctl is-active --quiet fake-llm"
check "MemoryMax = 300M" \
"[ \$(systemctl show -p MemoryMax --value fake-llm) = 314572800 ]"
check "Port 11434 lauscht" "ss -tln | grep -q '127.0.0.1:11434 '"
check "OOM im Journal von llm-leak" \
"sudo journalctl -u llm-leak | grep -q oom-kill"
check "PSI verfügbar" "[ -r /proc/pressure/memory ]"
EOF
bash ~/tag01/check.shExpected:
✔ strace vorhanden
✔ cgroup v2 aktiv
✔ OOM-Kill in llm-demo gezählt
✔ llm-cpu wurde gedrosselt
✔ fake-llm aktiv
✔ MemoryMax = 300M
✔ Port 11434 lauscht
✔ OOM im Journal von llm-leak
✔ PSI verfügbar
Challenge
Change llm-leak.service so that the service is
throttled at 150 MB instead of being killed at 200 MB,
and prove it with memory.events and
memory.pressure.
Hint: A drop-in via systemctl edit
overrides individual values without touching the unit file.
Solution
sudo mkdir -p /etc/systemd/system/llm-leak.service.d
sudo tee /etc/systemd/system/llm-leak.service.d/throttle.conf >/dev/null <<'EOF'
[Service]
ExecStart=
ExecStart=/usr/bin/python3 /opt/fake-llm/hog.py 10
MemoryHigh=150M
MemoryMax=400M
EOF
sudo systemctl daemon-reload
sudo systemctl restart llm-leak; sleep 20
CG=/sys/fs/cgroup/system.slice/llm-leak.service
grep -E '^(high|oom_kill) ' $CG/memory.events
head -1 $CG/memory.pressure
sudo systemctl stop llm-leakExpected (values vary):
high 9832
oom_kill 0
some avg10=24.36 avg60=7.23 avg300=1.66 total=5068050
high rises, oom_kill stays at 0. Instead of
reaching 1000 MB after 20 s, the process gets stuck at around 150 MB:
without swap, the kernel cannot reclaim anonymous memory and slows down
every further allocation. ExecStart= (empty) clears the old
value before the new one is set.
Save your progress? Sign up for free
Interview Note
- “A model server keeps getting restarted – how do you proceed?” –
Check
journalctl -u/kubectl describeforoom-kill, readmemory.eventsandmemory.peak, then calculate model size plus KV cache against the limit (Day 5). - Containers are not VMs: namespaces for the view, cgroups for the limits, one shared kernel.
- CPU limits cause throttling that shows up as latency – PSI and
nr_throttledare the evidence. - Bridge: With OpenShift customers using Vault Agent sidecars,
OOMKilledcaused by limits set too tight was a classic – the diagnosis went exactly through these cgroup counters.
Enterprise Note
- Real model servers need RAM for the weights plus
the KV cache, which grows with context length and concurrency –
determine limits with a load test (Day 3,
llmctl load). - GPU memory is not part of cgroups: a VRAM OOM shows up as an error in the server log (CUDA out of memory), not as a kernel OOM.
- Set
LimitNOFILE,TasksMaxand kernel parameters (net.core.somaxconn) via configuration management (Ansible, Day 12). - On-prem at the customer: often cgroup v1 on older RHEL systems –
paths and file names differ (
memory.limit_in_bytes). - strace and perf need root and generate load; get approval first in regulated environments.