Skip to content
Hands-on IT IT Drills

Day 1: Linux internals

Goal: Fully inspect and constrain a model server process using /proc, signals, strace, cgroups v2, namespaces and systemd · Duration: approx. 120 min · Prerequisite: Day 0

Briefing

  • To Linux, an LLM server is just a process: PID, file descriptors, memory, signals.
  • cgroups v2 limit resources (memory.max, cpu.max) and count events (memory.events).
  • Namespaces isolate the view (PID, network, hostname, mounts). Container = namespaces + cgroups.
  • systemd writes your limits into cgroup files and logs OOM kills in the journal.
  • PSI (/proc/pressure/*) shows how long processes wait for CPU, RAM or I/O.
systemd ── fake-llm.service ── cgroup /system.slice/fake-llm.service
                                  memory.max = 300M  cpu.max = 50000 100000
                                  └── PID 132 python3 (ns: pid, net, mnt, uts)

All drills run on the VM (multipass shell llm-vm). A Python HTTP server on port 11434 stands in for Ollama – from Day 5 on you replace it with the real model server.

Drill 1: A process under the microscope

mkdir -p ~/tag01 && cd ~/tag01
python3 -m http.server 11434 >/dev/null 2>&1 &
PID=$!; sleep 2
grep -E '^(Name|State|PPid|Threads|VmRSS)' /proc/$PID/status
ls -l /proc/$PID/fd
ss -tlnp | grep 11434

Expected (PIDs and inodes vary):

Name:   python3
State:  S (sleeping)
PPid:   5019
VmRSS:     18520 kB
Threads:    1
lr-x------ 1 root root 64 Oct  9 16:50 0 -> /dev/null
l-wx------ 1 root root 64 Oct  9 16:50 1 -> /dev/null
l-wx------ 1 root root 64 Oct  9 16:50 2 -> /dev/null
lrwx------ 1 root root 64 Oct  9 16:50 3 -> socket:[812183]
LISTEN 0      5      0.0.0.0:11434   0.0.0.0:*   users:(("python3",pid=5025,fd=3))

Why? fd 3 is the listening socket. Send-Q 5 for LISTEN is the backlog length – Python’s default is tiny, real servers use 4096 and more.

Read limits and change them at runtime:

grep -E 'open files|processes' /proc/$PID/limits
sudo prlimit --pid $PID --nofile=4096:4096
grep 'open files' /proc/$PID/limits
tr '\0' ' ' < /proc/$PID/cmdline; echo
readlink /proc/$PID/exe

Expected (abbreviated, values may differ):

Max processes             31398                31398                processes
Max open files            1024                 524288               files
Max open files            4096                 4096                 files
python3 -m http.server 11434
/usr/bin/python3.12

Tip: Too many open files with many concurrent streams? First check /proc/<pid>/limits, then ls /proc/<pid>/fd | wc -l – the cause is almost always a soft limit that is too small.

Save your progress? Sign up for free

Drill 2: Signals

kill -STOP $PID
grep State /proc/$PID/status
for i in 1 2 3; do curl -s -m 1 localhost:11434 >/dev/null & done; sleep 0.5
ss -tln sport = :11434

Expected:

State:  T (stopped)
State  Recv-Q Send-Q Local Address:Port  Peer Address:PortProcess
LISTEN 3      5            0.0.0.0:11434      0.0.0.0:*

Why? The kernel handles the TCP handshake. The three connections sit in the accept queue (Recv-Q 3), but nobody picks them up. An open port does not mean a healthy service – that is why health checks verify a real response (Day 4: /readyz).

kill -CONT $PID; kill -TERM $PID; wait $PID; echo "exit=$?"

Expected:

exit=143

Exit code 128 + signal number: 143 = SIGTERM (15), 137 = SIGKILL (9). You will see these numbers again later as exitCode in Kubernetes.

A server that handles SIGTERM cleanly:

cat > server.sh <<'EOF'
#!/usr/bin/env bash
graceful() {
  echo "SIGTERM: laufende Requests beenden ..."
  sleep 1
  echo "sauber beendet"
  exit 0
}
trap graceful TERM
echo "Server läuft als PID $$"
while true; do sleep 1; done
EOF
chmod +x server.sh
./server.sh & P=$!; sleep 1.5; kill -TERM $P; wait $P; echo "exit=$?"
./server.sh & P=$!; sleep 1.5; kill -KILL $P; wait $P; echo "exit=$?"

Expected:

Server läuft als PID 5075
SIGTERM: laufende Requests beenden ...
sauber beendet
exit=0
Server läuft als PID 5080
bash: line 12:  5080 Killed                  ./server.sh
exit=137

Why? No process can catch SIGKILL. On SIGTERM, an LLM server needs time to finish running streams (drain) – otherwise every response breaks off mid-sentence.

Save your progress? Sign up for free

Drill 3: strace – what is the process really doing?

python3 -m http.server 11434 >/dev/null 2>&1 &
PID=$!; sleep 2
(sleep 1; curl -s localhost:11434/ >/dev/null) &
sudo timeout 3 strace -f -p $PID -e trace=accept4,recvfrom,sendto,openat
kill $PID

Expected (abbreviated):

strace: Process 5127 attached
accept4(3, {sa_family=AF_INET, sin_port=htons(60510), ...}, [16], SOCK_CLOEXEC) = 4
strace: Process 5137 attached
[pid  5137] recvfrom(4, "GET / HTTP/1.1\r\nHost: localhost:"..., 8192, ...) = 78
[pid  5137] openat(AT_FDCWD, "//", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 5
[pid  5137] sendto(4, "HTTP/1.0 200 OK\r\nServer: SimpleH"..., 156, 0, NULL, 0) = 156
[pid  5137] +++ exited with 0 +++

Why -f? The server starts a thread per request (Process 5137 attached). Without -f you only see the accept4.

Where is a client hanging? -T shows the duration of each syscall:

strace -T -e trace=connect curl -s -m 2 http://10.255.255.1:80

Expected (abbreviated):

connect(5, {sa_family=AF_INET, sin_port=htons(80), ...}, 16) = -1 EINPROGRESS
  (Operation now in progress) <0.000452>
+++ exited with 28 +++

A non-blocking connect, followed by a timeout (curl exit 28). Summary of all syscalls: strace -c -f <befehl>.

Warning: strace slows the process down massively. In production use it only briefly and selectively (-e trace=...), never permanently on a model server under load.

Save your progress? Sign up for free

Drill 4: cgroups v2 – memory limit and OOM kill

The memory hog simulates a model that needs more RAM than allowed while loading:

cat > hog.py <<'EOF'
import sys
import time

step = int(sys.argv[1]) if len(sys.argv) > 1 else 50
buf = []
while True:
    buf.append(bytearray(step * 1024 * 1024))
    print(f"{len(buf) * step} MB belegt", flush=True)
    time.sleep(0.2)
EOF

cat /sys/fs/cgroup/cgroup.subtree_control
sudo mkdir /sys/fs/cgroup/llm-demo
echo 200M | sudo tee /sys/fs/cgroup/llm-demo/memory.max
echo 0 | sudo tee /sys/fs/cgroup/llm-demo/memory.swap.max
sudo bash -c 'echo $$ > /sys/fs/cgroup/llm-demo/cgroup.procs; exec python3 hog.py 50'
echo "exit=$?"

Expected (abbreviated, controller list varies):

cpu memory pids
200M
0
50 MB belegt
100 MB belegt
150 MB belegt
Killed
exit=137

Why echo $$ > cgroup.procs? A process ends up in a cgroup when you write its PID into cgroup.procs; through exec, Python inherits it.

cat /sys/fs/cgroup/llm-demo/memory.events
sudo dmesg | grep -i 'memory cgroup out of memory' | tail -1

Expected (abbreviated):

low 0
high 0
max 35
oom 1
oom_kill 1
oom_group_kill 0
Memory cgroup out of memory: Killed process 60519 (python3) total-vm:219196kB,
anon-rss:204160kB, file-rss:5632kB, shmem-rss:0kB, UID:0 pgtables:464kB

Why? oom_kill counts kills in exactly this cgroup, max counts how often the limit was hit. Later, cAdvisor exports these counters to Prometheus (Day 8).

Save your progress? Sign up for free

Drill 5: cgroups v2 – CPU limit and throttling

sudo mkdir /sys/fs/cgroup/llm-cpu
echo "50000 100000" | sudo tee /sys/fs/cgroup/llm-cpu/cpu.max
sudo bash -c 'echo $$ > /sys/fs/cgroup/llm-cpu/cgroup.procs; \
  exec stress-ng --cpu 2 --timeout 10s'
grep -E 'usage_usec|nr_periods|nr_throttled|throttled_usec' \
  /sys/fs/cgroup/llm-cpu/cpu.stat

Expected (abbreviated, values vary):

50000 100000
stress-ng: info:  [5200] successful run completed in 10.35 secs
usage_usec 5222961
nr_periods 104
nr_throttled 104
throttled_usec 14699286

Why? cpu.max = 50 ms quota per 100 ms period = 0.5 CPU. 10 s of runtime yield 5.2 s of CPU time; every period was throttled. On a model server this shows up as slow tokens, not as errors – that is why you monitor nr_throttled.

Save your progress? Sign up for free

Drill 6: Namespaces – and why containers are just processes

sudo unshare --pid --fork --mount-proc --uts --net \
  bash -c 'hostname llm-box; hostname; ps -e -o pid,comm; ip -br link'

Expected:

llm-box
    PID COMMAND
      1 bash
      4 ps
lo               DOWN           00:00:00:00:00:00 <LOOPBACK>

Its own hostname, PID 1, only a loopback with no connection to the outside. Now isolate a long-lived process and enter it from outside:

sudo unshare --pid --fork --mount-proc --uts --net \
  bash -c 'hostname llm-box; exec sleep 300' &
sleep 1
sudo lsns -t net -t uts | tail -2
P=$(pgrep -n -x sleep)
sudo nsenter -t $P --uts --net bash -c 'hostname; ip -br link'
sudo readlink /proc/1/ns/net /proc/$P/ns/net

Expected (inode numbers vary):

4026534205 uts       2  5218 root            unshare --pid --fork --mount-proc ...
4026534267 net       2  5218 root unassigned unshare --pid --fork --mount-proc ...
llm-box
lo               DOWN           00:00:00:00:00:00 <LOOPBACK>
net:[4026533204]
net:[4026534267]

Why? nsenter is the tool you use on Kubernetes nodes to debug a pod’s network when the image contains no tools.

Now both combined – a container without Docker:

sudo mkdir /sys/fs/cgroup/mini-ctr
echo 100M | sudo tee /sys/fs/cgroup/mini-ctr/memory.max
echo 0 | sudo tee /sys/fs/cgroup/mini-ctr/memory.swap.max
cat > mini-ctr.sh <<'EOF'
#!/usr/bin/env bash
# already running in the cgroup mini-ctr – now with its own namespaces
hostname mini-ctr
echo "Hostname: $(hostname), meine PID: $$"
cat /proc/self/cgroup
ip -br link
exec python3 hog.py 20
EOF
sudo bash -c 'echo $$ > /sys/fs/cgroup/mini-ctr/cgroup.procs; \
  exec unshare --pid --fork --mount-proc --uts --net bash ./mini-ctr.sh'

Expected:

100M
0
Hostname: mini-ctr, meine PID: 1
0::/mini-ctr
lo               DOWN           00:00:00:00:00:00 <LOOPBACK>
20 MB belegt
40 MB belegt
60 MB belegt
80 MB belegt
unshare: sigprocmask unblock failed: Invalid argument

Cross-check on the laptop – Docker writes the same files:

docker run --rm --memory 100m --cpus 0.5 ubuntu:24.04 \
  cat /sys/fs/cgroup/memory.max /sys/fs/cgroup/cpu.max

Expected:

104857600
50000 100000

Save your progress? Sign up for free

Drill 7: systemd unit with resource limits

sudo mkdir -p /opt/fake-llm && sudo cp ~/tag01/hog.py /opt/fake-llm/
sudo tee /etc/systemd/system/fake-llm.service >/dev/null <<'EOF'
[Unit]
Description=Fake-LLM-Server (Labor Tag 1)
After=network-online.target

[Service]
ExecStart=/usr/bin/python3 -m http.server 11434 --bind 127.0.0.1
DynamicUser=yes
MemoryMax=300M
MemoryHigh=250M
CPUQuota=50%
TasksMax=64
LimitNOFILE=65536
Restart=on-failure
RestartSec=2s

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now fake-llm.service
systemctl status fake-llm --no-pager | head -9

Expected (abbreviated):

● fake-llm.service - Fake-LLM-Server (Labor Tag 1)
     Loaded: loaded (/etc/systemd/system/fake-llm.service; enabled; ...)
     Active: active (running) since Fri 2026-10-09 17:00:29 UTC; 2s ago
   Main PID: 132 (python3)
      Tasks: 1 (limit: 64)
     Memory: 3.6M (high: 250.0M max: 300.0M available: 246.3M peak: 3.9M)
        CPU: 1.370s
     CGroup: /system.slice/fake-llm.service
             └─132 /usr/bin/python3 -m http.server 11434 --bind 127.0.0.1

The bridge to Drills 4 and 5:

CG=$(systemctl show -p ControlGroup --value fake-llm)
cat /sys/fs/cgroup$CG/memory.max /sys/fs/cgroup$CG/cpu.max

Expected:

314572800
50000 100000

MemoryHigh throttles (reclaim, slower), MemoryMax kills. DynamicUser=yes starts the service under an ephemeral UID without root privileges.

Now a service that blows its limit:

sudo tee /etc/systemd/system/llm-leak.service >/dev/null <<'EOF'
[Unit]
Description=Speicherleck-Simulation (Labor Tag 1)

[Service]
ExecStart=/usr/bin/python3 /opt/fake-llm/hog.py 50
DynamicUser=yes
MemoryMax=200M
MemorySwapMax=0
Restart=on-failure
RestartSec=5s
EOF
sudo systemctl daemon-reload
sudo systemctl start llm-leak; sleep 8
sudo journalctl -u llm-leak --no-pager -o cat | head -11
systemctl show llm-leak -p NRestarts
sudo systemctl stop llm-leak

Expected:

Started llm-leak.service - Speicherleck-Simulation (Labor Tag 1).
50 MB belegt
100 MB belegt
150 MB belegt
llm-leak.service: A process of this unit has been killed by the OOM killer.
llm-leak.service: Main process exited, code=killed, status=9/KILL
llm-leak.service: Failed with result 'oom-kill'.
llm-leak.service: Consumed 1.506s CPU time.
llm-leak.service: Scheduled restart job, restart counter is at 1.
Started llm-leak.service - Speicherleck-Simulation (Labor Tag 1).
50 MB belegt
NRestarts=1

Why? Exactly this pattern – start, load, OOM kill, restart – is what a model server shows when its model does not fit into the allowed RAM. In Kubernetes it is called OOMKilled and CrashLoopBackOff (Day 7).

Tip: Ad hoc without a unit file: sudo systemd-run --unit=adhoc -p MemoryMax=200M python3 hog.py (not tested live, needs D-Bus).

Save your progress? Sign up for free

Drill 8: Load, PSI and top consumers

cat /proc/loadavg
for r in cpu memory io; do echo "$r: $(head -1 /proc/pressure/$r)"; done
stress-ng --cpu 8 --timeout 20s >/dev/null &
sleep 15
cat /proc/loadavg
head -1 /proc/pressure/cpu
vmstat 1 3

Expected (abbreviated, sample values, may differ):

0.08 0.03 0.01 1/180 2210
cpu: some avg10=0.00 avg60=0.00 avg300=0.00 total=1998352
memory: some avg10=0.00 avg60=0.00 avg300=0.00 total=35116
io: some avg10=0.00 avg60=0.00 avg300=0.00 total=352737
6.38 2.59 1.11 9/190 2238
some avg10=46.78 avg60=16.83 avg300=4.66 total=2006413580
procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st gu
 9  0      0 1327512 103800 2185652    0    0   136   735 4180   18  5  3 92  0  0  0
 6  0      0 1327512 103808 2185712    0    0     0    72 7616 10237 98  2  0  0  0  0

Why PSI instead of load? A load of 6 on 4 CPUs only says “queue”. cpu some avg10=46 means: in 46 % of the last 10 s at least one task was waiting for CPU – directly usable as a latency signal. The same files exist per cgroup (cpu.pressure, memory.pressure).

Who is eating the CPU?

top -b -n 1 -o %CPU | head -12
sudo perf top -p $(pgrep -n stress-ng) --stdio 2>/dev/null | head -15

Expected (abbreviated, may differ depending on the version; perf not tested live):

top - 17:01:17 up  9:36,  0 user,  load average: 6.38, 2.59, 1.11
Tasks:  25 total,   9 running,  16 sleeping,   0 stopped,   0 zombie
%Cpu(s): 95.8 us,  4.2 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   2234 ubuntu    20   0   47380   5512   2428 R  90.9   0.1   0:07.19 stress-ng-cpu

Tip: On Ubuntu, perf needs the package matching the kernel: sudo apt-get install linux-tools-$(uname -r). For Python processes, py-spy top --pid <pid> gives more readable stacks.

Save your progress? Sign up for free

Checkpoint

cat > ~/tag01/check.sh <<'EOF'
#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }

check "strace vorhanden" "command -v strace"
check "cgroup v2 aktiv" "[ -f /sys/fs/cgroup/cgroup.controllers ]"
check "OOM-Kill in llm-demo gezählt" \
  "grep -Eq '^oom_kill [1-9]' /sys/fs/cgroup/llm-demo/memory.events"
check "llm-cpu wurde gedrosselt" \
  "grep -Eq '^nr_throttled [1-9]' /sys/fs/cgroup/llm-cpu/cpu.stat"
check "fake-llm aktiv" "systemctl is-active --quiet fake-llm"
check "MemoryMax = 300M" \
  "[ \$(systemctl show -p MemoryMax --value fake-llm) = 314572800 ]"
check "Port 11434 lauscht" "ss -tln | grep -q '127.0.0.1:11434 '"
check "OOM im Journal von llm-leak" \
  "sudo journalctl -u llm-leak | grep -q oom-kill"
check "PSI verfügbar" "[ -r /proc/pressure/memory ]"
EOF
bash ~/tag01/check.sh

Expected:

✔ strace vorhanden
✔ cgroup v2 aktiv
✔ OOM-Kill in llm-demo gezählt
✔ llm-cpu wurde gedrosselt
✔ fake-llm aktiv
✔ MemoryMax = 300M
✔ Port 11434 lauscht
✔ OOM im Journal von llm-leak
✔ PSI verfügbar

Challenge

Change llm-leak.service so that the service is throttled at 150 MB instead of being killed at 200 MB, and prove it with memory.events and memory.pressure.

Hint: A drop-in via systemctl edit overrides individual values without touching the unit file.

Solution

sudo mkdir -p /etc/systemd/system/llm-leak.service.d
sudo tee /etc/systemd/system/llm-leak.service.d/throttle.conf >/dev/null <<'EOF'
[Service]
ExecStart=
ExecStart=/usr/bin/python3 /opt/fake-llm/hog.py 10
MemoryHigh=150M
MemoryMax=400M
EOF
sudo systemctl daemon-reload
sudo systemctl restart llm-leak; sleep 20
CG=/sys/fs/cgroup/system.slice/llm-leak.service
grep -E '^(high|oom_kill) ' $CG/memory.events
head -1 $CG/memory.pressure
sudo systemctl stop llm-leak

Expected (values vary):

high 9832
oom_kill 0
some avg10=24.36 avg60=7.23 avg300=1.66 total=5068050

high rises, oom_kill stays at 0. Instead of reaching 1000 MB after 20 s, the process gets stuck at around 150 MB: without swap, the kernel cannot reclaim anonymous memory and slows down every further allocation. ExecStart= (empty) clears the old value before the new one is set.

Save your progress? Sign up for free

Interview Note

  • “A model server keeps getting restarted – how do you proceed?” – Check journalctl -u/kubectl describe for oom-kill, read memory.events and memory.peak, then calculate model size plus KV cache against the limit (Day 5).
  • Containers are not VMs: namespaces for the view, cgroups for the limits, one shared kernel.
  • CPU limits cause throttling that shows up as latency – PSI and nr_throttled are the evidence.
  • Bridge: With OpenShift customers using Vault Agent sidecars, OOMKilled caused by limits set too tight was a classic – the diagnosis went exactly through these cgroup counters.

Enterprise Note

  • Real model servers need RAM for the weights plus the KV cache, which grows with context length and concurrency – determine limits with a load test (Day 3, llmctl load).
  • GPU memory is not part of cgroups: a VRAM OOM shows up as an error in the server log (CUDA out of memory), not as a kernel OOM.
  • Set LimitNOFILE, TasksMax and kernel parameters (net.core.somaxconn) via configuration management (Ansible, Day 12).
  • On-prem at the customer: often cgroup v1 on older RHEL systems – paths and file names differ (memory.limit_in_bytes).
  • strace and perf need root and generate load; get approval first in regulated environments.