Skip to content
Hands-on IT IT Drills

Foreword

Read little, type a lot. This book prepares you for a role as an Applied AI Engineer / SRE at an LLM provider: Linux and networking from the bottom up, your own tools in Python and Go, Mistral models on a VM and in Kubernetes, observability, on-call, security – and finally operations on-prem at the customer site.

The training rules

  1. Type, don’t copy. Run every command yourself once.
  2. Compare with Expected. After every drill you will find what you should see. If it differs, that is where the real lesson is.
  3. Pass the checkpoint. Only move on when the check script shows nothing but ✔.
  4. Solve the challenge on your own. First the hint, then the solution.
  5. Explain out loud. Present the Interview Note freely once in two minutes, as if an interviewer were sitting across from you.
  6. Lab values stay in the lab. lab-key-acme, lab-key-globex and all passwords are play money. Real Mistral keys only via MISTRAL_API_KEY, never in code.

Structure of every training day

Section Purpose
Briefing The bare essentials in a few bullet points
Drills Commands with expected output
Checkpoint Check script: did everything work?
Challenge A task without instructions
Interview Note How you explain the topic in an interview
Enterprise Note What is different in production and at the customer

The training plan

Week Day Unit
1 – Foundations 0 Build the training ground
1 Linux internals: processes, /proc, cgroups v2, namespaces, systemd
2 Network debugging: ss, dig, curl timings, tcpdump, TLS, MTU
3 Python for SRE: build and test llmctl
4 Go for SRE: build, test and ship llm-gateway
2 – LLMs & Kubernetes 5 Mistral models on a Linux VM
6 Multi-tenant Kubernetes: RBAC, quotas, NetworkPolicy, PSA
7 Model serving on Kubernetes
3 – Observability & operations 8 Observability I: Prometheus, PromQL, Grafana
9 Observability II: Loki, OpenTelemetry, Tempo
10 SLOs, alerting and on-call
11 Incident response and post-mortem
4 – Automation, security, customer 12 Infrastructure as Code: OpenTofu and Ansible
13 Security: SDLC, SBOM, signatures, CVE response
14 Hybrid and on-prem at the customer
15 Competition: disaster drills
Appendix A Cheat Sheet
B Final Exam
C Interview preparation

The training ground

 Laptop (macOS/Linux) ~/sre-lab
 ┌───────────────────────────────────────────────────────────────────┐
 │ docker · kind · kubectl · helm · go · python · tofu · ansible     │
 │                                                                   │
 │  kind cluster "sre-lab" (1 control-plane + 2 worker)              │
 │  ┌──────────────────────────────────────────────────────────┐     │
 │  │ ns llm-system:   model server · llm-gateway              │     │
 │  │ ns monitoring:   Prometheus · Grafana · Loki · Tempo     │     │
 │  │ ns tenant-acme:  client apps  (lab-key-acme)             │     │
 │  │ ns tenant-globex: client apps (lab-key-globex)           │     │
 │  └──────────────────────────────────────────────────────────┘     │
 │                     │ HTTP /v1/chat/completions                   │
 │                     ▼                                             │
 │  Linux VM "llm-vm" (multipass, Ubuntu 24.04)                      │
 │  ┌──────────────────────────────────────────────────────────┐     │
 │  │ Ollama :11434 (CPU, Mistral model)                       │     │
 │  │ vLLM   :8000  (GPU only, e.g. cloud VM)                  │     │
 │  │ systemd · cgroups · journald                             │     │
 │  └──────────────────────────────────────────────────────────┘     │
 └───────────────────────────────────────────────────────────────────┘

Job posting → training day

Requirement Days
Linux internals, network debugging, distributed systems 1, 2, 15
Python and Go: build, test and ship tools 3, 4
LLM/model serving 5, 7
Multi-tenant Kubernetes: namespaces, NetworkPolicy, RBAC 6, 7
Observability: Prometheus, Grafana, OTel, Loki, Tempo 8, 9
On-call, incident response, post-mortems, runbooks 10, 11, 15
IaC: Terraform/OpenTofu and Ansible 12
Security mindset: secure SDLC, CVE, supply chain 13
Hybrid/on-prem at the customer, customer communication 11, 14
Interviews and written communication every Interview Note, Appendix C

Tip: Tested with go 1.25 and 1.26, python 3.14, Ollama 0.35, kind 0.33, kubectl 1.37, helm 4.3, tofu 1.12 and Ubuntu 24.04. If a flag differs: <tool> --help and the official documentation.

The companion code

All files from this book are publicly available on GitHub: https://github.com/thomaszachmann/llm-plattformen-zum-anfassen. Day 0 shows you how to get them.

Let’s go.

Thomas Zachmann, October 2026