Day 11: Incident Response and Post-Mortem
Goal: Trigger a real incident at the gateway, work through it by role, communicate with customers and write a blameless post-mortem with action items. · Duration: approx. 150 min · Prerequisite: Day 10
Briefing
- Mitigate first, understand later. Rollback, limit or failover come before the search for the cause.
- Four roles: the Incident Commander (IC) decides and coordinates but does not type. The Ops Lead works on the system. Comms writes the status page and customer updates. The Scribe keeps the timeline.
- Everything goes into one document with a UTC timestamp. Customer updates go out at fixed intervals, even when there is nothing new.
- Post-mortems are blameless: you ask about systems and processes, never about who is to blame.
Detect -> Triage/Declare -> Mitigate -> Resolve -> Post-Mortem -> Action Items
| | | | | |
Alert SEV + IC Rollback/Limit Monitoring blameless Owner + Due
Drill 1: Training ground – a fake model server with four slots
The fake server behaves like a GPU with 4 parallel generations. Each
response takes 0.5 s, so the capacity is about 8 requests per second.
Anyone who waits longer than 2 s for a slot gets
503 server busy.
mkdir -p ~/sre-lab/tag11 && cd ~/sre-lab/tag11
cat > fake_upstream.py <<'EOF'
"""Fake-Modellserver für Tag 11: OpenAI-kompatibel, begrenzte Slots wie eine GPU."""
import json
import os
import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
SLOTS = int(os.environ.get("SLOTS", "4")) # parallele Generierungen
DELAY = float(os.environ.get("DELAY", "0.5")) # Sekunden pro Antwort
QUEUE_WAIT = float(os.environ.get("QUEUE_WAIT", "2")) # max. Wartezeit auf Slot
sem = threading.BoundedSemaphore(SLOTS)
class Handler(BaseHTTPRequestHandler):
def send(self, code, obj):
body = json.dumps(obj).encode()
self.send_response(code)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def do_GET(self):
if self.path == "/v1/models":
return self.send(200, {"data": [{"id": "mistral-fake"}]})
self.send(404, {"error": "not found"})
def do_POST(self):
self.rfile.read(int(self.headers.get("Content-Length", 0)))
if not sem.acquire(timeout=QUEUE_WAIT):
return self.send(503, {"error": "server busy"})
try:
time.sleep(DELAY)
finally:
sem.release()
self.send(200, {
"choices": [{"message": {"role": "assistant", "content": "ok"}}],
"usage": {"prompt_tokens": 12, "completion_tokens": 30},
})
def log_message(self, *args):
pass
ThreadingHTTPServer(("0.0.0.0", 11434), Handler).serve_forever()
EOF
docker network create tag11
docker run -d --name fake-llm --network tag11 \
-v "$PWD/fake_upstream.py:/app.py:ro" python:3.13-alpine python -u /app.pyBefore every restart, the start script saves the logs of the old container. Without this backup, the evidence would be gone after the rollback.
cat > gw-start.sh <<'EOF'
#!/usr/bin/env bash
# Startet das Gateway neu (Lab-Keys!). Sichert vorher die Logs des alten Containers.
if docker inspect gw >/dev/null 2>&1; then
docker logs gw > "gw-$(date -u +%H%M%S).log" 2>&1
docker rm -f gw >/dev/null # harter Neustart, schnell
fi
docker run -d --name gw --network tag11 -p 8080:8080 \
-e UPSTREAM_URL=http://fake-llm:11434 \
-e TENANT_KEYS="acme:lab-key-acme,globex:lab-key-globex" \
-e RATE_LIMIT_RPS="${RPS:-5}" -e RATE_LIMIT_BURST="${BURST:-10}" \
llm-gateway:dev >/dev/null
sleep 1
echo "gw: rps=${RPS:-5} burst=${BURST:-10} ready=$(curl -s localhost:8080/readyz)"
EOF
cat > load.sh <<'EOF'
#!/usr/bin/env bash
# load.sh <tenant> <anzahl> <parallel> -> Statuscodes und p50/p95 in Sekunden
T=$1; N=${2:-20}; P=${3:-2}
seq "$N" | xargs -P "$P" -I{} curl -s -o /dev/null -m 10 \
-w '%{http_code} %{time_total}\n' \
-H "Authorization: Bearer lab-key-$T" -H 'Content-Type: application/json' \
-d '{"model":"mistral-fake","messages":[{"role":"user","content":"hi"}]}' \
localhost:8080/v1/chat/completions > "load-$T.txt"
echo "== $T: $(awk '{print $1}' "load-$T.txt" | sort | uniq -c | xargs)"
sort -n -k2 "load-$T.txt" | LC_ALL=C awk '{a[NR]=$2} END {
printf "p50=%.2fs p95=%.2fs\n", a[int(NR*0.5)], a[int(NR*0.95)] }'
EOF
chmod +x gw-start.sh load.sh
./gw-start.sh
./load.sh acme 20 2Expected:
gw: rps=5 burst=10 ready=ok
== acme: 20 200
p50=0.51s p95=0.51s
Why? This is your baseline. Without “normal” you cannot recognize “broken”. acme sends 2 parallel requests, which is about 4 per second and thus less than the limit of 5.
Tip: With
llmctl loadfrom Day 3 you get the same numbers plus p99.load.shis enough here and only needs curl.
Save your progress? Sign up for free
Drill 2: Roles, severity and the timeline tool
Set the severity levels before the pager rings. When in doubt, rate higher and downgrade later.
| SEV | Criterion | Example | Response |
|---|---|---|---|
| SEV1 | All tenants down, data leak, security incident | API returns only 5xx | Immediate page, IC mandatory, status page ≤ 15 min, updates 30 min |
| SEV2 | One tenant heavily affected or SLO burn critical | acme 50 % errors | Page, IC mandatory, status page ≤ 30 min, updates 60 min |
| SEV3 | Noticeably slower, SLO still holds | p95 twice as high | Ticket with priority, fix within the working day |
| SEV4 | No customer impact | Dashboard panel empty | Ticket, backlog |
The scribe needs a tool that writes an entry in 2 seconds.
tl appends a table row with the UTC time to
incident.md.
cat > tl <<'EOF'
#!/usr/bin/env bash
# tl "Text" -> hängt eine Zeile mit UTC-Zeit und Autor an die Timeline an
f=${INCIDENT_FILE:-incident.md}
[ $# -gt 0 ] || { echo "usage: tl <text>" >&2; exit 1; }
msg=${*//|/\\|} # Pipe würde die Tabelle zerbrechen
printf '| %s | %s | %s |\n' "$(date -u +%H:%M:%S)" "${WHO:-$USER}" "$msg" >> "$f"
tail -1 "$f"
EOF
chmod +x tl
cat > incident.md <<'EOF'
# INC-2026-1009-01: acme bekommt 503 vom Gateway
- Severity: SEV2 · Status: investigating
- IC: thomas · Ops Lead: thomas · Comms: thomas · Scribe: tl
## Timeline (UTC)
| Zeit | Wer | Ereignis |
|---|---|---|
EOF
export WHO=thomas PATH="$PWD:$PATH"
tl "Timeline angelegt"Expected (time varies):
| 16:24:30 | thomas | Timeline angelegt |
Why? A timeline that you only write from memory after the incident is wrong. In the lab you play all four roles. In real life you always separate IC and Ops Lead as soon as two people are there.
Save your progress? Sign up for free
Drill 3: Triggering, detecting and classifying the disruption
Change CHG-4711 raises the rate limit “so that globex’s batch job runs through faster”. Afterwards globex starts the batch with 40 parallel requests.
RPS=50 BURST=100 ./gw-start.sh
tl "CHG-4711 live: RPS 5->50, BURST 10->100 (globex-Batch)"
./load.sh globex 600 40 > globex.out 2>&1 & # Batch läuft ca. 60 s
sleep 3
./load.sh acme 20 2 # Probe aus Sicht von acmeExpected (the error rate fluctuates between 40 and 80 % depending on timing):
gw: rps=50 burst=100 ready=ok
| 16:24:49 | thomas | CHG-4711 live: RPS 5->50, BURST 10->100 (globex-Batch) |
== acme: 10 200 10 503
p50=2.02s p95=2.50s
acme has not changed anything and still gets 50 % errors, and latency has quadrupled. Now narrow it down: which tenant, which code, what has changed?
cat > impact.sh <<'EOF'
#!/usr/bin/env bash
# impact.sh [metrics-datei] -> Anfragen, 5xx, 429 und Fehlerquote pro Mandant
if [ -n "$1" ]; then cat "$1"; else curl -s localhost:8080/metrics; fi |
LC_ALL=C awk -F'[{}" ]+' '/^gateway_requests_total\{/ {
code=$3; t=$5; n=$NF; all[t]+=n
if (code ~ /^5/) err[t]+=n; if (code == 429) rl[t]+=n }
END { for (t in all) printf "%-7s total=%-4d 5xx=%-4d 429=%-4d err=%.1f%%\n",
t, all[t], err[t], rl[t], 100*err[t]/all[t] }'
EOF
chmod +x impact.sh
./impact.sh
docker logs gw 2>&1 | jq -r 'select(.status>=500) | "\(.tenant) \(.status)"' \
| sort | uniq -c
docker inspect gw --format '{{range .Config.Env}}{{println .}}{{end}}' | grep RATE
tl "acme ~50% 503, p50 2s (Baseline 0.5s), globex auch 503. SEV2, IC thomas"Expected (shortened, numbers fluctuate):
globex total=444 5xx=254 429=0 err=57.2%
acme total=20 5xx=10 429=0 err=50.0%
10 acme 503
254 globex 503
RATE_LIMIT_RPS=50
RATE_LIMIT_BURST=100
Why? The 503s come from the model server (“server busy”) and are
passed through. Occasional 502s for globex are possible: then the
connection backlog of the fake server overflows. No tenant got a 429, so
the gateway let everything through. grep RATE instead of
the whole env list is deliberate: TENANT_KEYS does not
belong in a chat or a ticket.
Save your progress? Sign up for free
Drill 4: Mitigation first – save evidence, roll back, verify
The IC decides: roll back CHG-4711. The cause is not proven yet, but the rollback is cheap and reversible. Before that, you save the metrics in one second.
curl -s localhost:8080/metrics > metrics-incident.txt
tl "IC: Rollback CHG-4711 (RPS 50->5). Metriken gesichert"
./gw-start.sh # Defaults: RPS=5, BURST=10
tl "Rollback fertig"
./load.sh acme 20 2
tl "Verifikation: acme 20/20 200, p50 0.5s. Status: monitoring"Expected:
gw: rps=5 burst=10 ready=ok
== acme: 20 200
p50=0.58s p95=1.83s
The p95 is still elevated while the batch is running: globex may now send 5 req/s, acme sends 4 req/s. Together that is 9 req/s, which is already more than the 8 req/s capacity. Remember this for the post-mortem.
wait; cat globex.out # Batch-Ergebnis von globex
ls gw-*.log metrics-incident.txtExpected (shortened):
== globex: 144 000 198 200 8 502 250 503
p50=2.07s p95=3.17s
gw-163218.log gw-163253.log metrics-incident.txt
Why? The 000s are dropped connections during the
restart. A rollback by container restart has a gap. In Kubernetes, a
rolling update (Day 7) does this without that gap. After the rollback,
globex gets 429 instead of 503. The gateway protects the shared capacity
again.
Warning: Only once the mitigation works do you start the root cause analysis. The most common mistake in an incident: 40 minutes of debugging while the damage continues.
Save your progress? Sign up for free
Drill 5: Customer communication – status page and updates
Customers are international, so you write in English. Prepare the templates before the incident. In an emergency, Comms only fills in the gaps.
cat > comms.md <<'EOF'
## Initial (within 30 min of declaring SEV2)
[Investigating] Elevated error rates on the Chat Completions API
We are seeing elevated error rates (HTTP 503) and increased latency for some
customers using the Chat Completions API since 14:05 UTC. Our engineers are
investigating. Next update by 15:00 UTC or earlier.
## Update
[Identified] Elevated error rates on the Chat Completions API
We identified a configuration change that allowed excessive load on a shared
model server. We rolled back the change at 14:29 UTC. Error rates are back to
normal and we are monitoring. Next update by 15:30 UTC.
## Resolved
[Resolved] Elevated error rates on the Chat Completions API
Between 14:05 and 14:29 UTC, about 50% of Chat Completions requests for affected
customers failed with HTTP 503, and median latency rose from 0.5 s to 2.0 s.
The cause was a configuration change, which we rolled back. No data was lost or
exposed. Requests that failed can be retried safely. A post-incident report
will follow within 5 business days.
EOF
grep -c '^\[' comms.md
tl "Comms: Initial-Update auf Status-Page veröffentlicht"Expected:
3
Rules for customer updates:
- What the customer notices (symptom, since when, which API), not what is broken internally.
- Always state a time for the next update and keep to it.
- No speculation about causes, no names of other customers (“a shared model server”, not “globex”).
- In the resolved update: time period, extent in numbers, data loss yes/no, retry safe yes/no.
Save your progress? Sign up for free
Drill 6: The blameless post-mortem
A post-mortem answers: What happened? How bad was it? Why could it happen? What do we change? The numbers below come from your simulation. The times are stretched to a real production duration (24 minutes).
cat > postmortem.md <<'EOF'
# Post-Mortem INC-2026-1009-01: 503 errors on Chat Completions
- Status: final · Severity: SEV2 · Date: 2026-10-08
- Authors: Thomas Zachmann · Reviewers: platform team
- Blameless: this document describes systems and processes, not people.
## Summary
A rate-limit change (CHG-4711) let one tenant's batch job send more traffic than the
shared model server can serve. The model server rejected the excess with HTTP 503,
which hit all tenants on that server. A rollback ended the impact after 24 minutes.
## Impact
- Duration: 14:05 to 14:29 UTC (24 min).
- Tenant acme: 50.0% of requests failed (HTTP 503), about 1,870 of 3,740.
- Tenant acme: median latency 0.51 s -> 2.02 s, p95 2.50 s.
- Tenant globex: 57.2% of batch requests failed (HTTP 503).
- Error budget acme (99.5% / 28 days): 5.5% used in this incident.
- No data loss, no data exposure between tenants.
## Detection
The burn-rate alert LLMGatewayHighErrorRate (Day 10) fired at 14:11 UTC, 6 minutes
after impact started. No customer reported the issue before the status page update.
EOFcat >> postmortem.md <<'EOF'
## Timeline (UTC)
| Time | Event |
|---|---|
| 14:02 | CHG-4711 deployed: RATE_LIMIT_RPS 5 -> 50, BURST 10 -> 100 |
| 14:05 | globex batch job starts with 40 parallel requests. Impact starts |
| 14:11 | LLMGatewayHighErrorRate fires, on-call acknowledges |
| 14:14 | SEV2 declared, IC assigned, incident channel opened |
| 14:19 | Initial status page update |
| 14:24 | Correlation with CHG-4711 found via container config |
| 14:27 | IC decides rollback of CHG-4711 |
| 14:29 | Rollback complete, acme error rate 0%. Impact ends |
| 14:49 | Status page update: identified, monitoring |
| 15:05 | Resolved after 30 min clean monitoring |
## Root cause and contributing factors
Root cause: the gateway's only admission control is a per-tenant rate limit. Nothing
limits the total traffic to the shared model server (4 slots, about 8 req/s).
CHG-4711 allowed one tenant 50 req/s, six times the total capacity.
Contributing factors:
- The change process has no capacity check for limit changes.
- RATE_LIMIT_RPS applies to all tenants. The change meant for globex raised every
tenant's limit.
- The gateway passes upstream 503 through without Retry-After.
- Deployments are not marked in Grafana. Correlating the change took 10 minutes.
- The lab-style rollback (restart) dropped 152 in-flight globex requests.
EOFcat >> postmortem.md <<'EOF'
## What went well
- The burn-rate alert detected the impact within 6 minutes.
- The rollback was a one-line config change and took 2 minutes.
- Logs and metrics were saved before the rollback.
## What went badly
- Nobody load-tested the new limit against the model server capacity.
- The first status page update came 14 minutes after the alert.
## Where we got lucky
- The batch job ran during low traffic. At peak, all tenants would have hit 503.
## Action items
| ID | Action | Owner / Prio | Due / Status |
|---|---|---|---|
| AI-1 | Gateway: global in-flight limit | thomas / P1 | 2026-10-22 / open |
| AI-2 | Per-tenant rate limits | thomas / P1 | 2026-10-22 / open |
| AI-3 | CI check: sum of limits <= capacity | platform / P2 | 2026-11-05 / open |
| AI-4 | Deploy annotations in Grafana (Day 8) | obs / P2 | 2026-10-15 / done |
| AI-5 | Status page template prefilled from alert | sre / P3 | 2026-11-30 / open |
EOF
wc -l postmortem.mdExpected:
70 postmortem.md
Why? “Root cause” plus “contributing factors” is more honest than a single cause. The “Where we got lucky” section shows risks that did not materialize this time.
Save your progress? Sign up for free
Drill 7: 5 Whys versus contributing factors
5 Whys forces a single chain. That is good for getting started, but dangerous:
Why 503 for acme? Model server full.
Why full? globex sent 40 parallel requests.
Why did that get through? Rate limit was 50 instead of 5.
Why 50? CHG-4711.
Why did CHG-4711 get through? No capacity check in the change process.
The chain ends at the change process. Had you stopped at “Why 50?”
with “Person X made a mistake”, the result would be blame instead of
learning. Equally dangerous is what the chain leaves out: the global
RATE_LIMIT_RPS, the missing 503 Retry-After and the missing
deploy markers do not appear in a single chain. That is why in the
post-mortem you use the list of contributing factors. Every factor has
an action item or a conscious decision “accepted because …”.
Check your post-mortem for blame language and for missing numbers:
grep -niwE 'should have|careless|mistake|blame|human error' postmortem.md \
&& echo "✘ Schuld-Sprache gefunden" || echo "✔ keine Schuld-Sprache"
grep -cE '[0-9]+(\.[0-9]+)?( ?%| s| min| req/s)' postmortem.mdExpected:
✔ keine Schuld-Sprache
15
Why? “Human error” is never a root cause but a symptom: the system allowed the mistake. The number shows how many lines contain measured values. A post-mortem without numbers is an opinion.
Save your progress? Sign up for free
Drill 8: Tracking action items and the review meeting
Action items without an owner and a date never get done. This script lists open and overdue items, for example for the weekly ops review.
cat > ai-status.sh <<'EOF'
#!/usr/bin/env bash
# ai-status.sh <postmortem.md> [stichtag YYYY-MM-DD] -> offene/überfällige Items
today=${2:-$(date -u +%F)}
LC_ALL=C awk -F' *\\| *' -v today="$today" '$2 ~ /^AI-[0-9]+$/ {
split($5, d, " / "); due=d[1]; st=d[2]
if (st == "done") { done++; next }
open++; flag = (due < today) ? "OVERDUE" : "open"
printf "%-7s %-5s %-11s %s\n", flag, $2, due, $3 }
END { printf "open=%d done=%d\n", open, done }' "$1"
EOF
chmod +x ai-status.sh
./ai-status.sh postmortem.md
./ai-status.sh postmortem.md 2026-10-30Expected:
open AI-1 2026-10-22 Gateway: global in-flight limit
open AI-2 2026-10-22 Per-tenant rate limits
open AI-3 2026-11-05 CI check: sum of limits <= capacity
open AI-5 2026-11-30 Status page template prefilled from alert
open=4 done=1
OVERDUE AI-1 2026-10-22 Gateway: global in-flight limit
OVERDUE AI-2 2026-10-22 Per-tenant rate limits
open AI-3 2026-11-05 CI check: sum of limits <= capacity
open AI-5 2026-11-30 Status page template prefilled from alert
open=4 done=1
In real life, action items live in the ticket system (Jira, GitLab Issues), and the post-mortem links to them. The table is only the index.
Review meeting (45 min, within 5 business days):
- 5 min: the author reads out the summary and impact. Everyone has read the document beforehand.
- 15 min: walk through the timeline. Where did we lose time (here: 14 min until the status page, 10 min until the correlation)?
- 15 min: contributing factors and action items. Every item gets an owner, a priority and a date.
- 10 min: What can other teams learn from this? Afterwards, share the post-mortem widely.
- Moderated by someone who was not involved in the incident.
Writing rules for runbooks and post-mortems:
- Short sentences in the active voice: “Roll back CHG-4711”, not “It should be rolled back”.
- Numbers instead of adjectives: “50 % 503 for 24 min”, not “lots of errors”.
- Times always in UTC, commands ready to copy, and the expected output belongs with them.
- Summary first. Anyone who reads only 3 sentences must still know what happened.
- A runbook step is one action plus one check command.
Save your progress? Sign up for free
Checkpoint
cd ~/sre-lab/tag11
[ "$(grep -c '^| [0-9][0-9]:' incident.md)" -ge 5 ] \
&& echo "✔ Timeline mit ≥ 5 Einträgen" || echo "✘ Timeline zu kurz"
for s in Summary Impact Timeline 'Root cause' 'Action items'; do
grep -q "^## $s" postmortem.md && echo "✔ PM: $s" || echo "✘ PM: $s fehlt"
done
awk -F' *\\| *' '$2 ~ /^AI-/ && ($4 !~ / \/ P[1-3]$/ || $5 !~ /^20[0-9-]+ \//)' \
postmortem.md | grep -q . && echo "✘ Action Item ohne Owner/Due" \
|| echo "✔ alle Action Items mit Owner, Prio, Due"
./load.sh acme 20 2 | grep -q '== acme: 20 200' \
&& echo "✔ acme wieder gesund" || echo "✘ acme hat noch Fehler"Expected:
✔ Timeline mit ≥ 5 Einträgen
✔ PM: Summary
✔ PM: Impact
✔ PM: Timeline
✔ PM: Root cause
✔ PM: Action items
✔ alle Action Items mit Owner, Prio, Due
✔ acme wieder gesund
Challenge
Stop the model server (docker stop fake-llm). Work
through the incident in a second file incident-2.md:
measure the symptom, set the SEV, write the initial update, mitigate,
verify. At the end the timeline has at least 5 entries.
Hint: INCIDENT_FILE points
tl to another file, and /readyz tells you
whether the gateway can still reach the upstream.
Solution
cd ~/sre-lab/tag11
export INCIDENT_FILE=incident-2.md
printf '# INC-2026-1009-02: Modellserver down\n\n| Zeit | Wer | Ereignis |\n' \
> incident-2.md
printf '|---|---|---|\n' >> incident-2.md
docker stop fake-llm
./load.sh acme 10 2; curl -s localhost:8080/readyz
tl "acme 100% 502, readyz: upstream not ready. SEV1 (alle Mandanten), IC thomas"
tl "Comms: Initial-Update 'Chat Completions API unavailable' veröffentlicht"
docker start fake-llm
until curl -sf localhost:8080/readyz; do sleep 1; done
tl "Mitigation: fake-llm neu gestartet, readyz ok"
./load.sh acme 10 2
tl "Verifikation: acme 10/10 200. Status: monitoring"
tl "Resolved (im Lab ohne die üblichen 30 min Monitoring)"
grep -c '^| [0-9]' incident-2.mdExpected:
== acme: 10 502
p50=0.08s p95=0.52s
upstream not ready
...
ok
== acme: 10 200
p50=0.54s p95=0.55s
5
Why 502 instead of 503? The server is gone, the connection fails, and
the ErrorHandler in the gateway responds with 502. SEV1,
because every tenant is affected.
Clean up:
docker rm -f gw fake-llm && docker network rm tag11Save your progress? Sign up for free
Interview Note
- Typical question: “Walk me through an incident you led.” Answer in the order detect, mitigate, communicate, learn, with numbers (duration, error rate, customers).
- A crisp answer on priorities: “Mitigation before root cause. I’ll roll back a suspicious change even before I can prove it caused the issue, if rollback is cheap.”
- Explain blameless like this: “If one person’s mistake can cause an outage, the system allowed it. The action item fixes the system.”
- Bridge from your own practice: a Vault/OpenBao outage (seal, Raft quorum, expired tokens) that affected many teams at once, and how you kept communication and the fix separate.
- Show your writing strength: offer to bring an anonymized post-mortem or runbook as a work sample.
Enterprise Note
- On-prem at the customer you often have no access to the systems. The runbook must be written so that the customer runs the diagnostic commands themselves and sends you the output (support bundle, Day 14).
- Contracts define response times per severity. Your SEV matrix must match the SLA clauses; otherwise you argue about definitions after the incident.
- Security incidents (a data leak between tenants) come with reporting deadlines, e.g. 72 h under GDPR Art. 33 and NIS2. Legal and the DPO then belong in the channel immediately.
- Post-mortems for customers are a separate, shortened version: without internal names, without names of other customers, with clear commitments.
- Tools: incident.io, PagerDuty, Grafana IRM or FireHydrant create the channel, roles and timeline automatically. The principles from this chapter stay the same.