Skip to content
Hands-on IT IT Drills

Day 11: Incident Response and Post-Mortem

Goal: Trigger a real incident at the gateway, work through it by role, communicate with customers and write a blameless post-mortem with action items. · Duration: approx. 150 min · Prerequisite: Day 10

Briefing

  • Mitigate first, understand later. Rollback, limit or failover come before the search for the cause.
  • Four roles: the Incident Commander (IC) decides and coordinates but does not type. The Ops Lead works on the system. Comms writes the status page and customer updates. The Scribe keeps the timeline.
  • Everything goes into one document with a UTC timestamp. Customer updates go out at fixed intervals, even when there is nothing new.
  • Post-mortems are blameless: you ask about systems and processes, never about who is to blame.
Detect -> Triage/Declare -> Mitigate -> Resolve -> Post-Mortem -> Action Items
  |           |               |           |            |              |
 Alert     SEV + IC      Rollback/Limit  Monitoring   blameless    Owner + Due

Drill 1: Training ground – a fake model server with four slots

The fake server behaves like a GPU with 4 parallel generations. Each response takes 0.5 s, so the capacity is about 8 requests per second. Anyone who waits longer than 2 s for a slot gets 503 server busy.

mkdir -p ~/sre-lab/tag11 && cd ~/sre-lab/tag11
cat > fake_upstream.py <<'EOF'
"""Fake-Modellserver für Tag 11: OpenAI-kompatibel, begrenzte Slots wie eine GPU."""
import json
import os
import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

SLOTS = int(os.environ.get("SLOTS", "4"))           # parallele Generierungen
DELAY = float(os.environ.get("DELAY", "0.5"))       # Sekunden pro Antwort
QUEUE_WAIT = float(os.environ.get("QUEUE_WAIT", "2"))  # max. Wartezeit auf Slot
sem = threading.BoundedSemaphore(SLOTS)


class Handler(BaseHTTPRequestHandler):
    def send(self, code, obj):
        body = json.dumps(obj).encode()
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def do_GET(self):
        if self.path == "/v1/models":
            return self.send(200, {"data": [{"id": "mistral-fake"}]})
        self.send(404, {"error": "not found"})

    def do_POST(self):
        self.rfile.read(int(self.headers.get("Content-Length", 0)))
        if not sem.acquire(timeout=QUEUE_WAIT):
            return self.send(503, {"error": "server busy"})
        try:
            time.sleep(DELAY)
        finally:
            sem.release()
        self.send(200, {
            "choices": [{"message": {"role": "assistant", "content": "ok"}}],
            "usage": {"prompt_tokens": 12, "completion_tokens": 30},
        })

    def log_message(self, *args):
        pass


ThreadingHTTPServer(("0.0.0.0", 11434), Handler).serve_forever()
EOF
docker network create tag11
docker run -d --name fake-llm --network tag11 \
  -v "$PWD/fake_upstream.py:/app.py:ro" python:3.13-alpine python -u /app.py

Before every restart, the start script saves the logs of the old container. Without this backup, the evidence would be gone after the rollback.

cat > gw-start.sh <<'EOF'
#!/usr/bin/env bash
# Startet das Gateway neu (Lab-Keys!). Sichert vorher die Logs des alten Containers.
if docker inspect gw >/dev/null 2>&1; then
  docker logs gw > "gw-$(date -u +%H%M%S).log" 2>&1
  docker rm -f gw >/dev/null                   # harter Neustart, schnell
fi
docker run -d --name gw --network tag11 -p 8080:8080 \
  -e UPSTREAM_URL=http://fake-llm:11434 \
  -e TENANT_KEYS="acme:lab-key-acme,globex:lab-key-globex" \
  -e RATE_LIMIT_RPS="${RPS:-5}" -e RATE_LIMIT_BURST="${BURST:-10}" \
  llm-gateway:dev >/dev/null
sleep 1
echo "gw: rps=${RPS:-5} burst=${BURST:-10} ready=$(curl -s localhost:8080/readyz)"
EOF
cat > load.sh <<'EOF'
#!/usr/bin/env bash
# load.sh <tenant> <anzahl> <parallel> -> Statuscodes und p50/p95 in Sekunden
T=$1; N=${2:-20}; P=${3:-2}
seq "$N" | xargs -P "$P" -I{} curl -s -o /dev/null -m 10 \
  -w '%{http_code} %{time_total}\n' \
  -H "Authorization: Bearer lab-key-$T" -H 'Content-Type: application/json' \
  -d '{"model":"mistral-fake","messages":[{"role":"user","content":"hi"}]}' \
  localhost:8080/v1/chat/completions > "load-$T.txt"
echo "== $T: $(awk '{print $1}' "load-$T.txt" | sort | uniq -c | xargs)"
sort -n -k2 "load-$T.txt" | LC_ALL=C awk '{a[NR]=$2} END {
  printf "p50=%.2fs p95=%.2fs\n", a[int(NR*0.5)], a[int(NR*0.95)] }'
EOF
chmod +x gw-start.sh load.sh
./gw-start.sh
./load.sh acme 20 2

Expected:

gw: rps=5 burst=10 ready=ok
== acme: 20 200
p50=0.51s p95=0.51s

Why? This is your baseline. Without “normal” you cannot recognize “broken”. acme sends 2 parallel requests, which is about 4 per second and thus less than the limit of 5.

Tip: With llmctl load from Day 3 you get the same numbers plus p99. load.sh is enough here and only needs curl.

Save your progress? Sign up for free

Drill 2: Roles, severity and the timeline tool

Set the severity levels before the pager rings. When in doubt, rate higher and downgrade later.

SEV Criterion Example Response
SEV1 All tenants down, data leak, security incident API returns only 5xx Immediate page, IC mandatory, status page ≤ 15 min, updates 30 min
SEV2 One tenant heavily affected or SLO burn critical acme 50 % errors Page, IC mandatory, status page ≤ 30 min, updates 60 min
SEV3 Noticeably slower, SLO still holds p95 twice as high Ticket with priority, fix within the working day
SEV4 No customer impact Dashboard panel empty Ticket, backlog

The scribe needs a tool that writes an entry in 2 seconds. tl appends a table row with the UTC time to incident.md.

cat > tl <<'EOF'
#!/usr/bin/env bash
# tl "Text" -> hängt eine Zeile mit UTC-Zeit und Autor an die Timeline an
f=${INCIDENT_FILE:-incident.md}
[ $# -gt 0 ] || { echo "usage: tl <text>" >&2; exit 1; }
msg=${*//|/\\|}                       # Pipe würde die Tabelle zerbrechen
printf '| %s | %s | %s |\n' "$(date -u +%H:%M:%S)" "${WHO:-$USER}" "$msg" >> "$f"
tail -1 "$f"
EOF
chmod +x tl
cat > incident.md <<'EOF'
# INC-2026-1009-01: acme bekommt 503 vom Gateway

- Severity: SEV2 · Status: investigating
- IC: thomas · Ops Lead: thomas · Comms: thomas · Scribe: tl

## Timeline (UTC)

| Zeit | Wer | Ereignis |
|---|---|---|
EOF
export WHO=thomas PATH="$PWD:$PATH"
tl "Timeline angelegt"

Expected (time varies):

| 16:24:30 | thomas | Timeline angelegt |

Why? A timeline that you only write from memory after the incident is wrong. In the lab you play all four roles. In real life you always separate IC and Ops Lead as soon as two people are there.

Save your progress? Sign up for free

Drill 3: Triggering, detecting and classifying the disruption

Change CHG-4711 raises the rate limit “so that globex’s batch job runs through faster”. Afterwards globex starts the batch with 40 parallel requests.

RPS=50 BURST=100 ./gw-start.sh
tl "CHG-4711 live: RPS 5->50, BURST 10->100 (globex-Batch)"
./load.sh globex 600 40 > globex.out 2>&1 &     # Batch läuft ca. 60 s
sleep 3
./load.sh acme 20 2                              # Probe aus Sicht von acme

Expected (the error rate fluctuates between 40 and 80 % depending on timing):

gw: rps=50 burst=100 ready=ok
| 16:24:49 | thomas | CHG-4711 live: RPS 5->50, BURST 10->100 (globex-Batch) |
== acme: 10 200 10 503
p50=2.02s p95=2.50s

acme has not changed anything and still gets 50 % errors, and latency has quadrupled. Now narrow it down: which tenant, which code, what has changed?

cat > impact.sh <<'EOF'
#!/usr/bin/env bash
# impact.sh [metrics-datei] -> Anfragen, 5xx, 429 und Fehlerquote pro Mandant
if [ -n "$1" ]; then cat "$1"; else curl -s localhost:8080/metrics; fi |
LC_ALL=C awk -F'[{}" ]+' '/^gateway_requests_total\{/ {
  code=$3; t=$5; n=$NF; all[t]+=n
  if (code ~ /^5/) err[t]+=n; if (code == 429) rl[t]+=n }
  END { for (t in all) printf "%-7s total=%-4d 5xx=%-4d 429=%-4d err=%.1f%%\n",
        t, all[t], err[t], rl[t], 100*err[t]/all[t] }'
EOF
chmod +x impact.sh
./impact.sh
docker logs gw 2>&1 | jq -r 'select(.status>=500) | "\(.tenant) \(.status)"' \
  | sort | uniq -c
docker inspect gw --format '{{range .Config.Env}}{{println .}}{{end}}' | grep RATE
tl "acme ~50% 503, p50 2s (Baseline 0.5s), globex auch 503. SEV2, IC thomas"

Expected (shortened, numbers fluctuate):

globex  total=444  5xx=254  429=0    err=57.2%
acme    total=20   5xx=10   429=0    err=50.0%
  10 acme 503
 254 globex 503
RATE_LIMIT_RPS=50
RATE_LIMIT_BURST=100

Why? The 503s come from the model server (“server busy”) and are passed through. Occasional 502s for globex are possible: then the connection backlog of the fake server overflows. No tenant got a 429, so the gateway let everything through. grep RATE instead of the whole env list is deliberate: TENANT_KEYS does not belong in a chat or a ticket.

Save your progress? Sign up for free

Drill 4: Mitigation first – save evidence, roll back, verify

The IC decides: roll back CHG-4711. The cause is not proven yet, but the rollback is cheap and reversible. Before that, you save the metrics in one second.

curl -s localhost:8080/metrics > metrics-incident.txt
tl "IC: Rollback CHG-4711 (RPS 50->5). Metriken gesichert"
./gw-start.sh                                    # Defaults: RPS=5, BURST=10
tl "Rollback fertig"
./load.sh acme 20 2
tl "Verifikation: acme 20/20 200, p50 0.5s. Status: monitoring"

Expected:

gw: rps=5 burst=10 ready=ok
== acme: 20 200
p50=0.58s p95=1.83s

The p95 is still elevated while the batch is running: globex may now send 5 req/s, acme sends 4 req/s. Together that is 9 req/s, which is already more than the 8 req/s capacity. Remember this for the post-mortem.

wait; cat globex.out                             # Batch-Ergebnis von globex
ls gw-*.log metrics-incident.txt

Expected (shortened):

== globex: 144 000 198 200 8 502 250 503
p50=2.07s p95=3.17s
gw-163218.log  gw-163253.log  metrics-incident.txt

Why? The 000s are dropped connections during the restart. A rollback by container restart has a gap. In Kubernetes, a rolling update (Day 7) does this without that gap. After the rollback, globex gets 429 instead of 503. The gateway protects the shared capacity again.

Warning: Only once the mitigation works do you start the root cause analysis. The most common mistake in an incident: 40 minutes of debugging while the damage continues.

Save your progress? Sign up for free

Drill 5: Customer communication – status page and updates

Customers are international, so you write in English. Prepare the templates before the incident. In an emergency, Comms only fills in the gaps.

cat > comms.md <<'EOF'
## Initial (within 30 min of declaring SEV2)
[Investigating] Elevated error rates on the Chat Completions API
We are seeing elevated error rates (HTTP 503) and increased latency for some
customers using the Chat Completions API since 14:05 UTC. Our engineers are
investigating. Next update by 15:00 UTC or earlier.

## Update
[Identified] Elevated error rates on the Chat Completions API
We identified a configuration change that allowed excessive load on a shared
model server. We rolled back the change at 14:29 UTC. Error rates are back to
normal and we are monitoring. Next update by 15:30 UTC.

## Resolved
[Resolved] Elevated error rates on the Chat Completions API
Between 14:05 and 14:29 UTC, about 50% of Chat Completions requests for affected
customers failed with HTTP 503, and median latency rose from 0.5 s to 2.0 s.
The cause was a configuration change, which we rolled back. No data was lost or
exposed. Requests that failed can be retried safely. A post-incident report
will follow within 5 business days.
EOF
grep -c '^\[' comms.md
tl "Comms: Initial-Update auf Status-Page veröffentlicht"

Expected:

3

Rules for customer updates:

  • What the customer notices (symptom, since when, which API), not what is broken internally.
  • Always state a time for the next update and keep to it.
  • No speculation about causes, no names of other customers (“a shared model server”, not “globex”).
  • In the resolved update: time period, extent in numbers, data loss yes/no, retry safe yes/no.

Save your progress? Sign up for free

Drill 6: The blameless post-mortem

A post-mortem answers: What happened? How bad was it? Why could it happen? What do we change? The numbers below come from your simulation. The times are stretched to a real production duration (24 minutes).

cat > postmortem.md <<'EOF'
# Post-Mortem INC-2026-1009-01: 503 errors on Chat Completions

- Status: final · Severity: SEV2 · Date: 2026-10-08
- Authors: Thomas Zachmann · Reviewers: platform team
- Blameless: this document describes systems and processes, not people.

## Summary
A rate-limit change (CHG-4711) let one tenant's batch job send more traffic than the
shared model server can serve. The model server rejected the excess with HTTP 503,
which hit all tenants on that server. A rollback ended the impact after 24 minutes.

## Impact
- Duration: 14:05 to 14:29 UTC (24 min).
- Tenant acme: 50.0% of requests failed (HTTP 503), about 1,870 of 3,740.
- Tenant acme: median latency 0.51 s -> 2.02 s, p95 2.50 s.
- Tenant globex: 57.2% of batch requests failed (HTTP 503).
- Error budget acme (99.5% / 28 days): 5.5% used in this incident.
- No data loss, no data exposure between tenants.

## Detection
The burn-rate alert LLMGatewayHighErrorRate (Day 10) fired at 14:11 UTC, 6 minutes
after impact started. No customer reported the issue before the status page update.
EOF
cat >> postmortem.md <<'EOF'

## Timeline (UTC)
| Time | Event |
|---|---|
| 14:02 | CHG-4711 deployed: RATE_LIMIT_RPS 5 -> 50, BURST 10 -> 100 |
| 14:05 | globex batch job starts with 40 parallel requests. Impact starts |
| 14:11 | LLMGatewayHighErrorRate fires, on-call acknowledges |
| 14:14 | SEV2 declared, IC assigned, incident channel opened |
| 14:19 | Initial status page update |
| 14:24 | Correlation with CHG-4711 found via container config |
| 14:27 | IC decides rollback of CHG-4711 |
| 14:29 | Rollback complete, acme error rate 0%. Impact ends |
| 14:49 | Status page update: identified, monitoring |
| 15:05 | Resolved after 30 min clean monitoring |

## Root cause and contributing factors
Root cause: the gateway's only admission control is a per-tenant rate limit. Nothing
limits the total traffic to the shared model server (4 slots, about 8 req/s).
CHG-4711 allowed one tenant 50 req/s, six times the total capacity.

Contributing factors:
- The change process has no capacity check for limit changes.
- RATE_LIMIT_RPS applies to all tenants. The change meant for globex raised every
  tenant's limit.
- The gateway passes upstream 503 through without Retry-After.
- Deployments are not marked in Grafana. Correlating the change took 10 minutes.
- The lab-style rollback (restart) dropped 152 in-flight globex requests.
EOF
cat >> postmortem.md <<'EOF'

## What went well
- The burn-rate alert detected the impact within 6 minutes.
- The rollback was a one-line config change and took 2 minutes.
- Logs and metrics were saved before the rollback.

## What went badly
- Nobody load-tested the new limit against the model server capacity.
- The first status page update came 14 minutes after the alert.

## Where we got lucky
- The batch job ran during low traffic. At peak, all tenants would have hit 503.

## Action items
| ID | Action | Owner / Prio | Due / Status |
|---|---|---|---|
| AI-1 | Gateway: global in-flight limit | thomas / P1 | 2026-10-22 / open |
| AI-2 | Per-tenant rate limits | thomas / P1 | 2026-10-22 / open |
| AI-3 | CI check: sum of limits <= capacity | platform / P2 | 2026-11-05 / open |
| AI-4 | Deploy annotations in Grafana (Day 8) | obs / P2 | 2026-10-15 / done |
| AI-5 | Status page template prefilled from alert | sre / P3 | 2026-11-30 / open |
EOF
wc -l postmortem.md

Expected:

      70 postmortem.md

Why? “Root cause” plus “contributing factors” is more honest than a single cause. The “Where we got lucky” section shows risks that did not materialize this time.

Save your progress? Sign up for free

Drill 7: 5 Whys versus contributing factors

5 Whys forces a single chain. That is good for getting started, but dangerous:

Why 503 for acme?             Model server full.
Why full?                     globex sent 40 parallel requests.
Why did that get through?     Rate limit was 50 instead of 5.
Why 50?                       CHG-4711.
Why did CHG-4711 get through? No capacity check in the change process.

The chain ends at the change process. Had you stopped at “Why 50?” with “Person X made a mistake”, the result would be blame instead of learning. Equally dangerous is what the chain leaves out: the global RATE_LIMIT_RPS, the missing 503 Retry-After and the missing deploy markers do not appear in a single chain. That is why in the post-mortem you use the list of contributing factors. Every factor has an action item or a conscious decision “accepted because …”.

Check your post-mortem for blame language and for missing numbers:

grep -niwE 'should have|careless|mistake|blame|human error' postmortem.md \
  && echo "✘ Schuld-Sprache gefunden" || echo "✔ keine Schuld-Sprache"
grep -cE '[0-9]+(\.[0-9]+)?( ?%| s| min| req/s)' postmortem.md

Expected:

✔ keine Schuld-Sprache
15

Why? “Human error” is never a root cause but a symptom: the system allowed the mistake. The number shows how many lines contain measured values. A post-mortem without numbers is an opinion.

Save your progress? Sign up for free

Drill 8: Tracking action items and the review meeting

Action items without an owner and a date never get done. This script lists open and overdue items, for example for the weekly ops review.

cat > ai-status.sh <<'EOF'
#!/usr/bin/env bash
# ai-status.sh <postmortem.md> [stichtag YYYY-MM-DD] -> offene/überfällige Items
today=${2:-$(date -u +%F)}
LC_ALL=C awk -F' *\\| *' -v today="$today" '$2 ~ /^AI-[0-9]+$/ {
  split($5, d, " / "); due=d[1]; st=d[2]
  if (st == "done") { done++; next }
  open++; flag = (due < today) ? "OVERDUE" : "open"
  printf "%-7s %-5s %-11s %s\n", flag, $2, due, $3 }
  END { printf "open=%d done=%d\n", open, done }' "$1"
EOF
chmod +x ai-status.sh
./ai-status.sh postmortem.md
./ai-status.sh postmortem.md 2026-10-30

Expected:

open    AI-1  2026-10-22  Gateway: global in-flight limit
open    AI-2  2026-10-22  Per-tenant rate limits
open    AI-3  2026-11-05  CI check: sum of limits <= capacity
open    AI-5  2026-11-30  Status page template prefilled from alert
open=4 done=1
OVERDUE AI-1  2026-10-22  Gateway: global in-flight limit
OVERDUE AI-2  2026-10-22  Per-tenant rate limits
open    AI-3  2026-11-05  CI check: sum of limits <= capacity
open    AI-5  2026-11-30  Status page template prefilled from alert
open=4 done=1

In real life, action items live in the ticket system (Jira, GitLab Issues), and the post-mortem links to them. The table is only the index.

Review meeting (45 min, within 5 business days):

  • 5 min: the author reads out the summary and impact. Everyone has read the document beforehand.
  • 15 min: walk through the timeline. Where did we lose time (here: 14 min until the status page, 10 min until the correlation)?
  • 15 min: contributing factors and action items. Every item gets an owner, a priority and a date.
  • 10 min: What can other teams learn from this? Afterwards, share the post-mortem widely.
  • Moderated by someone who was not involved in the incident.

Writing rules for runbooks and post-mortems:

  • Short sentences in the active voice: “Roll back CHG-4711”, not “It should be rolled back”.
  • Numbers instead of adjectives: “50 % 503 for 24 min”, not “lots of errors”.
  • Times always in UTC, commands ready to copy, and the expected output belongs with them.
  • Summary first. Anyone who reads only 3 sentences must still know what happened.
  • A runbook step is one action plus one check command.

Save your progress? Sign up for free

Checkpoint

cd ~/sre-lab/tag11
[ "$(grep -c '^| [0-9][0-9]:' incident.md)" -ge 5 ] \
  && echo "✔ Timeline mit ≥ 5 Einträgen" || echo "✘ Timeline zu kurz"
for s in Summary Impact Timeline 'Root cause' 'Action items'; do
  grep -q "^## $s" postmortem.md && echo "✔ PM: $s" || echo "✘ PM: $s fehlt"
done
awk -F' *\\| *' '$2 ~ /^AI-/ && ($4 !~ / \/ P[1-3]$/ || $5 !~ /^20[0-9-]+ \//)' \
  postmortem.md | grep -q . && echo "✘ Action Item ohne Owner/Due" \
  || echo "✔ alle Action Items mit Owner, Prio, Due"
./load.sh acme 20 2 | grep -q '== acme: 20 200' \
  && echo "✔ acme wieder gesund" || echo "✘ acme hat noch Fehler"

Expected:

✔ Timeline mit ≥ 5 Einträgen
✔ PM: Summary
✔ PM: Impact
✔ PM: Timeline
✔ PM: Root cause
✔ PM: Action items
✔ alle Action Items mit Owner, Prio, Due
✔ acme wieder gesund

Challenge

Stop the model server (docker stop fake-llm). Work through the incident in a second file incident-2.md: measure the symptom, set the SEV, write the initial update, mitigate, verify. At the end the timeline has at least 5 entries.

Hint: INCIDENT_FILE points tl to another file, and /readyz tells you whether the gateway can still reach the upstream.

Solution

cd ~/sre-lab/tag11
export INCIDENT_FILE=incident-2.md
printf '# INC-2026-1009-02: Modellserver down\n\n| Zeit | Wer | Ereignis |\n' \
  > incident-2.md
printf '|---|---|---|\n' >> incident-2.md
docker stop fake-llm
./load.sh acme 10 2; curl -s localhost:8080/readyz
tl "acme 100% 502, readyz: upstream not ready. SEV1 (alle Mandanten), IC thomas"
tl "Comms: Initial-Update 'Chat Completions API unavailable' veröffentlicht"
docker start fake-llm
until curl -sf localhost:8080/readyz; do sleep 1; done
tl "Mitigation: fake-llm neu gestartet, readyz ok"
./load.sh acme 10 2
tl "Verifikation: acme 10/10 200. Status: monitoring"
tl "Resolved (im Lab ohne die üblichen 30 min Monitoring)"
grep -c '^| [0-9]' incident-2.md

Expected:

== acme: 10 502
p50=0.08s p95=0.52s
upstream not ready
...
ok
== acme: 10 200
p50=0.54s p95=0.55s
5

Why 502 instead of 503? The server is gone, the connection fails, and the ErrorHandler in the gateway responds with 502. SEV1, because every tenant is affected.

Clean up:

docker rm -f gw fake-llm && docker network rm tag11

Save your progress? Sign up for free

Interview Note

  • Typical question: “Walk me through an incident you led.” Answer in the order detect, mitigate, communicate, learn, with numbers (duration, error rate, customers).
  • A crisp answer on priorities: “Mitigation before root cause. I’ll roll back a suspicious change even before I can prove it caused the issue, if rollback is cheap.”
  • Explain blameless like this: “If one person’s mistake can cause an outage, the system allowed it. The action item fixes the system.”
  • Bridge from your own practice: a Vault/OpenBao outage (seal, Raft quorum, expired tokens) that affected many teams at once, and how you kept communication and the fix separate.
  • Show your writing strength: offer to bring an anonymized post-mortem or runbook as a work sample.

Enterprise Note

  • On-prem at the customer you often have no access to the systems. The runbook must be written so that the customer runs the diagnostic commands themselves and sends you the output (support bundle, Day 14).
  • Contracts define response times per severity. Your SEV matrix must match the SLA clauses; otherwise you argue about definitions after the incident.
  • Security incidents (a data leak between tenants) come with reporting deadlines, e.g. 72 h under GDPR Art. 33 and NIS2. Legal and the DPO then belong in the channel immediately.
  • Post-mortems for customers are a separate, shortened version: without internal names, without names of other customers, with clear commitments.
  • Tools: incident.io, PagerDuty, Grafana IRM or FireHydrant create the channel, roles and timeline automatically. The principles from this chapter stay the same.