Day 10: SLOs, Alerting and On-Call
Goal: You define an SLO for the LLM API, alert with multi-window multi-burn-rate rules, prove the rules with unit tests, route alerts by severity and tenant, and ship a runbook with them. · Duration: approx. 150 min · Prerequisite: Day 8
Briefing
- SLI = measured share of good events, SLO = target for it over a time window, error budget = 1 − SLO.
- You do not alert on “error rate > X” but on how fast the budget burns (burn rate).
- Page only when a human must act now. Everything else becomes a ticket.
- Every alert has a runbook (
runbook_url), every rule has a unit test.
Burn rate 14.4 ─┐ 1h AND 5m ──► page ─┐
Burn rate 6 ──┤ 6h AND 30m ──► page ├─► Alertmanager ─► pager / csm-acme
Burn rate 3 ──┤ 1d AND 2h ──► ticket │ (inhibit: page swallows ticket)
Burn rate 1 ──┘ 3d AND 6h ──► ticket ─┘ └► ticket
Drill 1: Choosing SLIs for an LLM API
| SLI | Good | Counted | Source |
|---|---|---|---|
| Availability | no 5xx | all except 4xx | gateway_requests_total |
| Latency (e2e) | response ≤ 10 s | successful requests | …_duration_seconds_bucket{le="10"} |
| TTFT (streaming) | first token ≤ 1 s | all requests | vllm:time_to_first_token_seconds |
| Throughput (optional) | ≥ N tokens/s | per tenant | gateway_tokens_total |
Why leave out 4xx? 401 (wrong key) and 429 (tenant over its limit)
are client behavior. If they count, a customer with a wrong key can burn
your budget. Warning: llmctl slo from Day 3 divides by
all requests – with many 429s things look better there
than they are.
mkdir -p ~/sre-lab/tag10 && cd ~/sre-lab/tag10
OBS=~/sre-lab/code/observability
kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus \
9090:9090 >/dev/null 2>&1 &
pq() {
curl -s localhost:9090/api/v1/query --data-urlencode "query=$1" | jq -r \
'.data.result[] | "\(.metric | del(.__name__) | tostring) \(.value[1]
| tonumber * 1000 | round / 1000)"'
}
pq 'sum by (tenant) (rate(gateway_request_duration_seconds_bucket{le="10"}[1h]))
/ sum by (tenant) (rate(gateway_request_duration_seconds_count[1h]))'Expected (shortened, may differ by version):
{"tenant":"acme"} 1
{"tenant":"globex"} 1
For non-streaming, e2e latency depends on the response length. When streaming with vLLM, TTFT is the better latency SLI – the gateway does not measure it.
Save your progress? Sign up for free
Drill 2: Calculating the error budget
cat > budget.py <<'EOF'
"""Fehlerbudget und Burn Rates für SLO 99,5 % über 28 Tage."""
slo, days = 0.995, 28
budget = 1 - slo
period_h = days * 24
print(f"Budget: {budget:.3%} = {budget * period_h * 60:.0f} min Ausfall "
f"in {days} Tagen")
print(f"bei 10 req/s: {budget * 10 * period_h * 3600:,.0f} fehlerhafte Anfragen")
print("Burn Fenster Schwelle Budget leer nach Verbrauch im Fenster")
for burn, win_h in [(14.4, 1), (6, 6), (3, 24), (1, 72)]:
empty_h = period_h / burn
used = burn * win_h / period_h
print(f"{burn:>4}x {win_h:>5} h {burn * budget:>8.2%} {empty_h / 24:>9.1f} Tage"
f" {used:>12.1%}")
EOF
python3 budget.pyExpected:
Budget: 0.500% = 202 min Ausfall in 28 Tagen
bei 10 req/s: 120,960 fehlerhafte Anfragen
Burn Fenster Schwelle Budget leer nach Verbrauch im Fenster
14.4x 1 h 7.20% 1.9 Tage 2.1%
6x 6 h 3.00% 4.7 Tage 5.4%
3x 24 h 1.50% 9.3 Tage 10.7%
1x 72 h 0.50% 28.0 Tage 10.7%
The factors 14.4/6/3/1 come from the Google SRE Workbook (chapter “Alerting on SLOs”) and are calculated for 30 days: 2 %, 5 % and 10 % of the budget. For 28 days the same calculation gives 13.44/5.6/2.8/0.93 – Drill 7 shows this with Sloth. Both are defensible; what matters is that the team and the customer know the same number.
Save your progress? Sign up for free
Drill 3: SLO rules – recording and burn-rate alerts
sed -n '1,13p' $OBS/rules/llm-gateway-slo.rules.yaml
docker run --rm -v $OBS/rules:/r -w /r --entrypoint promtool \
prom/prometheus:v3.15.0 check rules llm-gateway-slo.rules.yamlExpected:
# Tag 10: SLO "Verfügbarkeit 99,5 % / 28 Tage" für das llm-gateway, pro Mandant.
# SLI = Anteil 5xx an allen Anfragen ohne 4xx (401/429 sind Fehler des Clients).
# Fehlerbudget = 1 - 0.995 = 0.005. Burn Rate 1 = Budget ist nach 28 Tagen leer.
groups:
- name: llm-gateway.slo.recording
interval: 30s
rules:
- record: tenant:slo_errors:ratio_rate5m
expr: |
(sum by (tenant) (rate(gateway_requests_total{code=~"5.."}[5m]))
or sum by (tenant) (rate(gateway_requests_total[5m])) * 0)
/ sum by (tenant) (rate(gateway_requests_total{code!~"4.."}[5m]))
- record: tenant:slo_errors:ratio_rate30m
Checking llm-gateway-slo.rules.yaml
SUCCESS: 7 rules found
Why or … * 0? A tenant without a single 5xx has no 5xx
time series. Without the trick its ratio is missing entirely (Day 8,
Drill 4) – dashboards show gaps instead of 0. Seven windows (5m to 3d)
are precomputed once; otherwise every alert evaluation recalculates 3
days of raw data.
The alerts build on these windows:
sed -n '4,22p' $OBS/rules/llm-gateway-alerts.rules.yamlExpected:
- name: llm-gateway.slo.alerts
rules:
- alert: LLMGatewayHighErrorRate
expr: |
tenant:slo_errors:ratio_rate1h > (14.4 * 0.005)
and tenant:slo_errors:ratio_rate5m > (14.4 * 0.005)
for: 2m
labels: { severity: page, long_window: 1h }
annotations: &page
summary: "llm-gateway: hohe 5xx-Quote für Mandant {{ $labels.tenant }}"
description: >-
{{ $value | humanizePercentage }} Fehler. Bei diesem Tempo ist das
28-Tage-Fehlerbudget in wenigen Tagen aufgebraucht.
runbook_url: >-
https://runbooks.example.com/llm-gateway/LLMGatewayHighErrorRate
dashboard_url: https://grafana.example.com/d/llm-gateway
- alert: LLMGatewayHighErrorRate
expr: |
tenant:slo_errors:ratio_rate6h > (6 * 0.005)
Why two windows? The long window (1h) provides significance, the
short one (5m) makes sure the alert resolves quickly after the fix. The
file also contains LLMGatewayErrorBudgetBurn (3x/1x,
ticket), LLMGatewayDown and
LLMGatewayHighLatency (p95 > 20 s, ticket).
&page/*page is a YAML anchor: both page
rules share the same annotations.
Save your progress? Sign up for free
Drill 4: Unit-testing rules with promtool
The test file feeds three tenants with synthetic counters: acme 12.5 % errors (fast burn), globex 2 % (slow burn), initech 0 %.
sed -n '10,21p' $OBS/rules/llm-gateway.test.yaml
grep -c 'eval_time' $OBS/rules/llm-gateway.test.yaml
docker run --rm -v $OBS/rules:/r -w /r --entrypoint promtool \
prom/prometheus:v3.15.0 test rules llm-gateway.test.yamlExpected:
# acme: 7 rps ok, 1 rps 5xx, 2 rps 429 -> SLI-Fehlerquote 1/8 = 12,5 %
- series: 'gateway_requests_total{tenant="acme",code="200"}'
values: '0+420x240'
- series: 'gateway_requests_total{tenant="acme",code="502"}'
values: '0+60x240'
- series: 'gateway_requests_total{tenant="acme",code="429"}'
values: '0+120x240'
- series: 'gateway_ratelimited_total{tenant="acme"}'
values: '0+120x240'
# globex: 2 % Fehler -> langsamer Burn (nur Ticket)
- series: 'gateway_requests_total{tenant="globex",code="200"}'
values: '0+588x240'
10
SUCCESS
Ten checks: six recording rule values, four alert points in time
(fires, ticket instead of page, for not yet satisfied,
gateway down). '0+420x240' means: start at 0, +420 per
interval (1m), 240 times. Now break it on purpose – the threshold of the
1h window wrong by a factor of 10:
cp $OBS/rules/*.yaml .
sed -i.bak 's/ratio_rate1h > (14.4 \* 0.005)/ratio_rate1h > (14.4 * 0.05)/' \
llm-gateway-alerts.rules.yaml
docker run --rm -v $PWD:/r -w /r --entrypoint promtool \
prom/prometheus:v3.15.0 test rules llm-gateway.test.yaml 2>&1 | grep -v '^$' | head -4
docker run --rm -v $PWD:/r -w /r --entrypoint promtool \
prom/prometheus:v3.15.0 test rules llm-gateway.test.yaml >/dev/null 2>&1
echo "exit=$?"Expected (shortened: what follows is exp with both page
alerts and got with only the 6h alert):
FAILED:
alertname: LLMGatewayHighErrorRate, time: 30m,
exp:[
0:
exit=1
No review reliably finds exactly this mistake (a factor of 10 in the
budget) – the test does. In CI: promtool check rules and
promtool test rules on every merge.
Save your progress? Sign up for free
Drill 5: Rolling out rules and routing
mk_rule() { # Regeldatei -> PrometheusRule-Objekt (wie Tag 8)
printf 'apiVersion: monitoring.coreos.com/v1\nkind: PrometheusRule\n'
printf 'metadata:\n name: %s\n namespace: monitoring\nspec:\n' "$1"
sed -e '/^#/d' -e 's/^/ /' "$2"
}
mk_rule llm-gateway-slo $OBS/rules/llm-gateway-slo.rules.yaml | kubectl apply -f -
mk_rule llm-gateway-alerts $OBS/rules/llm-gateway-alerts.rules.yaml \
| kubectl apply -f -
am() { docker run --rm -v $OBS/alertmanager:/c --entrypoint amtool \
prom/alertmanager:v0.34.1 "$@"; }
am check-config /c/alertmanager.yaml | head -1
am config routes show --config.file=/c/alertmanager.yaml
for l in "severity=page tenant=acme" "severity=ticket tenant=acme" \
"severity=page tenant=globex" "severity=ticket tenant=globex"; do
printf '%-30s -> ' "$l"; am config routes test --config.file=/c/alertmanager.yaml $l
doneExpected (amtool live, kubectl
shortened):
prometheusrule.monitoring.coreos.com/llm-gateway-slo created
prometheusrule.monitoring.coreos.com/llm-gateway-alerts created
Checking '/c/alertmanager.yaml' SUCCESS
Routing tree:
.
└── default-route receiver: ticket
├── {alertname="Watchdog"} receiver: deadmans-switch
├── {alertname="InfoInhibitor"} receiver: null
├── {tenant="acme"} continue: true receiver: csm-acme
├── {severity="page"} receiver: pager
└── default-route receiver: ticket
severity=page tenant=acme -> csm-acme,pager
severity=ticket tenant=acme -> csm-acme,ticket
severity=page tenant=globex -> pager
severity=ticket tenant=globex -> ticket
Why the empty route at the end? Once a route with
continue: true has matched, Alertmanager no longer falls
back to the root receiver. Without a catch-all, an acme ticket would
only be seen by the CSM team. Inhibition: a page suppresses the ticket
of the same tenant, and LLMGatewayDown is supposed to
suppress all other gateway alerts – whether
equal: [namespace] really does that is something you check
on Day 15, Drill 5.
Now start a lab receiver and load the config via Helm:
cat > sink.py <<'EOF'
import http.server, json
class H(http.server.BaseHTTPRequestHandler):
def do_POST(self):
body = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
for a in body["alerts"]:
print(self.path, a["status"], a["labels"].get("alertname"),
a["labels"].get("tenant", "-"), flush=True)
self.send_response(200)
self.end_headers()
http.server.HTTPServer(("", 8080), H).serve_forever()
EOF
kubectl -n monitoring run alert-sink --image=python:3.13-alpine --port=8080 \
--command -- python -u -c "$(cat sink.py)"
kubectl -n monitoring expose pod alert-sink --port=8080
helm upgrade kps prometheus-community/kube-prometheus-stack --version 92.2.0 \
-n monitoring -f $OBS/helm/kps-values.yaml \
-f $OBS/helm/grafana-datasources.yaml -f $OBS/helm/alertmanager-values.yaml --waitalertmanager-values.yaml is
alertmanager.yaml, indented under
alertmanager.config. The chart itself appends
templates: [/etc/alertmanager/config/*.tmpl].
Save your progress? Sign up for free
Drill 6: Firing, inhibiting and silencing alerts
kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-alertmanager \
9093:9093 >/dev/null 2>&1 &
amk() { # amtool im Alertmanager-Pod gegen die laufende Instanz
kubectl -n monitoring exec alertmanager-kps-kube-prometheus-stack-alertmanager-0 \
-c alertmanager -- amtool --alertmanager.url=http://localhost:9093 "$@"
}
amk alert add alertname=LLMGatewayHighErrorRate severity=page tenant=acme
amk alert add alertname=LLMGatewayErrorBudgetBurn severity=ticket tenant=acme
amk alert add alertname=LLMGatewayHighErrorRate severity=page tenant=globex
amk alert add alertname=LLMGatewayErrorBudgetBurn severity=ticket tenant=initech
sleep 40
kubectl -n monitoring logs alert-sink | grep '^/'
amk alert query --inhibitedExpected (live against Alertmanager 0.34.1):
/page firing LLMGatewayHighErrorRate acme
/csm-acme firing LLMGatewayHighErrorRate acme
/page firing LLMGatewayHighErrorRate globex
/ticket firing LLMGatewayErrorBudgetBurn initech
/heartbeat firing Watchdog -
Alertname Starts At Summary State
LLMGatewayErrorBudgetBurn 2026-10-09 16:38:58 UTC suppressed
The acme ticket is inhibited because acme is already paging. Maintenance at globex? A silence with a ticket number and an expiry time:
ID=$(amk silence add alertname=LLMGatewayErrorBudgetBurn tenant=globex \
--duration=2h --author=thomas --comment="Wartung globex, Ticket OPS-123")
amk silence query -o json \
| jq -r '.[] | "\(.id[0:8]) \(.endsAt) \(.createdBy): \(.comment)"'
amk silence expire $ID
amk silence query -q | wc -lExpected (live, ID and time differ):
9101b457 2026-10-09T19:21:04.331Z thomas: Wartung globex, Ticket OPS-123
0
The real test: model server gone, load on, wait. After about 3–7 min the 1h/5m alert fires:
kubectl -n llm-system scale deploy/ollama --replicas=0 # Variante A, Tag 7
LLMCTL_API_KEY=lab-key-acme llmctl load -n 300 -c 2 >/dev/null
pq 'ALERTS{alertname="LLMGatewayHighErrorRate", alertstate="firing"}'
kubectl -n llm-system scale deploy/ollama --replicas=1Expected (shortened, may differ by version):
{"alertname":"LLMGatewayHighErrorRate","alertstate":"firing","long_window":"1h",
"severity":"page","tenant":"acme"} 1
Save your progress? Sign up for free
Drill 7: Writing a runbook, knowing SLO generators
Template: meaning · impact · diagnosis (command + expectation per
step) · remediation · escalation · follow-up. The runbook lives in Git
next to the rules, and the runbook_url points to it.
mkdir -p runbooks && cat > runbooks/LLMGatewayHighErrorRate.md <<'EOF'
# Runbook: LLMGatewayHighErrorRate
Severity: page | Owner: Plattform-SRE | SLO: 99,5 % Verfügbarkeit / 28 Tage
Dashboard: https://grafana.example.com/d/llm-gateway
## Bedeutung
Die 5xx-Quote eines Mandanten verbrennt das Fehlerbudget mindestens 14,4-fach (1h)
oder 6-fach (6h). Label `tenant` = betroffener Mandant.
## Auswirkung
Kunden des Mandanten bekommen 502/503/504 auf /v1/chat/completions.
## Diagnose
1. Ein oder alle Mandanten? pq 'tenant:slo_errors:ratio_rate5m'
Alle betroffen -> Gateway/Modellserver. Einer -> Mandanten-Last, Quota.
2. Welche Codes? pq 'sum by (code) (rate(gateway_requests_total{code=~"5.."}[5m]))'
502 = Modellserver nicht erreichbar, 503 = überlastet, 504 = Timeout.
3. Pods und Endpoints: kubectl -n llm-system get pods,endpoints -o wide
Erwartet: alle Running/Ready, Endpoints für Gateway und Modellserver.
4. Letzte Änderung? kubectl -n llm-system rollout history deploy/llm-gateway
und Deploy-Markierung im Dashboard.
5. Logs -> Trace (Grafana Explore, Loki):
{app="llm-gateway"} | json | status >= 500 -> "Trace öffnen" -> roter Span?
6. Modellserver: OOMKilled? GPU-Speicher (DCGM_FI_DEV_FB_USED),
Warteschlange (vllm:num_requests_waiting).
## Abhilfe
- Deploy schuld: kubectl -n llm-system rollout undo deploy/llm-gateway
- Überlast: Modellserver skalieren, RATE_LIMIT_RPS des lauten Mandanten senken
- Pod hängt: kubectl -n llm-system delete pod <pod> (Controller startet neu)
- Prüfen: pq 'tenant:slo_errors:ratio_rate5m' fällt unter 0.072
## Eskalation
- Nach 15 min ohne Besserung: Incident ausrufen (Tag 11), Incident Commander holen.
- Premium-Kunde (acme): CSM ist über die Route csm-acme schon informiert.
## Nachbereitung
- Silences nur mit Ticket-ID und Ablaufzeit.
- SEV1/SEV2: Post-Mortem (Tag 11). Dieses Runbook im selben PR anpassen.
EOF
grep -c '^## ' runbooks/LLMGatewayHighErrorRate.mdExpected:
6
Sloth and Pyrra generate the same rules from a short SLO specification. With Sloth (container, no installation):
cat > sloth.yaml <<'EOF'
version: "prometheus/v1"
service: "llm-gateway"
slos:
- name: "availability"
objective: 99.5
sli:
events:
error_query: >-
sum by (tenant) (rate(gateway_requests_total{code=~"5.."}[{{.window}}]))
total_query: >-
sum by (tenant) (rate(gateway_requests_total{code!~"4.."}[{{.window}}]))
alerting:
name: LLMGatewaySLOBurn
page_alert: { labels: { severity: page } }
ticket_alert: { labels: { severity: ticket } }
EOF
docker run --rm -v $PWD:/s ghcr.io/slok/sloth:v0.16.0 generate -i /s/sloth.yaml \
--default-slo-period=28d 2>/dev/null > sloth-rules.yaml
grep -c 'record:' sloth-rules.yaml
grep -oE '> \([0-9.]+ \* 0.005\)' sloth-rules.yaml | sort -uExpected:
15
> (0.9333333333333333 * 0.005)
> (13.44 * 0.005)
> (2.8000000000000003 * 0.005)
> (5.6000000000000005 * 0.005)
Same windows as yours, factors exact for 28 days. Pyrra does the same
as a Kubernetes operator (CRD ServiceLevelObjective) with
its own UI. Handwritten, you understand the rules – generated, they
scale across hundreds of services.
Save your progress? Sign up for free
Drill 8: On-call hygiene
pq 'sort_desc(sum by (alertname, severity)
(count_over_time(ALERTS{alertstate="firing"}[7d])) * 15 / 60)'
cat > handover.txt <<'EOF'
Übergabe On-Call <Datum> von <Name> an <Name>
Offene Incidents: <ID, Status, nächster Schritt>
Aktive Silences: <amtool silence query> (alle mit Ticket + Ablauf?)
Laute Alerts (7 Tage): <Top 3 aus der Abfrage oben> -> Ticket zum Fixen/Löschen
Toil diese Woche: <manuelle Handgriffe, Minuten> -> Automatisierungs-Ticket
Fehlerbudget: <pq 'tenant:slo_errors:ratio_rate1d' je Mandant>
EOFExpected (shortened, may differ by version):
{"alertname":"Watchdog","severity":"none"} 10080
{"alertname":"LLMGatewayHighErrorRate","severity":"page"} 37.5
{"alertname":"KubePodCrashLooping","severity":"warning"} 12
The value is the firing duration in minutes (evaluation every 15 s). Rules for the weekly alert review: every page must have triggered an action; otherwise it becomes a ticket or gets deleted. The target according to Google SRE: at most two incidents per 12-hour shift, toil below 50 % of working time.
Save your progress? Sign up for free
Checkpoint
#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
OBS=~/sre-lab/code/observability
check "promtool-Tests grün" "docker run --rm -v $OBS/rules:/r -w /r \
--entrypoint promtool prom/prometheus:v3.15.0 test rules llm-gateway.test.yaml"
check "Alertmanager-Config gültig" "docker run --rm -v $OBS/alertmanager:/c \
--entrypoint amtool prom/alertmanager:v0.34.1 check-config /c/alertmanager.yaml"
check "SLO-Regeln geladen" "curl -s localhost:9090/api/v1/rules \
| jq -e '[.data.groups[].name] | index(\"llm-gateway.slo.alerts\")'"
check "Recording Rule liefert" "curl -s localhost:9090/api/v1/query \
--data-urlencode 'query=tenant:slo_errors:ratio_rate1h' \
| jq -e '.data.result | length > 0'"
check "Alertmanager erreichbar" "curl -sf localhost:9093/-/ready"
check "Runbook vollständig" \
"[ \$(grep -c '^## ' ~/sre-lab/tag10/runbooks/LLMGatewayHighErrorRate.md) -eq 6 ]"Challenge
Prove with a unit test that globex, at a constant 2 % errors, gets no page after 10 hours, but does get a ticket.
Hint: Your own test file with only globex series;
exp_alerts: [] checks that an alert does not fire.
Solution
cat > globex.test.yaml <<'EOF'
rule_files:
- llm-gateway-recording.rules.yaml
- llm-gateway-slo.rules.yaml
- llm-gateway-alerts.rules.yaml
evaluation_interval: 30s
tests:
- interval: 1m
input_series:
- series: 'gateway_requests_total{tenant="globex",code="200"}'
values: '0+588x600'
- series: 'gateway_requests_total{tenant="globex",code="500"}'
values: '0+12x600'
alert_rule_test:
- eval_time: 10h
alertname: LLMGatewayHighErrorRate
exp_alerts: []
- eval_time: 10h
alertname: LLMGatewayErrorBudgetBurn
exp_alerts:
- exp_labels: { severity: ticket, tenant: globex }
exp_annotations:
summary: "llm-gateway: Fehlerbudget von globex schmilzt"
runbook_url: >-
https://runbooks.example.com/llm-gateway/LLMGatewayErrorBudgetBurn
EOF
cp $OBS/rules/*.rules.yaml .
docker run --rm -v $PWD:/r -w /r --entrypoint promtool \
prom/prometheus:v3.15.0 test rules globex.test.yamlSave your progress? Sign up for free
Interview Note
- Typical question: “How would you set up alerting for our inference API?” – SLIs from the customer’s point of view (availability without 4xx, TTFT), agree on the SLO with product, burn-rate alerts, page only on a fast burn.
- Crisp: “I alert on symptoms at the customer, not on causes like CPU – causes go in the runbook.”
- Do the math live: 99.5 % over 28 days is 202 minutes; burn rate 14.4 empties the budget in just under two days.
- Bridge from your own practice: for regulated customers using Vault/OpenBao you wrote operations documentation and emergency manuals – runbooks with “command + expected result” are the same discipline.
- Rules are code:
promtool test rulesin CI, an alert review every week, every page needs an action.
Enterprise Note
- SLOs per customer by contract (the SLA is usually looser than the internal SLO), a monthly error budget report to the customer and product.
- Real paging via PagerDuty, Opsgenie or Grafana IRM; Alertmanager in an HA cluster (3 replicas, gossip).
- Dead man’s switch: the
Watchdogalert must arrive externally (e.g. Healthchecks.io); otherwise nobody notices that alerting itself has failed. - On-prem at the customer: often their Alertmanager or ITSM (ServiceNow). Ship rules, a routing proposal and runbooks in the chart.
- Follow-the-sun rotation across EMEA/US, clear escalation levels and rest periods after night pages.