Skip to content
Hands-on IT IT Drills

Day 10: SLOs, Alerting and On-Call

Goal: You define an SLO for the LLM API, alert with multi-window multi-burn-rate rules, prove the rules with unit tests, route alerts by severity and tenant, and ship a runbook with them. · Duration: approx. 150 min · Prerequisite: Day 8

Briefing

  • SLI = measured share of good events, SLO = target for it over a time window, error budget = 1 − SLO.
  • You do not alert on “error rate > X” but on how fast the budget burns (burn rate).
  • Page only when a human must act now. Everything else becomes a ticket.
  • Every alert has a runbook (runbook_url), every rule has a unit test.
 Burn rate 14.4 ─┐ 1h AND 5m  ──► page   ─┐
 Burn rate 6   ──┤ 6h AND 30m ──► page    ├─► Alertmanager ─► pager / csm-acme
 Burn rate 3   ──┤ 1d AND 2h  ──► ticket  │    (inhibit: page swallows ticket)
 Burn rate 1   ──┘ 3d AND 6h  ──► ticket ─┘                 └► ticket

Drill 1: Choosing SLIs for an LLM API

SLI Good Counted Source
Availability no 5xx all except 4xx gateway_requests_total
Latency (e2e) response ≤ 10 s successful requests …_duration_seconds_bucket{le="10"}
TTFT (streaming) first token ≤ 1 s all requests vllm:time_to_first_token_seconds
Throughput (optional) ≥ N tokens/s per tenant gateway_tokens_total

Why leave out 4xx? 401 (wrong key) and 429 (tenant over its limit) are client behavior. If they count, a customer with a wrong key can burn your budget. Warning: llmctl slo from Day 3 divides by all requests – with many 429s things look better there than they are.

mkdir -p ~/sre-lab/tag10 && cd ~/sre-lab/tag10
OBS=~/sre-lab/code/observability
kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus \
  9090:9090 >/dev/null 2>&1 &
pq() {
  curl -s localhost:9090/api/v1/query --data-urlencode "query=$1" | jq -r \
    '.data.result[] | "\(.metric | del(.__name__) | tostring) \(.value[1]
      | tonumber * 1000 | round / 1000)"'
}
pq 'sum by (tenant) (rate(gateway_request_duration_seconds_bucket{le="10"}[1h]))
    / sum by (tenant) (rate(gateway_request_duration_seconds_count[1h]))'

Expected (shortened, may differ by version):

{"tenant":"acme"} 1
{"tenant":"globex"} 1

For non-streaming, e2e latency depends on the response length. When streaming with vLLM, TTFT is the better latency SLI – the gateway does not measure it.

Save your progress? Sign up for free

Drill 2: Calculating the error budget

cat > budget.py <<'EOF'
"""Fehlerbudget und Burn Rates für SLO 99,5 % über 28 Tage."""
slo, days = 0.995, 28
budget = 1 - slo
period_h = days * 24
print(f"Budget: {budget:.3%} = {budget * period_h * 60:.0f} min Ausfall "
      f"in {days} Tagen")
print(f"bei 10 req/s: {budget * 10 * period_h * 3600:,.0f} fehlerhafte Anfragen")
print("Burn  Fenster  Schwelle  Budget leer nach  Verbrauch im Fenster")
for burn, win_h in [(14.4, 1), (6, 6), (3, 24), (1, 72)]:
    empty_h = period_h / burn
    used = burn * win_h / period_h
    print(f"{burn:>4}x {win_h:>5} h  {burn * budget:>8.2%}  {empty_h / 24:>9.1f} Tage"
          f"  {used:>12.1%}")
EOF
python3 budget.py

Expected:

Budget: 0.500% = 202 min Ausfall in 28 Tagen
bei 10 req/s: 120,960 fehlerhafte Anfragen
Burn  Fenster  Schwelle  Budget leer nach  Verbrauch im Fenster
14.4x     1 h     7.20%        1.9 Tage          2.1%
   6x     6 h     3.00%        4.7 Tage          5.4%
   3x    24 h     1.50%        9.3 Tage         10.7%
   1x    72 h     0.50%       28.0 Tage         10.7%

The factors 14.4/6/3/1 come from the Google SRE Workbook (chapter “Alerting on SLOs”) and are calculated for 30 days: 2 %, 5 % and 10 % of the budget. For 28 days the same calculation gives 13.44/5.6/2.8/0.93 – Drill 7 shows this with Sloth. Both are defensible; what matters is that the team and the customer know the same number.

Save your progress? Sign up for free

Drill 3: SLO rules – recording and burn-rate alerts

sed -n '1,13p' $OBS/rules/llm-gateway-slo.rules.yaml
docker run --rm -v $OBS/rules:/r -w /r --entrypoint promtool \
  prom/prometheus:v3.15.0 check rules llm-gateway-slo.rules.yaml

Expected:

# Tag 10: SLO "Verfügbarkeit 99,5 % / 28 Tage" für das llm-gateway, pro Mandant.
# SLI = Anteil 5xx an allen Anfragen ohne 4xx (401/429 sind Fehler des Clients).
# Fehlerbudget = 1 - 0.995 = 0.005. Burn Rate 1 = Budget ist nach 28 Tagen leer.
groups:
  - name: llm-gateway.slo.recording
    interval: 30s
    rules:
      - record: tenant:slo_errors:ratio_rate5m
        expr: |
          (sum by (tenant) (rate(gateway_requests_total{code=~"5.."}[5m]))
            or sum by (tenant) (rate(gateway_requests_total[5m])) * 0)
          / sum by (tenant) (rate(gateway_requests_total{code!~"4.."}[5m]))
      - record: tenant:slo_errors:ratio_rate30m
Checking llm-gateway-slo.rules.yaml
  SUCCESS: 7 rules found

Why or … * 0? A tenant without a single 5xx has no 5xx time series. Without the trick its ratio is missing entirely (Day 8, Drill 4) – dashboards show gaps instead of 0. Seven windows (5m to 3d) are precomputed once; otherwise every alert evaluation recalculates 3 days of raw data.

The alerts build on these windows:

sed -n '4,22p' $OBS/rules/llm-gateway-alerts.rules.yaml

Expected:

  - name: llm-gateway.slo.alerts
    rules:
      - alert: LLMGatewayHighErrorRate
        expr: |
          tenant:slo_errors:ratio_rate1h > (14.4 * 0.005)
          and tenant:slo_errors:ratio_rate5m > (14.4 * 0.005)
        for: 2m
        labels: { severity: page, long_window: 1h }
        annotations: &page
          summary: "llm-gateway: hohe 5xx-Quote für Mandant {{ $labels.tenant }}"
          description: >-
            {{ $value | humanizePercentage }} Fehler. Bei diesem Tempo ist das
            28-Tage-Fehlerbudget in wenigen Tagen aufgebraucht.
          runbook_url: >-
            https://runbooks.example.com/llm-gateway/LLMGatewayHighErrorRate
          dashboard_url: https://grafana.example.com/d/llm-gateway
      - alert: LLMGatewayHighErrorRate
        expr: |
          tenant:slo_errors:ratio_rate6h > (6 * 0.005)

Why two windows? The long window (1h) provides significance, the short one (5m) makes sure the alert resolves quickly after the fix. The file also contains LLMGatewayErrorBudgetBurn (3x/1x, ticket), LLMGatewayDown and LLMGatewayHighLatency (p95 > 20 s, ticket). &page/*page is a YAML anchor: both page rules share the same annotations.

Save your progress? Sign up for free

Drill 4: Unit-testing rules with promtool

The test file feeds three tenants with synthetic counters: acme 12.5 % errors (fast burn), globex 2 % (slow burn), initech 0 %.

sed -n '10,21p' $OBS/rules/llm-gateway.test.yaml
grep -c 'eval_time' $OBS/rules/llm-gateway.test.yaml
docker run --rm -v $OBS/rules:/r -w /r --entrypoint promtool \
  prom/prometheus:v3.15.0 test rules llm-gateway.test.yaml

Expected:

      # acme: 7 rps ok, 1 rps 5xx, 2 rps 429 -> SLI-Fehlerquote 1/8 = 12,5 %
      - series: 'gateway_requests_total{tenant="acme",code="200"}'
        values: '0+420x240'
      - series: 'gateway_requests_total{tenant="acme",code="502"}'
        values: '0+60x240'
      - series: 'gateway_requests_total{tenant="acme",code="429"}'
        values: '0+120x240'
      - series: 'gateway_ratelimited_total{tenant="acme"}'
        values: '0+120x240'
      # globex: 2 % Fehler -> langsamer Burn (nur Ticket)
      - series: 'gateway_requests_total{tenant="globex",code="200"}'
        values: '0+588x240'
10
  SUCCESS

Ten checks: six recording rule values, four alert points in time (fires, ticket instead of page, for not yet satisfied, gateway down). '0+420x240' means: start at 0, +420 per interval (1m), 240 times. Now break it on purpose – the threshold of the 1h window wrong by a factor of 10:

cp $OBS/rules/*.yaml .
sed -i.bak 's/ratio_rate1h > (14.4 \* 0.005)/ratio_rate1h > (14.4 * 0.05)/' \
  llm-gateway-alerts.rules.yaml
docker run --rm -v $PWD:/r -w /r --entrypoint promtool \
  prom/prometheus:v3.15.0 test rules llm-gateway.test.yaml 2>&1 | grep -v '^$' | head -4
docker run --rm -v $PWD:/r -w /r --entrypoint promtool \
  prom/prometheus:v3.15.0 test rules llm-gateway.test.yaml >/dev/null 2>&1
echo "exit=$?"

Expected (shortened: what follows is exp with both page alerts and got with only the 6h alert):

  FAILED:
    alertname: LLMGatewayHighErrorRate, time: 30m,
        exp:[
            0:
exit=1

No review reliably finds exactly this mistake (a factor of 10 in the budget) – the test does. In CI: promtool check rules and promtool test rules on every merge.

Save your progress? Sign up for free

Drill 5: Rolling out rules and routing

mk_rule() {  # Regeldatei -> PrometheusRule-Objekt (wie Tag 8)
  printf 'apiVersion: monitoring.coreos.com/v1\nkind: PrometheusRule\n'
  printf 'metadata:\n  name: %s\n  namespace: monitoring\nspec:\n' "$1"
  sed -e '/^#/d' -e 's/^/  /' "$2"
}
mk_rule llm-gateway-slo $OBS/rules/llm-gateway-slo.rules.yaml | kubectl apply -f -
mk_rule llm-gateway-alerts $OBS/rules/llm-gateway-alerts.rules.yaml \
  | kubectl apply -f -
am() { docker run --rm -v $OBS/alertmanager:/c --entrypoint amtool \
  prom/alertmanager:v0.34.1 "$@"; }
am check-config /c/alertmanager.yaml | head -1
am config routes show --config.file=/c/alertmanager.yaml
for l in "severity=page tenant=acme" "severity=ticket tenant=acme" \
         "severity=page tenant=globex" "severity=ticket tenant=globex"; do
  printf '%-30s -> ' "$l"; am config routes test --config.file=/c/alertmanager.yaml $l
done

Expected (amtool live, kubectl shortened):

prometheusrule.monitoring.coreos.com/llm-gateway-slo created
prometheusrule.monitoring.coreos.com/llm-gateway-alerts created
Checking '/c/alertmanager.yaml'  SUCCESS
Routing tree:
.
└── default-route  receiver: ticket
    ├── {alertname="Watchdog"}  receiver: deadmans-switch
    ├── {alertname="InfoInhibitor"}  receiver: null
    ├── {tenant="acme"}  continue: true  receiver: csm-acme
    ├── {severity="page"}  receiver: pager
    └── default-route  receiver: ticket
severity=page tenant=acme      -> csm-acme,pager
severity=ticket tenant=acme    -> csm-acme,ticket
severity=page tenant=globex    -> pager
severity=ticket tenant=globex  -> ticket

Why the empty route at the end? Once a route with continue: true has matched, Alertmanager no longer falls back to the root receiver. Without a catch-all, an acme ticket would only be seen by the CSM team. Inhibition: a page suppresses the ticket of the same tenant, and LLMGatewayDown is supposed to suppress all other gateway alerts – whether equal: [namespace] really does that is something you check on Day 15, Drill 5.

Now start a lab receiver and load the config via Helm:

cat > sink.py <<'EOF'
import http.server, json
class H(http.server.BaseHTTPRequestHandler):
    def do_POST(self):
        body = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
        for a in body["alerts"]:
            print(self.path, a["status"], a["labels"].get("alertname"),
                  a["labels"].get("tenant", "-"), flush=True)
        self.send_response(200)
        self.end_headers()
http.server.HTTPServer(("", 8080), H).serve_forever()
EOF
kubectl -n monitoring run alert-sink --image=python:3.13-alpine --port=8080 \
  --command -- python -u -c "$(cat sink.py)"
kubectl -n monitoring expose pod alert-sink --port=8080
helm upgrade kps prometheus-community/kube-prometheus-stack --version 92.2.0 \
  -n monitoring -f $OBS/helm/kps-values.yaml \
  -f $OBS/helm/grafana-datasources.yaml -f $OBS/helm/alertmanager-values.yaml --wait

alertmanager-values.yaml is alertmanager.yaml, indented under alertmanager.config. The chart itself appends templates: [/etc/alertmanager/config/*.tmpl].

Save your progress? Sign up for free

Drill 6: Firing, inhibiting and silencing alerts

kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-alertmanager \
  9093:9093 >/dev/null 2>&1 &
amk() {  # amtool im Alertmanager-Pod gegen die laufende Instanz
  kubectl -n monitoring exec alertmanager-kps-kube-prometheus-stack-alertmanager-0 \
    -c alertmanager -- amtool --alertmanager.url=http://localhost:9093 "$@"
}
amk alert add alertname=LLMGatewayHighErrorRate severity=page tenant=acme
amk alert add alertname=LLMGatewayErrorBudgetBurn severity=ticket tenant=acme
amk alert add alertname=LLMGatewayHighErrorRate severity=page tenant=globex
amk alert add alertname=LLMGatewayErrorBudgetBurn severity=ticket tenant=initech
sleep 40
kubectl -n monitoring logs alert-sink | grep '^/'
amk alert query --inhibited

Expected (live against Alertmanager 0.34.1):

/page firing LLMGatewayHighErrorRate acme
/csm-acme firing LLMGatewayHighErrorRate acme
/page firing LLMGatewayHighErrorRate globex
/ticket firing LLMGatewayErrorBudgetBurn initech
/heartbeat firing Watchdog -
Alertname                  Starts At                Summary  State
LLMGatewayErrorBudgetBurn  2026-10-09 16:38:58 UTC           suppressed

The acme ticket is inhibited because acme is already paging. Maintenance at globex? A silence with a ticket number and an expiry time:

ID=$(amk silence add alertname=LLMGatewayErrorBudgetBurn tenant=globex \
  --duration=2h --author=thomas --comment="Wartung globex, Ticket OPS-123")
amk silence query -o json \
  | jq -r '.[] | "\(.id[0:8]) \(.endsAt) \(.createdBy): \(.comment)"'
amk silence expire $ID
amk silence query -q | wc -l

Expected (live, ID and time differ):

9101b457 2026-10-09T19:21:04.331Z thomas: Wartung globex, Ticket OPS-123
       0

The real test: model server gone, load on, wait. After about 3–7 min the 1h/5m alert fires:

kubectl -n llm-system scale deploy/ollama --replicas=0   # Variante A, Tag 7
LLMCTL_API_KEY=lab-key-acme llmctl load -n 300 -c 2 >/dev/null
pq 'ALERTS{alertname="LLMGatewayHighErrorRate", alertstate="firing"}'
kubectl -n llm-system scale deploy/ollama --replicas=1

Expected (shortened, may differ by version):

{"alertname":"LLMGatewayHighErrorRate","alertstate":"firing","long_window":"1h",
"severity":"page","tenant":"acme"} 1

Save your progress? Sign up for free

Drill 7: Writing a runbook, knowing SLO generators

Template: meaning · impact · diagnosis (command + expectation per step) · remediation · escalation · follow-up. The runbook lives in Git next to the rules, and the runbook_url points to it.

mkdir -p runbooks && cat > runbooks/LLMGatewayHighErrorRate.md <<'EOF'
# Runbook: LLMGatewayHighErrorRate
Severity: page | Owner: Plattform-SRE | SLO: 99,5 % Verfügbarkeit / 28 Tage
Dashboard: https://grafana.example.com/d/llm-gateway

## Bedeutung
Die 5xx-Quote eines Mandanten verbrennt das Fehlerbudget mindestens 14,4-fach (1h)
oder 6-fach (6h). Label `tenant` = betroffener Mandant.

## Auswirkung
Kunden des Mandanten bekommen 502/503/504 auf /v1/chat/completions.

## Diagnose
1. Ein oder alle Mandanten?   pq 'tenant:slo_errors:ratio_rate5m'
   Alle betroffen -> Gateway/Modellserver. Einer -> Mandanten-Last, Quota.
2. Welche Codes?   pq 'sum by (code) (rate(gateway_requests_total{code=~"5.."}[5m]))'
   502 = Modellserver nicht erreichbar, 503 = überlastet, 504 = Timeout.
3. Pods und Endpoints:   kubectl -n llm-system get pods,endpoints -o wide
   Erwartet: alle Running/Ready, Endpoints für Gateway und Modellserver.
4. Letzte Änderung?   kubectl -n llm-system rollout history deploy/llm-gateway
   und Deploy-Markierung im Dashboard.
5. Logs -> Trace (Grafana Explore, Loki):
   {app="llm-gateway"} | json | status >= 500   -> "Trace öffnen" -> roter Span?
6. Modellserver: OOMKilled? GPU-Speicher (DCGM_FI_DEV_FB_USED),
   Warteschlange (vllm:num_requests_waiting).

## Abhilfe
- Deploy schuld:     kubectl -n llm-system rollout undo deploy/llm-gateway
- Überlast:          Modellserver skalieren, RATE_LIMIT_RPS des lauten Mandanten senken
- Pod hängt:         kubectl -n llm-system delete pod <pod>  (Controller startet neu)
- Prüfen:            pq 'tenant:slo_errors:ratio_rate5m' fällt unter 0.072

## Eskalation
- Nach 15 min ohne Besserung: Incident ausrufen (Tag 11), Incident Commander holen.
- Premium-Kunde (acme): CSM ist über die Route csm-acme schon informiert.

## Nachbereitung
- Silences nur mit Ticket-ID und Ablaufzeit.
- SEV1/SEV2: Post-Mortem (Tag 11). Dieses Runbook im selben PR anpassen.
EOF
grep -c '^## ' runbooks/LLMGatewayHighErrorRate.md

Expected:

6

Sloth and Pyrra generate the same rules from a short SLO specification. With Sloth (container, no installation):

cat > sloth.yaml <<'EOF'
version: "prometheus/v1"
service: "llm-gateway"
slos:
  - name: "availability"
    objective: 99.5
    sli:
      events:
        error_query: >-
          sum by (tenant) (rate(gateway_requests_total{code=~"5.."}[{{.window}}]))
        total_query: >-
          sum by (tenant) (rate(gateway_requests_total{code!~"4.."}[{{.window}}]))
    alerting:
      name: LLMGatewaySLOBurn
      page_alert: { labels: { severity: page } }
      ticket_alert: { labels: { severity: ticket } }
EOF
docker run --rm -v $PWD:/s ghcr.io/slok/sloth:v0.16.0 generate -i /s/sloth.yaml \
  --default-slo-period=28d 2>/dev/null > sloth-rules.yaml
grep -c 'record:' sloth-rules.yaml
grep -oE '> \([0-9.]+ \* 0.005\)' sloth-rules.yaml | sort -u

Expected:

15
> (0.9333333333333333 * 0.005)
> (13.44 * 0.005)
> (2.8000000000000003 * 0.005)
> (5.6000000000000005 * 0.005)

Same windows as yours, factors exact for 28 days. Pyrra does the same as a Kubernetes operator (CRD ServiceLevelObjective) with its own UI. Handwritten, you understand the rules – generated, they scale across hundreds of services.

Save your progress? Sign up for free

Drill 8: On-call hygiene

pq 'sort_desc(sum by (alertname, severity)
      (count_over_time(ALERTS{alertstate="firing"}[7d])) * 15 / 60)'
cat > handover.txt <<'EOF'
Übergabe On-Call  <Datum>  von <Name> an <Name>
Offene Incidents:      <ID, Status, nächster Schritt>
Aktive Silences:       <amtool silence query>  (alle mit Ticket + Ablauf?)
Laute Alerts (7 Tage): <Top 3 aus der Abfrage oben>  -> Ticket zum Fixen/Löschen
Toil diese Woche:      <manuelle Handgriffe, Minuten>  -> Automatisierungs-Ticket
Fehlerbudget:          <pq 'tenant:slo_errors:ratio_rate1d' je Mandant>
EOF

Expected (shortened, may differ by version):

{"alertname":"Watchdog","severity":"none"} 10080
{"alertname":"LLMGatewayHighErrorRate","severity":"page"} 37.5
{"alertname":"KubePodCrashLooping","severity":"warning"} 12

The value is the firing duration in minutes (evaluation every 15 s). Rules for the weekly alert review: every page must have triggered an action; otherwise it becomes a ticket or gets deleted. The target according to Google SRE: at most two incidents per 12-hour shift, toil below 50 % of working time.

Save your progress? Sign up for free

Checkpoint

#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
OBS=~/sre-lab/code/observability
check "promtool-Tests grün" "docker run --rm -v $OBS/rules:/r -w /r \
  --entrypoint promtool prom/prometheus:v3.15.0 test rules llm-gateway.test.yaml"
check "Alertmanager-Config gültig" "docker run --rm -v $OBS/alertmanager:/c \
  --entrypoint amtool prom/alertmanager:v0.34.1 check-config /c/alertmanager.yaml"
check "SLO-Regeln geladen" "curl -s localhost:9090/api/v1/rules \
  | jq -e '[.data.groups[].name] | index(\"llm-gateway.slo.alerts\")'"
check "Recording Rule liefert" "curl -s localhost:9090/api/v1/query \
  --data-urlencode 'query=tenant:slo_errors:ratio_rate1h' \
  | jq -e '.data.result | length > 0'"
check "Alertmanager erreichbar" "curl -sf localhost:9093/-/ready"
check "Runbook vollständig" \
  "[ \$(grep -c '^## ' ~/sre-lab/tag10/runbooks/LLMGatewayHighErrorRate.md) -eq 6 ]"

Challenge

Prove with a unit test that globex, at a constant 2 % errors, gets no page after 10 hours, but does get a ticket.

Hint: Your own test file with only globex series; exp_alerts: [] checks that an alert does not fire.

Solution

cat > globex.test.yaml <<'EOF'
rule_files:
  - llm-gateway-recording.rules.yaml
  - llm-gateway-slo.rules.yaml
  - llm-gateway-alerts.rules.yaml
evaluation_interval: 30s
tests:
  - interval: 1m
    input_series:
      - series: 'gateway_requests_total{tenant="globex",code="200"}'
        values: '0+588x600'
      - series: 'gateway_requests_total{tenant="globex",code="500"}'
        values: '0+12x600'
    alert_rule_test:
      - eval_time: 10h
        alertname: LLMGatewayHighErrorRate
        exp_alerts: []
      - eval_time: 10h
        alertname: LLMGatewayErrorBudgetBurn
        exp_alerts:
          - exp_labels: { severity: ticket, tenant: globex }
            exp_annotations:
              summary: "llm-gateway: Fehlerbudget von globex schmilzt"
              runbook_url: >-
                https://runbooks.example.com/llm-gateway/LLMGatewayErrorBudgetBurn
EOF
cp $OBS/rules/*.rules.yaml .
docker run --rm -v $PWD:/r -w /r --entrypoint promtool \
  prom/prometheus:v3.15.0 test rules globex.test.yaml

Save your progress? Sign up for free

Interview Note

  • Typical question: “How would you set up alerting for our inference API?” – SLIs from the customer’s point of view (availability without 4xx, TTFT), agree on the SLO with product, burn-rate alerts, page only on a fast burn.
  • Crisp: “I alert on symptoms at the customer, not on causes like CPU – causes go in the runbook.”
  • Do the math live: 99.5 % over 28 days is 202 minutes; burn rate 14.4 empties the budget in just under two days.
  • Bridge from your own practice: for regulated customers using Vault/OpenBao you wrote operations documentation and emergency manuals – runbooks with “command + expected result” are the same discipline.
  • Rules are code: promtool test rules in CI, an alert review every week, every page needs an action.

Enterprise Note

  • SLOs per customer by contract (the SLA is usually looser than the internal SLO), a monthly error budget report to the customer and product.
  • Real paging via PagerDuty, Opsgenie or Grafana IRM; Alertmanager in an HA cluster (3 replicas, gossip).
  • Dead man’s switch: the Watchdog alert must arrive externally (e.g. Healthchecks.io); otherwise nobody notices that alerting itself has failed.
  • On-prem at the customer: often their Alertmanager or ITSM (ServiceNow). Ship rules, a routing proposal and runbooks in the chart.
  • Follow-the-sun rotation across EMEA/US, clear escalation levels and rest periods after night pages.