Skip to content
Hands-on IT IT Drills

Day 11: Notifications

Goal: Have Flux report what it does and what goes wrong, show the state of every commit on GitHub and let Flux reconcile immediately via webhook instead of waiting for the interval. · Duration: approx. 75 min · Prerequisite: Day 10

Briefing

The notification-controller works in two directions:

Direction Objects What for
out Provider + Alert send events to Slack, Teams, webhooks, GitHub …
in Receiver receive webhooks from GitHub, registries … and trigger Flux
  • A Provider is a destination: type (slack, msteams, generic, github …), address, credentials.
  • An Alert selects events (which objects, from which severity) and sends them to a provider. The severity is info (everything) or error (errors only).
  • A Receiver opens a secret URL. When a valid webhook arrives there, it triggers a reconciliation, for example of the GitRepository after a push.

There is no Slack in the lab. Instead, a small echo server runs that writes every request to its log. Whatever arrives there would arrive in Slack in exactly the same way.

 kustomize-controller ─── event ──► notification-controller ─ Alert ─► Provider echo
 helm-controller                                                   └─► Provider github
                                                                         (commit status)
 curl (instead of GitHub) ── POST /hook/… ──► Receiver ── triggers ──► GitRepository flux-system

Drill 1: A receiver

cd ~/flux-lab/fleet
mkdir -p infrastructure/echo
cd infrastructure/echo
printf 'apiVersion: v1\nkind: Namespace\nmetadata:\n  name: echo\n' > namespace.yaml
cat > deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo
  namespace: echo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo
  template:
    metadata:
      labels:
        app: echo
    spec:
      containers:
        - name: echo
          image: mendhak/http-https-echo:39
          ports:
            - containerPort: 8080
EOF
cat > service.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: echo
  namespace: echo
spec:
  selector:
    app: echo
  ports:
    - port: 80
      targetPort: 8080
EOF
cat > kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - namespace.yaml
  - deployment.yaml
  - service.yaml
EOF
cd ~/flux-lab/fleet
flux create kustomization echo --source=GitRepository/flux-system \
  --path=./infrastructure/echo --prune=true --interval=10m --wait=true \
  --export > clusters/lab/echo.yaml
git add -A
git commit -m "Webhook-Empfänger echo"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization echo
kubectl -n echo logs deploy/echo

Expected (shortened):

...
Listening on ports 8080 for http, and 8443 for https.

Save your progress? Sign up for free

Drill 2: Provider and Alert

F=clusters/lab/alerts.yaml
flux create alert-provider echo --type=generic \
  --address=http://echo.echo.svc.cluster.local/ --export > $F
flux create alert notes --provider-ref=echo --event-severity=info \
  --event-source="Kustomization/notes" --event-source="HelmRelease/*.notes-helm" \
  --export >> $F
cat $F

Expected:

---
apiVersion: notification.toolkit.fluxcd.io/v1
kind: Provider
metadata:
  name: echo
  namespace: flux-system
spec:
  address: http://echo.echo.svc.cluster.local/
  type: generic
---
apiVersion: notification.toolkit.fluxcd.io/v1
kind: Alert
metadata:
  name: notes
  namespace: flux-system
spec:
  eventSeverity: info
  eventSources:
  - kind: Kustomization
    name: notes
  - kind: HelmRelease
    name: '*'
    namespace: notes-helm
  providerRef:
    name: echo

HelmRelease/*.notes-helm means: all HelmReleases in the namespace notes-helm.

git add -A
git commit -m "Alerts an echo"
git push
flux reconcile kustomization flux-system --with-source
flux get alert-providers
flux get alerts

Expected (shortened):

NAME    READY   MESSAGE
echo    True    Provider is Ready
NAME    SUSPENDED   READY   MESSAGE
notes   False       True    Alert is Ready

Now something happens. The colleague with the kubectl scale is back:

kubectl -n notes scale deploy notes --replicas=4
flux reconcile kustomization notes
kubectl -n echo logs deploy/echo | grep -E '^        "(message|severity|reason)"'

Expected:

        "severity": "info",
        "message": "Deployment/notes/notes configured",
        "reason": "Progressing",
        "severity": "info",
        "message": "Reconciliation finished in 6.648211995s, next run in 5m0s",
        "reason": "ReconciliationSucceeded",

The echo server writes the entire request to the log. The grep picks three fields out of the JSON that Flux sent. The first message is the drift correction. That is exactly what you want to see in a team channel.

Save your progress? Sign up for free

Drill 3: Errors only

With info, every reconciliation arrives, every few minutes with ten Kustomizations. For a channel that humans read, error is the right threshold. For testing, you build in an error right away:

sed -i 's/  eventSeverity: info/  eventSeverity: error/' clusters/lab/alerts.yaml
sed -i 's/NOTES_REPLICAS: "2"/NOTES_REPLICAS: zwei/' clusters/lab/notes.yaml
git commit -am "Alerts nur bei Fehlern, notes kaputt"
git push
flux reconcile kustomization flux-system --with-source

A few seconds later:

kubectl -n echo logs deploy/echo --since=60s | grep -E '^        "(message|severity|reason)"'

Expected (shortened):

        "severity": "error",
        "message": "Deployment/notes/notes dry-run failed: failed to create typed patch object (notes/notes; apps/v1, Kind=Deployment): .spec.replicas: expected numeric (int or float), got string\n",
        "reason": "ReconciliationFailed",

The repair comes in the next drill, together with a new kind of notification.

Save your progress? Sign up for free

Drill 4: Commit status on GitHub

The github provider does not write text anywhere; instead it sets a status on the commit on GitHub: a green check mark or a red cross, as with CI. For this it needs a token with Commit statuses: Read and write. You already set that permission on Day 0.

kubectl -n flux-system create secret generic github-status \
  --from-literal=token=$GITHUB_TOKEN
flux create alert-provider github --type=github \
  --address=https://github.com/$GITHUB_USER/flux-fleet \
  --secret-ref=github-status --export >> clusters/lab/alerts.yaml
flux create alert github-status --provider-ref=github --event-severity=info \
  --event-source="Kustomization/*" --export >> clusters/lab/alerts.yaml
sed -i 's/NOTES_REPLICAS: zwei/NOTES_REPLICAS: "2"/' clusters/lab/notes.yaml
git commit -am "notes repariert, Commit-Status auf GitHub"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes
  • The token goes into the cluster via kubectl create secret, not via Git. With SOPS from Day 8, it could also be stored encrypted in the repo.
  • The status needs the info events, because “successful” is not an error message.

On GitHub, open the commit list of flux-fleet. The topmost commit has a green check mark. Clicking it shows which Kustomizations applied it. The API returns the same:

curl -s -H "Authorization: Bearer $GITHUB_TOKEN" \
  https://api.github.com/repos/$GITHUB_USER/flux-fleet/commits/$(git rev-parse HEAD)/statuses \
  | grep -E '"(state|context)"'

Expected (shortened, the hash in the context differs):

    "state": "success",
    "context": "kustomization/notes/39d0a9b7",
    "state": "success",
    "context": "kustomization/flux-system/39d0a9b7",

For a team this means: whoever merges a pull request sees on the commit whether the cluster has picked it up. With several clusters, the context also shows which one it was.

Save your progress? Sign up for free

Drill 5: A Receiver

It takes up to a minute (interval: 1m) for Flux to notice a push. A webhook from GitHub shortens that to seconds. GitHub signs every webhook with a shared secret, and the Receiver checks the signature.

The secret, without a trailing newline:

cd ~/flux-lab
head -c 24 /dev/urandom | base64 | tr -d '\n' > webhook.token
kubectl -n flux-system create secret generic webhook-token \
  --from-file=token=webhook.token

Tried it: Without tr -d '\n', the file ends with a newline, and that ends up in the Secret. The signature you are about to compute with $(cat webhook.token) does not have it. The Receiver then answers every webhook with 400, and the log of the notification-controller says payload signature check failed.

The Receiver itself and a NodePort so you can reach it from the VM:

cd fleet
F=clusters/lab/receiver.yaml
cat > $F <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: webhook-receiver-nodeport
  namespace: flux-system
spec:
  type: NodePort
  selector:
    app: notification-controller
  ports:
    - port: 80
      targetPort: http-webhook
      nodePort: 30900
EOF
flux create receiver github-fleet --type=github --event=ping --event=push \
  --secret-ref=webhook-token --resource=GitRepository/flux-system --export >> $F
git add -A
git commit -m "Receiver für GitHub-Webhooks"
git push
flux reconcile kustomization flux-system --with-source
flux get receivers

Expected (shortened, the path differs):

NAME            SUSPENDED   READY   MESSAGE
github-fleet    False       True    Receiver initialized for path: /hook/0b49313d55e70e12666dc1ef0f2f794caa74ad1e169dcdde61c7bed689e78b6f

The path is a hash of name, namespace and token. Anyone who does not know it cannot trigger reconciliations.

GitHub cannot reach your VM. You play GitHub yourself: a push message, signed the way GitHub does it, with HMAC-SHA256 over the content:

P=$(kubectl -n flux-system get receiver github-fleet -o jsonpath='{.status.webhookPath}')
BODY='{"ref":"refs/heads/main"}'
SIG=$(printf '%s' "$BODY" | openssl dgst -sha256 -hmac "$(cat ~/flux-lab/webhook.token)" \
  | cut -d' ' -f2)
curl -s -w '%{http_code}\n' -H "X-GitHub-Event: push" \
  -H "X-Hub-Signature-256: sha256=$SIG" -d "$BODY" localhost:30900$P

Expected:

200
kubectl -n flux-system logs deploy/notification-controller --since=1m \
  | grep receiver-server | tail -2

Expected (shortened):

{"level":"info",…,"msg":"handling GitHub event: push",…}
{"level":"info",…,"msg":"resource 'GitRepository/flux-system.flux-system' annotated",…}

annotated is the same thing flux reconcile does: the Receiver sets the annotation, and the source-controller fetches immediately.

With a wrong signature:

curl -s -w '%{http_code}\n' -H "X-GitHub-Event: push" \
  -H "X-Hub-Signature-256: sha256=falsch" -d "$BODY" localhost:30900$P

Expected:

400

On GitHub you set this up under flux-fleet → Settings → Webhooks → Add webhook: Payload URL https://<public address>/hook/…, content type application/json, Secret = content of webhook.token, event “Just the push event”. In operations, an ingress with TLS sits in front of it.

Save your progress? Sign up for free

Checkpoint

flux get alerts

Expected: notes and github-status, both READY True.

flux get receivers

Expected: github-fleet, READY True.

diff ~/flux-lab/fleet/clusters/lab/receiver.yaml \
  ~/flux-lab/code/loesung/tag11/clusters/lab/receiver.yaml

Expected: no output.

Challenge

The team wants to see in its channel when fluxcdbot commits a new version, but still no ordinary reconciliations. Set up an alert and trigger an update.

Hint: The ImageUpdateAutomation reports a push as info. A separate alert with its own threshold does not interfere with the alert notes.

Solution

flux create alert image-updates --provider-ref=echo --event-severity=info \
  --event-source="ImageUpdateAutomation/*" --export >> clusters/lab/alerts.yaml
sed -i 's/NOTES_VERSION: 0.7.1/NOTES_VERSION: 0.7.0/' clusters/lab/cluster-vars.yaml
git commit -am "Alert für Image-Updates"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile image update notes
kubectl -n echo logs deploy/echo --since=60s | grep -E '^        "(message|reason)"'

Expected (shortened):

        "message": "pushed commit '161c323' to branch 'flux/notes-updates'\n0.7.0 -> 0.7.1\n",
        "reason": "Succeeded",

Then “merge” the pull request and delete the branch, as on Day 10:

git fetch -p
git merge --no-edit origin/flux/notes-updates
git push
git push origin --delete flux/notes-updates
flux reconcile kustomization flux-system --with-source

Save your progress? Sign up for free

Enterprise Note

  • Alert fatigue: one channel per team, only error, and for drift a separate alert with inclusionList or exclusionList, which filter messages by text. Anyone who sees every reconciliation soon reads none at all.
  • Monitoring instead of chat: for on-call duty, metrics work better than messages. With kube-state-metrics and the configuration from the flux2-monitoring-example example, you get the metric gotk_resource_info with the ready state of every Flux object (Day 13), and Alertmanager takes over the escalation.
  • Making the Receiver public: the Receiver is the only part of Flux that has to be reachable from outside. It belongs behind an ingress with TLS and a rate limit, and the NetworkPolicy allow-webhooks from the Flux installation only lets requests through to this port anyway.