Day 11: Notifications
Goal: Have Flux report what it does and what goes wrong, show the state of every commit on GitHub and let Flux reconcile immediately via webhook instead of waiting for the interval. · Duration: approx. 75 min · Prerequisite: Day 10
Briefing
The notification-controller works in two directions:
| Direction | Objects | What for |
|---|---|---|
| out | Provider + Alert |
send events to Slack, Teams, webhooks, GitHub … |
| in | Receiver |
receive webhooks from GitHub, registries … and trigger Flux |
- A Provider is a destination: type
(
slack,msteams,generic,github…), address, credentials. - An Alert selects events (which objects, from which
severity) and sends them to a provider. The severity is
info(everything) orerror(errors only). - A Receiver opens a secret URL. When a valid webhook arrives there, it triggers a reconciliation, for example of the GitRepository after a push.
There is no Slack in the lab. Instead, a small echo server runs that writes every request to its log. Whatever arrives there would arrive in Slack in exactly the same way.
kustomize-controller ─── event ──► notification-controller ─ Alert ─► Provider echo
helm-controller └─► Provider github
(commit status)
curl (instead of GitHub) ── POST /hook/… ──► Receiver ── triggers ──► GitRepository flux-system
Drill 1: A receiver
cd ~/flux-lab/fleet
mkdir -p infrastructure/echo
cd infrastructure/echo
printf 'apiVersion: v1\nkind: Namespace\nmetadata:\n name: echo\n' > namespace.yaml
cat > deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: echo
namespace: echo
spec:
replicas: 1
selector:
matchLabels:
app: echo
template:
metadata:
labels:
app: echo
spec:
containers:
- name: echo
image: mendhak/http-https-echo:39
ports:
- containerPort: 8080
EOF
cat > service.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
name: echo
namespace: echo
spec:
selector:
app: echo
ports:
- port: 80
targetPort: 8080
EOF
cat > kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- deployment.yaml
- service.yaml
EOF
cd ~/flux-lab/fleet
flux create kustomization echo --source=GitRepository/flux-system \
--path=./infrastructure/echo --prune=true --interval=10m --wait=true \
--export > clusters/lab/echo.yaml
git add -A
git commit -m "Webhook-Empfänger echo"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization echo
kubectl -n echo logs deploy/echoExpected (shortened):
...
Listening on ports 8080 for http, and 8443 for https.
Save your progress? Sign up for free
Drill 2: Provider and Alert
F=clusters/lab/alerts.yaml
flux create alert-provider echo --type=generic \
--address=http://echo.echo.svc.cluster.local/ --export > $F
flux create alert notes --provider-ref=echo --event-severity=info \
--event-source="Kustomization/notes" --event-source="HelmRelease/*.notes-helm" \
--export >> $F
cat $FExpected:
---
apiVersion: notification.toolkit.fluxcd.io/v1
kind: Provider
metadata:
name: echo
namespace: flux-system
spec:
address: http://echo.echo.svc.cluster.local/
type: generic
---
apiVersion: notification.toolkit.fluxcd.io/v1
kind: Alert
metadata:
name: notes
namespace: flux-system
spec:
eventSeverity: info
eventSources:
- kind: Kustomization
name: notes
- kind: HelmRelease
name: '*'
namespace: notes-helm
providerRef:
name: echo
HelmRelease/*.notes-helm means: all HelmReleases in the
namespace notes-helm.
git add -A
git commit -m "Alerts an echo"
git push
flux reconcile kustomization flux-system --with-source
flux get alert-providers
flux get alertsExpected (shortened):
NAME READY MESSAGE
echo True Provider is Ready
NAME SUSPENDED READY MESSAGE
notes False True Alert is Ready
Now something happens. The colleague with the
kubectl scale is back:
kubectl -n notes scale deploy notes --replicas=4
flux reconcile kustomization notes
kubectl -n echo logs deploy/echo | grep -E '^ "(message|severity|reason)"'Expected:
"severity": "info",
"message": "Deployment/notes/notes configured",
"reason": "Progressing",
"severity": "info",
"message": "Reconciliation finished in 6.648211995s, next run in 5m0s",
"reason": "ReconciliationSucceeded",
The echo server writes the entire request to the log. The
grep picks three fields out of the JSON that Flux sent. The
first message is the drift correction. That is exactly what you want to
see in a team channel.
Save your progress? Sign up for free
Drill 3: Errors only
With info, every reconciliation arrives, every few
minutes with ten Kustomizations. For a channel that humans read,
error is the right threshold. For testing, you build in an
error right away:
sed -i 's/ eventSeverity: info/ eventSeverity: error/' clusters/lab/alerts.yaml
sed -i 's/NOTES_REPLICAS: "2"/NOTES_REPLICAS: zwei/' clusters/lab/notes.yaml
git commit -am "Alerts nur bei Fehlern, notes kaputt"
git push
flux reconcile kustomization flux-system --with-sourceA few seconds later:
kubectl -n echo logs deploy/echo --since=60s | grep -E '^ "(message|severity|reason)"'Expected (shortened):
"severity": "error",
"message": "Deployment/notes/notes dry-run failed: failed to create typed patch object (notes/notes; apps/v1, Kind=Deployment): .spec.replicas: expected numeric (int or float), got string\n",
"reason": "ReconciliationFailed",
The repair comes in the next drill, together with a new kind of notification.
Save your progress? Sign up for free
Drill 4: Commit status on GitHub
The github provider does not write text anywhere;
instead it sets a status on the commit on GitHub: a green check mark or
a red cross, as with CI. For this it needs a token with Commit
statuses: Read and write. You already set that permission on
Day 0.
kubectl -n flux-system create secret generic github-status \
--from-literal=token=$GITHUB_TOKEN
flux create alert-provider github --type=github \
--address=https://github.com/$GITHUB_USER/flux-fleet \
--secret-ref=github-status --export >> clusters/lab/alerts.yaml
flux create alert github-status --provider-ref=github --event-severity=info \
--event-source="Kustomization/*" --export >> clusters/lab/alerts.yaml
sed -i 's/NOTES_REPLICAS: zwei/NOTES_REPLICAS: "2"/' clusters/lab/notes.yaml
git commit -am "notes repariert, Commit-Status auf GitHub"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes- The token goes into the cluster via
kubectl create secret, not via Git. With SOPS from Day 8, it could also be stored encrypted in the repo. - The status needs the
infoevents, because “successful” is not an error message.
On GitHub, open the commit list of flux-fleet. The
topmost commit has a green check mark. Clicking it shows which
Kustomizations applied it. The API returns the same:
curl -s -H "Authorization: Bearer $GITHUB_TOKEN" \
https://api.github.com/repos/$GITHUB_USER/flux-fleet/commits/$(git rev-parse HEAD)/statuses \
| grep -E '"(state|context)"'Expected (shortened, the hash in the context differs):
"state": "success",
"context": "kustomization/notes/39d0a9b7",
"state": "success",
"context": "kustomization/flux-system/39d0a9b7",
For a team this means: whoever merges a pull request sees on the commit whether the cluster has picked it up. With several clusters, the context also shows which one it was.
Save your progress? Sign up for free
Drill 5: A Receiver
It takes up to a minute (interval: 1m) for Flux to
notice a push. A webhook from GitHub shortens that to seconds. GitHub
signs every webhook with a shared secret, and the Receiver checks the
signature.
The secret, without a trailing newline:
cd ~/flux-lab
head -c 24 /dev/urandom | base64 | tr -d '\n' > webhook.token
kubectl -n flux-system create secret generic webhook-token \
--from-file=token=webhook.tokenTried it: Without
tr -d '\n', the file ends with a newline, and that ends up in the Secret. The signature you are about to compute with$(cat webhook.token)does not have it. The Receiver then answers every webhook with400, and the log of the notification-controller sayspayload signature check failed.
The Receiver itself and a NodePort so you can reach it from the VM:
cd fleet
F=clusters/lab/receiver.yaml
cat > $F <<'EOF'
apiVersion: v1
kind: Service
metadata:
name: webhook-receiver-nodeport
namespace: flux-system
spec:
type: NodePort
selector:
app: notification-controller
ports:
- port: 80
targetPort: http-webhook
nodePort: 30900
EOF
flux create receiver github-fleet --type=github --event=ping --event=push \
--secret-ref=webhook-token --resource=GitRepository/flux-system --export >> $F
git add -A
git commit -m "Receiver für GitHub-Webhooks"
git push
flux reconcile kustomization flux-system --with-source
flux get receiversExpected (shortened, the path differs):
NAME SUSPENDED READY MESSAGE
github-fleet False True Receiver initialized for path: /hook/0b49313d55e70e12666dc1ef0f2f794caa74ad1e169dcdde61c7bed689e78b6f
The path is a hash of name, namespace and token. Anyone who does not know it cannot trigger reconciliations.
GitHub cannot reach your VM. You play GitHub yourself: a push message, signed the way GitHub does it, with HMAC-SHA256 over the content:
P=$(kubectl -n flux-system get receiver github-fleet -o jsonpath='{.status.webhookPath}')
BODY='{"ref":"refs/heads/main"}'
SIG=$(printf '%s' "$BODY" | openssl dgst -sha256 -hmac "$(cat ~/flux-lab/webhook.token)" \
| cut -d' ' -f2)
curl -s -w '%{http_code}\n' -H "X-GitHub-Event: push" \
-H "X-Hub-Signature-256: sha256=$SIG" -d "$BODY" localhost:30900$PExpected:
200
kubectl -n flux-system logs deploy/notification-controller --since=1m \
| grep receiver-server | tail -2Expected (shortened):
{"level":"info",…,"msg":"handling GitHub event: push",…}
{"level":"info",…,"msg":"resource 'GitRepository/flux-system.flux-system' annotated",…}
annotated is the same thing flux reconcile
does: the Receiver sets the annotation, and the source-controller
fetches immediately.
With a wrong signature:
curl -s -w '%{http_code}\n' -H "X-GitHub-Event: push" \
-H "X-Hub-Signature-256: sha256=falsch" -d "$BODY" localhost:30900$PExpected:
400
On GitHub you set this up under flux-fleet → Settings →
Webhooks → Add webhook: Payload URL
https://<public address>/hook/…, content type
application/json, Secret = content of
webhook.token, event “Just the push event”. In operations,
an ingress with TLS sits in front of it.
Save your progress? Sign up for free
Checkpoint
flux get alertsExpected: notes and github-status, both
READY True.
flux get receiversExpected: github-fleet, READY
True.
diff ~/flux-lab/fleet/clusters/lab/receiver.yaml \
~/flux-lab/code/loesung/tag11/clusters/lab/receiver.yamlExpected: no output.
Challenge
The team wants to see in its channel when fluxcdbot
commits a new version, but still no ordinary reconciliations. Set up an
alert and trigger an update.
Hint: The ImageUpdateAutomation reports a push as
info. A separate alert with its own threshold does not
interfere with the alert notes.
Solution
flux create alert image-updates --provider-ref=echo --event-severity=info \
--event-source="ImageUpdateAutomation/*" --export >> clusters/lab/alerts.yaml
sed -i 's/NOTES_VERSION: 0.7.1/NOTES_VERSION: 0.7.0/' clusters/lab/cluster-vars.yaml
git commit -am "Alert für Image-Updates"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile image update notes
kubectl -n echo logs deploy/echo --since=60s | grep -E '^ "(message|reason)"'Expected (shortened):
"message": "pushed commit '161c323' to branch 'flux/notes-updates'\n0.7.0 -> 0.7.1\n",
"reason": "Succeeded",
Then “merge” the pull request and delete the branch, as on Day 10:
git fetch -p
git merge --no-edit origin/flux/notes-updates
git push
git push origin --delete flux/notes-updates
flux reconcile kustomization flux-system --with-sourceSave your progress? Sign up for free
Enterprise Note
- Alert fatigue: one channel per team, only
error, and for drift a separate alert withinclusionListorexclusionList, which filter messages by text. Anyone who sees every reconciliation soon reads none at all. - Monitoring instead of chat: for on-call duty,
metrics work better than messages. With kube-state-metrics and the
configuration from the
flux2-monitoring-exampleexample, you get the metricgotk_resource_infowith thereadystate of every Flux object (Day 13), and Alertmanager takes over the escalation. - Making the Receiver public: the Receiver is the
only part of Flux that has to be reachable from outside. It belongs
behind an ingress with TLS and a rate limit, and the NetworkPolicy
allow-webhooksfrom the Flux installation only lets requests through to this port anyway.