Day 5: Status and Conditions
Goal: The site-operator reports via
status, Conditions and Events how the website is doing – readable for humans (kubectl get), scripts (kubectl wait) and monitoring · Duration: approx. 90 min · Prerequisite: Day 4
Briefing
specbelongs to the user,statusto the operator. The status is written via the/statussubresource.- Conditions are a list
{type, status, reason, message, observedGeneration, lastTransitionTime};meta.SetStatusConditionmaintains them and changeslastTransitionTimeonly on real transitions. observedGenerationsays: “I have seen this state of the spec.” Without it,Available=Truemay be stale.- Write the status only when it changes – otherwise every update wakes your own watch (hot loop).
Reconcile -> reconcileResources() --(dep, err)--> updateStatus()
updateStatus: Degraded error? ContentMissing / ReconcileError
Available readyReplicas >= spec.replicas
Progressing rollout done?
Events only on state transitions
Drill 1: Condition types as constants
New day, new copy. Types and reasons belong in the API package – tests, tools and other operators import them from there.
source ~/op-lab/env.sh
cp -r ~/op-lab/tag04-site-operator ~/op-lab/tag05-site-operator
cd ~/op-lab/tag05-site-operator
cat > api/v1alpha1/conditions.go <<'EOF'
package v1alpha1
// Condition-Typen einer StaticSite (Kubernetes-API-Konventionen).
const (
ConditionAvailable = "Available" // genug Pods bereit
ConditionProgressing = "Progressing" // Rollout läuft
ConditionDegraded = "Degraded" // Operator kann nicht abgleichen
)
// Reasons sind CamelCase, maschinenlesbar und stabil – Alerts hängen daran.
const (
ReasonAllReplicasReady = "AllReplicasReady"
ReasonReplicasNotReady = "ReplicasNotReady"
ReasonRolloutInProgress = "RolloutInProgress"
ReasonRolloutComplete = "RolloutComplete"
ReasonContentMissing = "ContentMissing"
ReasonReconcileError = "ReconcileError"
ReasonAsExpected = "AsExpected"
)
EOFReasons are a contract: alerts, runbooks and
kubectl wait scripts depend on them. Renaming one is a
breaking change.
Save your progress? Sign up for free
Drill 2: Printer columns
Markers above the StaticSite type control what
kubectl get shows. You insert them with sed
directly below the shortName marker from Day 3:
mkdir -p lab
cat > lab/columns.txt <<'EOF'
// +kubebuilder:printcolumn:name="Ready",type=integer,JSONPath=`.status.readyReplicas`
// +kubebuilder:printcolumn:name="Available",type=string,JSONPath=`.status.conditions[?(@.type=="Available")].status`
// +kubebuilder:printcolumn:name="URL",type=string,JSONPath=`.status.url`
// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`
EOF
sed -i.bak '/shortName=site/r lab/columns.txt' api/v1alpha1/staticsite_types.go
rm api/v1alpha1/staticsite_types.go.bak
make manifests generate
grep -B1 'name: Available' config/crd/bases/web.lab.example_staticsites.yamlExpected:
- jsonPath: .status.conditions[?(@.type=="Available")].status
name: Available
The JSONPath filter [?(@.type=="Available")] picks one
Condition out of the list – you’ll use the same syntax in scripts
shortly.
Warning: Markers must not be wrapped. That’s why the
Availableline is longer than usual – don’t split it by hand.
Save your progress? Sign up for free
Drill 3: Compute the status
updateStatus gets the Deployment and the error from the
reconciliation and derives everything from them. The comparison with
equality.Semantic.DeepEqual prevents writes without a
change.
cat > internal/controller/status.go <<'EOF'
package controller
import (
"context"
"errors"
"fmt"
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/equality"
"k8s.io/apimachinery/pkg/api/meta"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"sigs.k8s.io/controller-runtime/pkg/client"
webv1alpha1 "example.com/site-operator/api/v1alpha1"
)
// updateStatus leitet den Status aus Deployment und Reconcile-Fehler ab und
// schreibt ihn nur, wenn sich etwas geändert hat (sonst: Hotloop-Gefahr).
func (r *StaticSiteReconciler) updateStatus(ctx context.Context,
site *webv1alpha1.StaticSite, dep *appsv1.Deployment, recErr error) error {
orig := site.DeepCopy()
st := &site.Status
st.ObservedGeneration = site.Generation
st.URL = fmt.Sprintf("http://%s.%s.svc", site.Name, site.Namespace)
set := func(typ string, s metav1.ConditionStatus, reason, msg string) {
meta.SetStatusCondition(&st.Conditions, metav1.Condition{
Type: typ, Status: s, Reason: reason, Message: msg,
ObservedGeneration: site.Generation,
})
}
switch {
case errors.Is(recErr, errContentMissing):
set(webv1alpha1.ConditionDegraded, metav1.ConditionTrue,
webv1alpha1.ReasonContentMissing, recErr.Error())
case recErr != nil:
set(webv1alpha1.ConditionDegraded, metav1.ConditionTrue,
webv1alpha1.ReasonReconcileError, recErr.Error())
default:
set(webv1alpha1.ConditionDegraded, metav1.ConditionFalse,
webv1alpha1.ReasonAsExpected, "alle Objekte abgeglichen")
}
if dep == nil { // Deployment existiert (noch) nicht
st.ReadyReplicas = 0
set(webv1alpha1.ConditionAvailable, metav1.ConditionFalse,
webv1alpha1.ReasonReplicasNotReady, "kein Deployment")
} else {
setReplicaConditions(site, dep, set)
}
if equality.Semantic.DeepEqual(orig.Status, site.Status) {
return nil
}
r.recordTransitions(orig, site)
// MergeFrom: nur der Status-Diff geht an den API-Server.
return r.Status().Patch(ctx, site, client.MergeFrom(orig))
}
func setReplicaConditions(site *webv1alpha1.StaticSite, dep *appsv1.Deployment,
set func(string, metav1.ConditionStatus, string, string)) {
want, ds := site.Spec.Replicas, dep.Status
site.Status.ReadyReplicas = ds.ReadyReplicas
msg := fmt.Sprintf("%d/%d Pods bereit", ds.ReadyReplicas, want)
if ds.ReadyReplicas >= want {
set(webv1alpha1.ConditionAvailable, metav1.ConditionTrue,
webv1alpha1.ReasonAllReplicasReady, msg)
} else {
set(webv1alpha1.ConditionAvailable, metav1.ConditionFalse,
webv1alpha1.ReasonReplicasNotReady, msg)
}
// Rollout fertig: Deployment-Controller hat die aktuelle Generation gesehen
// und alle Pods sind aktualisiert und bereit.
done := ds.ObservedGeneration >= dep.Generation &&
ds.UpdatedReplicas >= want && ds.ReadyReplicas >= want
if done {
set(webv1alpha1.ConditionProgressing, metav1.ConditionFalse,
webv1alpha1.ReasonRolloutComplete, msg)
} else {
set(webv1alpha1.ConditionProgressing, metav1.ConditionTrue,
webv1alpha1.ReasonRolloutInProgress, msg)
}
}
// recordTransitions erzeugt Events nur bei Zustandswechseln, nicht bei jedem Lauf.
func (r *StaticSiteReconciler) recordTransitions(orig, site *webv1alpha1.StaticSite) {
if r.Recorder == nil { // z. B. im Test ohne Recorder
return
}
for _, typ := range []string{
webv1alpha1.ConditionAvailable, webv1alpha1.ConditionDegraded,
} {
before := meta.FindStatusCondition(orig.Status.Conditions, typ)
now := meta.FindStatusCondition(site.Status.Conditions, typ)
if now == nil || (before != nil && before.Status == now.Status) {
continue
}
evType := corev1.EventTypeNormal
if typ == webv1alpha1.ConditionDegraded && now.Status == metav1.ConditionTrue {
evType = corev1.EventTypeWarning
}
r.Recorder.Eventf(site, nil, evType, now.Reason, "Reconcile",
"%s=%s: %s", typ, now.Status, now.Message)
}
}
EOFWhy Status().Patch with MergeFrom? The
patch contains only the status diff and no resourceVersion
– no conflict if someone changes the spec in parallel. You go deeper
into optimistic locking and conflicts on Day 13.
Save your progress? Sign up for free
Drill 4: Restructure Reconcile
The reconciliation from Day 4 moves into
reconcileResources; Reconcile then
always writes the status – even when there’s an error.
The head of the file (up to just before resolveContent) is
replaced, the rest stays:
F=internal/controller/staticsite_controller.go
sed -n '/^\/\/ resolveContent liefert/,$p' $F > lab/rest.txt
cat > $F <<'EOF'
package controller
import (
"context"
"errors"
"fmt"
"time"
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/client-go/tools/events"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
logf "sigs.k8s.io/controller-runtime/pkg/log"
webv1alpha1 "example.com/site-operator/api/v1alpha1"
)
// errContentMissing: die referenzierte ConfigMap fehlt (noch).
var errContentMissing = errors.New("content source not found")
// StaticSiteReconciler gleicht StaticSites mit ConfigMap, Deployment und Service ab.
type StaticSiteReconciler struct {
client.Client
Scheme *runtime.Scheme
Recorder events.EventRecorder
}
// +kubebuilder:rbac:groups=web.lab.example,resources=staticsites,verbs=get;list;watch
// +kubebuilder:rbac:groups=web.lab.example,resources=staticsites,verbs=update;patch
// +kubebuilder:rbac:groups=web.lab.example,resources=staticsites/status,verbs=get;patch
// +kubebuilder:rbac:groups=web.lab.example,resources=staticsites/status,verbs=update
// +kubebuilder:rbac:groups=web.lab.example,resources=staticsites/finalizers,verbs=update
// +kubebuilder:rbac:groups="",resources=configmaps;services,verbs=get;list;watch
// +kubebuilder:rbac:groups="",resources=configmaps;services,verbs=create;update;patch
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=create;update;patch
// +kubebuilder:rbac:groups=events.k8s.io,resources=events,verbs=create;patch
func (r *StaticSiteReconciler) Reconcile(ctx context.Context,
req ctrl.Request) (ctrl.Result, error) {
var site webv1alpha1.StaticSite
if err := r.Get(ctx, req.NamespacedName, &site); err != nil {
return ctrl.Result{}, client.IgnoreNotFound(err) // gelöscht: nichts zu tun
}
dep, recErr := r.reconcileResources(ctx, &site)
// Status immer schreiben – gerade im Fehlerfall will der Nutzer wissen, warum.
if err := r.updateStatus(ctx, &site, dep, recErr); err != nil {
return ctrl.Result{}, errors.Join(recErr, err)
}
if errors.Is(recErr, errContentMissing) {
logf.FromContext(ctx).Info("Inhalt fehlt, neuer Versuch in 30s",
"error", recErr.Error())
return ctrl.Result{RequeueAfter: 30 * time.Second}, nil
}
return ctrl.Result{}, recErr // nil oder Backoff durch die Workqueue
}
// reconcileResources gleicht ConfigMap, Deployment und Service ab.
func (r *StaticSiteReconciler) reconcileResources(ctx context.Context,
site *webv1alpha1.StaticSite) (*appsv1.Deployment, error) {
content, err := r.resolveContent(ctx, site)
if err != nil {
return nil, err
}
hash, err := r.reconcileConfigMap(ctx, site, content)
if err != nil {
return nil, err
}
dep, err := r.reconcileDeployment(ctx, site, hash)
if err != nil {
return nil, err
}
return dep, r.reconcileService(ctx, site)
}
EOF
cat lab/rest.txt >> $F && rm lab/rest.txterrors.Join returns nil if both errors are
nil – and both if both go wrong. The recorder comes from
the manager; you wire it up in cmd/main.go with the same
sed technique:
printf '\t\tRecorder: mgr.GetEventRecorder("site-operator"),\n' > lab/recorder.txt
sed -i.bak '/Scheme: mgr.GetScheme(),/r lab/recorder.txt' cmd/main.go
rm cmd/main.go.bak
make manifests generate fmt vet
go build ./... && echo build-ok
grep -A2 'StaticSiteReconciler{' cmd/main.go
grep -A3 'events.k8s.io' config/rbac/role.yamlExpected (shortened):
build-ok
if err := (&controller.StaticSiteReconciler{
Client: mgr.GetClient(),
Scheme: mgr.GetScheme(),
- events.k8s.io
resources:
- events
verbs:
GetEventRecorder uses the new Events API
(events.k8s.io/v1); the older
GetEventRecorderFor is deprecated in controller-runtime
v0.25. Signature:
Eventf(regarding, related, typ, reason, action, note, args...).
Save your progress? Sign up for free
Drill 5: First run with status
The CRD has new printer columns, so you restart envtest in Terminal 2 (Ctrl+C, then):
cd ~/op-lab/tag05-site-operator
envtest-up -crds config/crd/basesTerminal 3 starts the operator:
cd ~/op-lab/tag05-site-operator
go run ./cmd/main.go -health-probe-bind-address=0 2>&1 | tee run.logIn Terminal 1:
kubectl apply -f lab/demo.yaml
sleep 2
kubectl get staticsite
kubectl get staticsite demo -o json | \
jq -c '.status.conditions[] | {type, status, reason}'
ev() { kubectl events --for "staticsite/$1" -o json | \
jq -r '.items[] | "\(.type) \(.reason): \(.message)"'; }
ev demoExpected:
staticsite.web.lab.example/demo created
NAME READY AVAILABLE URL AGE
demo False http://demo.default.svc 2s
{"type":"Degraded","status":"False","reason":"AsExpected"}
{"type":"Available","status":"False","reason":"ReplicasNotReady"}
{"type":"Progressing","status":"True","reason":"RolloutInProgress"}
Normal ReplicasNotReady: Available=False: 0/1 Pods bereit
Normal AsExpected: Degraded=False: alle Objekte abgeglichen
READY is empty because readyReplicas: 0 is
dropped by omitempty. kubectl events without
-o json often shows (x2 over 2s): the second
Reconcile read the StaticSite from the cache while it still had no
status – a harmless effect of the delayed informers.
Save your progress? Sign up for free
Drill 6: Simulate
readiness and kubectl wait
There are no Pods in envtest. You play the Deployment controller and patch the Deployment status via the subresource:
fake_ready() { # $1 = Deployment, $2 = bereite Pods (Default 1)
local n=${2:-1} gen
gen=$(kubectl get deployment "$1" -o jsonpath='{.metadata.generation}')
kubectl patch deployment "$1" --subresource=status --type=merge -p "{\"status\":{
\"observedGeneration\":$gen, \"replicas\":$n, \"updatedReplicas\":$n,
\"readyReplicas\":$n, \"availableReplicas\":$n}}"
}
kubectl wait staticsite/demo --for=condition=Available --timeout=3s
fake_ready demo
kubectl wait staticsite/demo --for=condition=Available --timeout=10s
kubectl get staticsite
ev demo | tail -1Expected:
error: timed out waiting for the condition on staticsites/demo
deployment.apps/demo patched
staticsite.web.lab.example/demo condition met
NAME READY AVAILABLE URL AGE
demo 1 True http://demo.default.svc 11s
Normal AllReplicasReady: Available=True: 1/1 Pods bereit
The status change on the Deployment wakes the StaticSite via
Owns(). kubectl wait --for=condition=… is
exactly what CI pipelines and Argo CD health checks wait for.
Save your progress? Sign up for free
Drill 7: observedGeneration
metadata.generation counts spec changes. Only once
status.observedGeneration catches up does the status
describe the current spec.
kubectl patch staticsite demo --type=merge -p '{"spec":{"replicas":2}}'
kubectl wait staticsite/demo --for=jsonpath='{.status.observedGeneration}'=2 \
--timeout=10s
kubectl get staticsite demo -o json | jq -c '.metadata.generation,
(.status.conditions[] | [.type, .status, .reason, .observedGeneration])'
fake_ready demo 2
kubectl wait staticsite/demo --for=condition=Progressing=False --timeout=10s
kubectl get staticsite demoExpected:
staticsite.web.lab.example/demo patched
staticsite.web.lab.example/demo condition met
2
["Degraded","False","AsExpected",2]
["Available","False","ReplicasNotReady",2]
["Progressing","True","RolloutInProgress",2]
deployment.apps/demo patched
staticsite.web.lab.example/demo condition met
NAME READY AVAILABLE URL AGE
demo 2 True http://demo.default.svc 16s
A script that only waits for Available=True would have
been wrongly satisfied right after the patch – the old status was still
True. That’s why you wait for
observedGeneration first, then for the Condition.
Save your progress? Sign up for free
Drill 8: Degraded in action
Now the error case from Day 4 – this time even someone who isn’t allowed to read the operator logs can see it:
kubectl apply -f lab/extern.yaml
kubectl wait staticsite/extern --for=condition=Degraded --timeout=10s
kubectl get staticsite extern -o json | jq -c '.status.conditions[]
| select(.type=="Degraded") | {reason, message}'
ev extern | grep Warning
kubectl create configmap extern-html \
--from-literal=index.html='<h1>Aus der ConfigMap</h1>'
kubectl wait staticsite/extern --for=condition=Degraded=False --timeout=40s
kubectl get staticsiteExpected (the second wait takes up to 30 s):
staticsite.web.lab.example/extern created
staticsite.web.lab.example/extern condition met
{"reason":"ContentMissing","message":"configmap extern-html: content source not found"}
Warning ContentMissing: Degraded=True: configmap extern-html: content source not found
configmap/extern-html created
staticsite.web.lab.example/extern condition met
NAME READY AVAILABLE URL AGE
demo 2 True http://demo.default.svc 50s
extern False http://extern.default.svc 30s
extern is no longer degraded, but not available yet
either – two independent statements. That is exactly why there are
several Condition types instead of a single phase
field.
Save your progress? Sign up for free
Checkpoint
The operator is running, demo and extern
exist:
cat > lab/check.sh <<'EOF'
ok() { echo "✔ $1"; }
nok() { echo "✘ $1"; }
cond() { kubectl get staticsite "$1" \
-o jsonpath="{.status.conditions[?(@.type==\"$2\")].status}"; }
[ "$(kubectl get crd staticsites.web.lab.example \
-o jsonpath='{.spec.versions[0].additionalPrinterColumns[*].name}')" \
= "Ready Available URL Age" ] && ok "Printer-Columns" || nok "Printer-Columns"
[ "$(cond demo Available)" = True ] && ok "demo Available=True" || nok "demo"
[ "$(kubectl get staticsite demo -o jsonpath='{.status.observedGeneration}')" \
= "$(kubectl get staticsite demo -o jsonpath='{.metadata.generation}')" ] \
&& ok "observedGeneration aktuell" || nok "observedGeneration"
[ "$(cond extern Degraded)" = False ] && ok "extern erholt" || nok "extern"
kubectl events --for staticsite/extern --types=Warning | grep -q ContentMissing \
&& ok "Warning-Event ContentMissing" || nok "Event fehlt"
rv=$(kubectl get staticsite demo -o jsonpath='{.metadata.resourceVersion}')
sleep 5
[ "$rv" = "$(kubectl get staticsite demo -o jsonpath='{.metadata.resourceVersion}')" ] \
&& ok "kein Status-Hotloop" || nok "Status ändert sich ständig"
EOF
bash lab/check.shExpected:
✔ Printer-Columns
✔ demo Available=True
✔ observedGeneration aktuell
✔ extern erholt
✔ Warning-Event ContentMissing
✔ kein Status-Hotloop
Challenge
Add a Reason column (the reason of the
Available Condition) that only appears with
kubectl get staticsite -o wide.
Hint: Printer columns have a priority
attribute; kubectl shows anything above 0 only in the wide view.
Solution
cat > lab/reason.txt <<'EOF'
// +kubebuilder:printcolumn:name="Reason",type=string,priority=1,JSONPath=`.status.conditions[?(@.type=="Available")].reason`
EOF
sed -i.bak '/printcolumn:name="URL"/r lab/reason.txt' api/v1alpha1/staticsite_types.go
rm api/v1alpha1/staticsite_types.go.bak
make manifests && make install
kubectl get staticsite -o wideExpected (shortened):
customresourcedefinition.apiextensions.k8s.io/staticsites.web.lab.example configured
NAME READY AVAILABLE URL REASON AGE
demo 2 True http://demo.default.svc AllReplicasReady 2m53s
extern False http://extern.default.svc ReplicasNotReady 2m9s
make install updates the CRD in the running envtest –
the operator doesn’t need a restart for that, only kubectl reads the new
schema. The code state in code/tag05-site-operator does not
include the challenge.
Save your progress? Sign up for free
Enterprise Note
- Stick to the Kubernetes API conventions for Conditions
(
metav1.Condition, positive polarity forAvailable,Degradedas “abnormal-true”); tools like kstatus, Argo CD and Flux evaluate them. - Reasons and Condition types are public API: use them in alerts
(
kube_customresource_…via kube-state-metrics CustomResourceState) and runbooks, never rename them silently. - Messages can contain customer data (paths, hostnames, error texts from third-party systems) – don’t write secrets or tokens into them, not in Events either.
- Events are short-lived (default TTL one hour) and not an audit log. For traceability: logs and metrics (Day 11).
- With many objects, batch status updates and only write on changes – every unnecessary status change is an etcd write and a watch event for all clients.