Skip to content
Hands-on IT IT Drills

Day 5: Status and Conditions

Goal: The site-operator reports via status, Conditions and Events how the website is doing – readable for humans (kubectl get), scripts (kubectl wait) and monitoring · Duration: approx. 90 min · Prerequisite: Day 4

Briefing

  • spec belongs to the user, status to the operator. The status is written via the /status subresource.
  • Conditions are a list {type, status, reason, message, observedGeneration, lastTransitionTime}; meta.SetStatusCondition maintains them and changes lastTransitionTime only on real transitions.
  • observedGeneration says: “I have seen this state of the spec.” Without it, Available=True may be stale.
  • Write the status only when it changes – otherwise every update wakes your own watch (hot loop).
Reconcile -> reconcileResources() --(dep, err)--> updateStatus()
  updateStatus: Degraded     error? ContentMissing / ReconcileError
                Available    readyReplicas >= spec.replicas
                Progressing  rollout done?
                Events       only on state transitions

Drill 1: Condition types as constants

New day, new copy. Types and reasons belong in the API package – tests, tools and other operators import them from there.

source ~/op-lab/env.sh
cp -r ~/op-lab/tag04-site-operator ~/op-lab/tag05-site-operator
cd ~/op-lab/tag05-site-operator
cat > api/v1alpha1/conditions.go <<'EOF'
package v1alpha1

// Condition-Typen einer StaticSite (Kubernetes-API-Konventionen).
const (
    ConditionAvailable   = "Available"   // genug Pods bereit
    ConditionProgressing = "Progressing" // Rollout läuft
    ConditionDegraded    = "Degraded"    // Operator kann nicht abgleichen
)

// Reasons sind CamelCase, maschinenlesbar und stabil – Alerts hängen daran.
const (
    ReasonAllReplicasReady  = "AllReplicasReady"
    ReasonReplicasNotReady  = "ReplicasNotReady"
    ReasonRolloutInProgress = "RolloutInProgress"
    ReasonRolloutComplete   = "RolloutComplete"
    ReasonContentMissing    = "ContentMissing"
    ReasonReconcileError    = "ReconcileError"
    ReasonAsExpected        = "AsExpected"
)
EOF

Reasons are a contract: alerts, runbooks and kubectl wait scripts depend on them. Renaming one is a breaking change.

Save your progress? Sign up for free

Drill 2: Printer columns

Markers above the StaticSite type control what kubectl get shows. You insert them with sed directly below the shortName marker from Day 3:

mkdir -p lab
cat > lab/columns.txt <<'EOF'
// +kubebuilder:printcolumn:name="Ready",type=integer,JSONPath=`.status.readyReplicas`
// +kubebuilder:printcolumn:name="Available",type=string,JSONPath=`.status.conditions[?(@.type=="Available")].status`
// +kubebuilder:printcolumn:name="URL",type=string,JSONPath=`.status.url`
// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`
EOF
sed -i.bak '/shortName=site/r lab/columns.txt' api/v1alpha1/staticsite_types.go
rm api/v1alpha1/staticsite_types.go.bak
make manifests generate
grep -B1 'name: Available' config/crd/bases/web.lab.example_staticsites.yaml

Expected:

    - jsonPath: .status.conditions[?(@.type=="Available")].status
      name: Available

The JSONPath filter [?(@.type=="Available")] picks one Condition out of the list – you’ll use the same syntax in scripts shortly.

Warning: Markers must not be wrapped. That’s why the Available line is longer than usual – don’t split it by hand.

Save your progress? Sign up for free

Drill 3: Compute the status

updateStatus gets the Deployment and the error from the reconciliation and derives everything from them. The comparison with equality.Semantic.DeepEqual prevents writes without a change.

cat > internal/controller/status.go <<'EOF'
package controller

import (
    "context"
    "errors"
    "fmt"

    appsv1 "k8s.io/api/apps/v1"
    corev1 "k8s.io/api/core/v1"
    "k8s.io/apimachinery/pkg/api/equality"
    "k8s.io/apimachinery/pkg/api/meta"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "sigs.k8s.io/controller-runtime/pkg/client"

    webv1alpha1 "example.com/site-operator/api/v1alpha1"
)

// updateStatus leitet den Status aus Deployment und Reconcile-Fehler ab und
// schreibt ihn nur, wenn sich etwas geändert hat (sonst: Hotloop-Gefahr).
func (r *StaticSiteReconciler) updateStatus(ctx context.Context,
    site *webv1alpha1.StaticSite, dep *appsv1.Deployment, recErr error) error {
    orig := site.DeepCopy()
    st := &site.Status
    st.ObservedGeneration = site.Generation
    st.URL = fmt.Sprintf("http://%s.%s.svc", site.Name, site.Namespace)

    set := func(typ string, s metav1.ConditionStatus, reason, msg string) {
        meta.SetStatusCondition(&st.Conditions, metav1.Condition{
            Type: typ, Status: s, Reason: reason, Message: msg,
            ObservedGeneration: site.Generation,
        })
    }

    switch {
    case errors.Is(recErr, errContentMissing):
        set(webv1alpha1.ConditionDegraded, metav1.ConditionTrue,
            webv1alpha1.ReasonContentMissing, recErr.Error())
    case recErr != nil:
        set(webv1alpha1.ConditionDegraded, metav1.ConditionTrue,
            webv1alpha1.ReasonReconcileError, recErr.Error())
    default:
        set(webv1alpha1.ConditionDegraded, metav1.ConditionFalse,
            webv1alpha1.ReasonAsExpected, "alle Objekte abgeglichen")
    }

    if dep == nil { // Deployment existiert (noch) nicht
        st.ReadyReplicas = 0
        set(webv1alpha1.ConditionAvailable, metav1.ConditionFalse,
            webv1alpha1.ReasonReplicasNotReady, "kein Deployment")
    } else {
        setReplicaConditions(site, dep, set)
    }

    if equality.Semantic.DeepEqual(orig.Status, site.Status) {
        return nil
    }
    r.recordTransitions(orig, site)
    // MergeFrom: nur der Status-Diff geht an den API-Server.
    return r.Status().Patch(ctx, site, client.MergeFrom(orig))
}

func setReplicaConditions(site *webv1alpha1.StaticSite, dep *appsv1.Deployment,
    set func(string, metav1.ConditionStatus, string, string)) {
    want, ds := site.Spec.Replicas, dep.Status
    site.Status.ReadyReplicas = ds.ReadyReplicas
    msg := fmt.Sprintf("%d/%d Pods bereit", ds.ReadyReplicas, want)
    if ds.ReadyReplicas >= want {
        set(webv1alpha1.ConditionAvailable, metav1.ConditionTrue,
            webv1alpha1.ReasonAllReplicasReady, msg)
    } else {
        set(webv1alpha1.ConditionAvailable, metav1.ConditionFalse,
            webv1alpha1.ReasonReplicasNotReady, msg)
    }
    // Rollout fertig: Deployment-Controller hat die aktuelle Generation gesehen
    // und alle Pods sind aktualisiert und bereit.
    done := ds.ObservedGeneration >= dep.Generation &&
        ds.UpdatedReplicas >= want && ds.ReadyReplicas >= want
    if done {
        set(webv1alpha1.ConditionProgressing, metav1.ConditionFalse,
            webv1alpha1.ReasonRolloutComplete, msg)
    } else {
        set(webv1alpha1.ConditionProgressing, metav1.ConditionTrue,
            webv1alpha1.ReasonRolloutInProgress, msg)
    }
}

// recordTransitions erzeugt Events nur bei Zustandswechseln, nicht bei jedem Lauf.
func (r *StaticSiteReconciler) recordTransitions(orig, site *webv1alpha1.StaticSite) {
    if r.Recorder == nil { // z. B. im Test ohne Recorder
        return
    }
    for _, typ := range []string{
        webv1alpha1.ConditionAvailable, webv1alpha1.ConditionDegraded,
    } {
        before := meta.FindStatusCondition(orig.Status.Conditions, typ)
        now := meta.FindStatusCondition(site.Status.Conditions, typ)
        if now == nil || (before != nil && before.Status == now.Status) {
            continue
        }
        evType := corev1.EventTypeNormal
        if typ == webv1alpha1.ConditionDegraded && now.Status == metav1.ConditionTrue {
            evType = corev1.EventTypeWarning
        }
        r.Recorder.Eventf(site, nil, evType, now.Reason, "Reconcile",
            "%s=%s: %s", typ, now.Status, now.Message)
    }
}
EOF

Why Status().Patch with MergeFrom? The patch contains only the status diff and no resourceVersion – no conflict if someone changes the spec in parallel. You go deeper into optimistic locking and conflicts on Day 13.

Save your progress? Sign up for free

Drill 4: Restructure Reconcile

The reconciliation from Day 4 moves into reconcileResources; Reconcile then always writes the status – even when there’s an error. The head of the file (up to just before resolveContent) is replaced, the rest stays:

F=internal/controller/staticsite_controller.go
sed -n '/^\/\/ resolveContent liefert/,$p' $F > lab/rest.txt
cat > $F <<'EOF'
package controller

import (
    "context"
    "errors"
    "fmt"
    "time"

    appsv1 "k8s.io/api/apps/v1"
    corev1 "k8s.io/api/core/v1"
    apierrors "k8s.io/apimachinery/pkg/api/errors"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/runtime"
    "k8s.io/client-go/tools/events"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
    logf "sigs.k8s.io/controller-runtime/pkg/log"

    webv1alpha1 "example.com/site-operator/api/v1alpha1"
)

// errContentMissing: die referenzierte ConfigMap fehlt (noch).
var errContentMissing = errors.New("content source not found")

// StaticSiteReconciler gleicht StaticSites mit ConfigMap, Deployment und Service ab.
type StaticSiteReconciler struct {
    client.Client
    Scheme   *runtime.Scheme
    Recorder events.EventRecorder
}

// +kubebuilder:rbac:groups=web.lab.example,resources=staticsites,verbs=get;list;watch
// +kubebuilder:rbac:groups=web.lab.example,resources=staticsites,verbs=update;patch
// +kubebuilder:rbac:groups=web.lab.example,resources=staticsites/status,verbs=get;patch
// +kubebuilder:rbac:groups=web.lab.example,resources=staticsites/status,verbs=update
// +kubebuilder:rbac:groups=web.lab.example,resources=staticsites/finalizers,verbs=update
// +kubebuilder:rbac:groups="",resources=configmaps;services,verbs=get;list;watch
// +kubebuilder:rbac:groups="",resources=configmaps;services,verbs=create;update;patch
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=create;update;patch
// +kubebuilder:rbac:groups=events.k8s.io,resources=events,verbs=create;patch

func (r *StaticSiteReconciler) Reconcile(ctx context.Context,
    req ctrl.Request) (ctrl.Result, error) {
    var site webv1alpha1.StaticSite
    if err := r.Get(ctx, req.NamespacedName, &site); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err) // gelöscht: nichts zu tun
    }

    dep, recErr := r.reconcileResources(ctx, &site)
    // Status immer schreiben – gerade im Fehlerfall will der Nutzer wissen, warum.
    if err := r.updateStatus(ctx, &site, dep, recErr); err != nil {
        return ctrl.Result{}, errors.Join(recErr, err)
    }
    if errors.Is(recErr, errContentMissing) {
        logf.FromContext(ctx).Info("Inhalt fehlt, neuer Versuch in 30s",
            "error", recErr.Error())
        return ctrl.Result{RequeueAfter: 30 * time.Second}, nil
    }
    return ctrl.Result{}, recErr // nil oder Backoff durch die Workqueue
}

// reconcileResources gleicht ConfigMap, Deployment und Service ab.
func (r *StaticSiteReconciler) reconcileResources(ctx context.Context,
    site *webv1alpha1.StaticSite) (*appsv1.Deployment, error) {
    content, err := r.resolveContent(ctx, site)
    if err != nil {
        return nil, err
    }
    hash, err := r.reconcileConfigMap(ctx, site, content)
    if err != nil {
        return nil, err
    }
    dep, err := r.reconcileDeployment(ctx, site, hash)
    if err != nil {
        return nil, err
    }
    return dep, r.reconcileService(ctx, site)
}

EOF
cat lab/rest.txt >> $F && rm lab/rest.txt

errors.Join returns nil if both errors are nil – and both if both go wrong. The recorder comes from the manager; you wire it up in cmd/main.go with the same sed technique:

printf '\t\tRecorder: mgr.GetEventRecorder("site-operator"),\n' > lab/recorder.txt
sed -i.bak '/Scheme: mgr.GetScheme(),/r lab/recorder.txt' cmd/main.go
rm cmd/main.go.bak
make manifests generate fmt vet
go build ./... && echo build-ok
grep -A2 'StaticSiteReconciler{' cmd/main.go
grep -A3 'events.k8s.io' config/rbac/role.yaml

Expected (shortened):

build-ok
    if err := (&controller.StaticSiteReconciler{
        Client:   mgr.GetClient(),
        Scheme:   mgr.GetScheme(),
  - events.k8s.io
  resources:
  - events
  verbs:

GetEventRecorder uses the new Events API (events.k8s.io/v1); the older GetEventRecorderFor is deprecated in controller-runtime v0.25. Signature: Eventf(regarding, related, typ, reason, action, note, args...).

Save your progress? Sign up for free

Drill 5: First run with status

The CRD has new printer columns, so you restart envtest in Terminal 2 (Ctrl+C, then):

cd ~/op-lab/tag05-site-operator
envtest-up -crds config/crd/bases

Terminal 3 starts the operator:

cd ~/op-lab/tag05-site-operator
go run ./cmd/main.go -health-probe-bind-address=0 2>&1 | tee run.log

In Terminal 1:

kubectl apply -f lab/demo.yaml
sleep 2
kubectl get staticsite
kubectl get staticsite demo -o json | \
  jq -c '.status.conditions[] | {type, status, reason}'
ev() { kubectl events --for "staticsite/$1" -o json | \
  jq -r '.items[] | "\(.type) \(.reason): \(.message)"'; }
ev demo

Expected:

staticsite.web.lab.example/demo created
NAME   READY   AVAILABLE   URL                       AGE
demo           False       http://demo.default.svc   2s
{"type":"Degraded","status":"False","reason":"AsExpected"}
{"type":"Available","status":"False","reason":"ReplicasNotReady"}
{"type":"Progressing","status":"True","reason":"RolloutInProgress"}
Normal ReplicasNotReady: Available=False: 0/1 Pods bereit
Normal AsExpected: Degraded=False: alle Objekte abgeglichen

READY is empty because readyReplicas: 0 is dropped by omitempty. kubectl events without -o json often shows (x2 over 2s): the second Reconcile read the StaticSite from the cache while it still had no status – a harmless effect of the delayed informers.

Save your progress? Sign up for free

Drill 6: Simulate readiness and kubectl wait

There are no Pods in envtest. You play the Deployment controller and patch the Deployment status via the subresource:

fake_ready() { # $1 = Deployment, $2 = bereite Pods (Default 1)
  local n=${2:-1} gen
  gen=$(kubectl get deployment "$1" -o jsonpath='{.metadata.generation}')
  kubectl patch deployment "$1" --subresource=status --type=merge -p "{\"status\":{
    \"observedGeneration\":$gen, \"replicas\":$n, \"updatedReplicas\":$n,
    \"readyReplicas\":$n, \"availableReplicas\":$n}}"
}
kubectl wait staticsite/demo --for=condition=Available --timeout=3s
fake_ready demo
kubectl wait staticsite/demo --for=condition=Available --timeout=10s
kubectl get staticsite
ev demo | tail -1

Expected:

error: timed out waiting for the condition on staticsites/demo
deployment.apps/demo patched
staticsite.web.lab.example/demo condition met
NAME   READY   AVAILABLE   URL                       AGE
demo   1       True        http://demo.default.svc   11s
Normal AllReplicasReady: Available=True: 1/1 Pods bereit

The status change on the Deployment wakes the StaticSite via Owns(). kubectl wait --for=condition=… is exactly what CI pipelines and Argo CD health checks wait for.

Save your progress? Sign up for free

Drill 7: observedGeneration

metadata.generation counts spec changes. Only once status.observedGeneration catches up does the status describe the current spec.

kubectl patch staticsite demo --type=merge -p '{"spec":{"replicas":2}}'
kubectl wait staticsite/demo --for=jsonpath='{.status.observedGeneration}'=2 \
  --timeout=10s
kubectl get staticsite demo -o json | jq -c '.metadata.generation,
  (.status.conditions[] | [.type, .status, .reason, .observedGeneration])'
fake_ready demo 2
kubectl wait staticsite/demo --for=condition=Progressing=False --timeout=10s
kubectl get staticsite demo

Expected:

staticsite.web.lab.example/demo patched
staticsite.web.lab.example/demo condition met
2
["Degraded","False","AsExpected",2]
["Available","False","ReplicasNotReady",2]
["Progressing","True","RolloutInProgress",2]
deployment.apps/demo patched
staticsite.web.lab.example/demo condition met
NAME   READY   AVAILABLE   URL                       AGE
demo   2       True        http://demo.default.svc   16s

A script that only waits for Available=True would have been wrongly satisfied right after the patch – the old status was still True. That’s why you wait for observedGeneration first, then for the Condition.

Save your progress? Sign up for free

Drill 8: Degraded in action

Now the error case from Day 4 – this time even someone who isn’t allowed to read the operator logs can see it:

kubectl apply -f lab/extern.yaml
kubectl wait staticsite/extern --for=condition=Degraded --timeout=10s
kubectl get staticsite extern -o json | jq -c '.status.conditions[]
  | select(.type=="Degraded") | {reason, message}'
ev extern | grep Warning
kubectl create configmap extern-html \
  --from-literal=index.html='<h1>Aus der ConfigMap</h1>'
kubectl wait staticsite/extern --for=condition=Degraded=False --timeout=40s
kubectl get staticsite

Expected (the second wait takes up to 30 s):

staticsite.web.lab.example/extern created
staticsite.web.lab.example/extern condition met
{"reason":"ContentMissing","message":"configmap extern-html: content source not found"}
Warning ContentMissing: Degraded=True: configmap extern-html: content source not found
configmap/extern-html created
staticsite.web.lab.example/extern condition met
NAME     READY   AVAILABLE   URL                         AGE
demo     2       True        http://demo.default.svc     50s
extern           False       http://extern.default.svc   30s

extern is no longer degraded, but not available yet either – two independent statements. That is exactly why there are several Condition types instead of a single phase field.

Save your progress? Sign up for free

Checkpoint

The operator is running, demo and extern exist:

cat > lab/check.sh <<'EOF'
ok()  { echo "✔ $1"; }
nok() { echo "✘ $1"; }
cond() { kubectl get staticsite "$1" \
  -o jsonpath="{.status.conditions[?(@.type==\"$2\")].status}"; }
[ "$(kubectl get crd staticsites.web.lab.example \
  -o jsonpath='{.spec.versions[0].additionalPrinterColumns[*].name}')" \
  = "Ready Available URL Age" ] && ok "Printer-Columns" || nok "Printer-Columns"
[ "$(cond demo Available)" = True ] && ok "demo Available=True" || nok "demo"
[ "$(kubectl get staticsite demo -o jsonpath='{.status.observedGeneration}')" \
  = "$(kubectl get staticsite demo -o jsonpath='{.metadata.generation}')" ] \
  && ok "observedGeneration aktuell" || nok "observedGeneration"
[ "$(cond extern Degraded)" = False ] && ok "extern erholt" || nok "extern"
kubectl events --for staticsite/extern --types=Warning | grep -q ContentMissing \
  && ok "Warning-Event ContentMissing" || nok "Event fehlt"
rv=$(kubectl get staticsite demo -o jsonpath='{.metadata.resourceVersion}')
sleep 5
[ "$rv" = "$(kubectl get staticsite demo -o jsonpath='{.metadata.resourceVersion}')" ] \
  && ok "kein Status-Hotloop" || nok "Status ändert sich ständig"
EOF
bash lab/check.sh

Expected:

✔ Printer-Columns
✔ demo Available=True
✔ observedGeneration aktuell
✔ extern erholt
✔ Warning-Event ContentMissing
✔ kein Status-Hotloop

Challenge

Add a Reason column (the reason of the Available Condition) that only appears with kubectl get staticsite -o wide.

Hint: Printer columns have a priority attribute; kubectl shows anything above 0 only in the wide view.

Solution

cat > lab/reason.txt <<'EOF'
// +kubebuilder:printcolumn:name="Reason",type=string,priority=1,JSONPath=`.status.conditions[?(@.type=="Available")].reason`
EOF
sed -i.bak '/printcolumn:name="URL"/r lab/reason.txt' api/v1alpha1/staticsite_types.go
rm api/v1alpha1/staticsite_types.go.bak
make manifests && make install
kubectl get staticsite -o wide

Expected (shortened):

customresourcedefinition.apiextensions.k8s.io/staticsites.web.lab.example configured
NAME     READY   AVAILABLE   URL                         REASON             AGE
demo     2       True        http://demo.default.svc     AllReplicasReady   2m53s
extern           False       http://extern.default.svc   ReplicasNotReady   2m9s

make install updates the CRD in the running envtest – the operator doesn’t need a restart for that, only kubectl reads the new schema. The code state in code/tag05-site-operator does not include the challenge.

Save your progress? Sign up for free

Enterprise Note

  • Stick to the Kubernetes API conventions for Conditions (metav1.Condition, positive polarity for Available, Degraded as “abnormal-true”); tools like kstatus, Argo CD and Flux evaluate them.
  • Reasons and Condition types are public API: use them in alerts (kube_customresource_… via kube-state-metrics CustomResourceState) and runbooks, never rename them silently.
  • Messages can contain customer data (paths, hostnames, error texts from third-party systems) – don’t write secrets or tokens into them, not in Events either.
  • Events are short-lived (default TTL one hour) and not an audit log. For traceability: logs and metrics (Day 11).
  • With many objects, batch status updates and only write on changes – every unnecessary status change is an etcd write and a watch event for all clients.