Skip to content
Hands-on IT IT Drills

Day 11: RBAC, Leader Election, Metrics, Health

Goal: The site-operator runs with exactly the permissions from its RBAC markers, elects a leader, delivers protected Prometheus metrics (including its own), reports health and readiness and logs in a structured way · Duration: approx. 150 min · Prerequisite: Day 10

Briefing

  • RBAC comes from +kubebuilder:rbac markers: make manifests writes config/rbac/role.yaml. If a marker is missing, you only notice it with a real ServiceAccount.
  • Leader election: several replicas, but only one reconciles. Whoever holds the Lease is the leader.
  • The manager serves metrics at /metrics – over HTTPS, with token checking (TokenReview) and RBAC checking (SubjectAccessReview).
  • /healthz (liveness) and /readyz (readiness) on port 8081, your own checks via AddHealthzCheck/AddReadyzCheck.
  • Logs are structured (zap/logr): key-value pairs instead of free text, as JSON in production.
           ServiceAccount site-operator-controller-manager
                    | ClusterRole manager-role (from markers)
envtest <-----------+ Role leader-election (Lease)
   ^  TokenReview/SAR          manager :8443 /metrics  <-- curl + token
   '---------------------------        :8081 /healthz /readyz

Drill 1: RBAC from markers

source ~/op-lab/env.sh
cp -r ~/op-lab/tag10-site-operator ~/op-lab/tag11-site-operator
cd ~/op-lab/tag11-site-operator
grep -h 'kubebuilder:rbac' internal/controller/*.go | cut -c1-80

Expected (shortened):

// +kubebuilder:rbac:groups=web.lab.example,resources=staticsites,verbs=get;list;wa
// +kubebuilder:rbac:groups=web.lab.example,resources=staticsites/status,verbs=get;
// +kubebuilder:rbac:groups="",resources=configmaps;services,verbs=get;list;watch
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch
// +kubebuilder:rbac:groups=events.k8s.io,resources=events,verbs=create;patch

controller-gen collects all markers in the package and merges them into one ClusterRole manager-role:

make manifests
grep -A3 'apps' config/rbac/role.yaml

Expected:

  - apps
  resources:
  - deployments
  verbs:

Next to role.yaml, config/rbac/ holds the bindings, the ServiceAccount, the role for the leader lease and the roles for metrics protection. config/default puts the namespace site-operator-system and the prefix site-operator- in front. For the lab you build a small overlay that creates only these RBAC objects – with exactly the names the real Deployment will use later:

mkdir -p config/lab-rbac
cat > config/lab-rbac/kustomization.yaml <<'EOF'
# Lab only: RBAC from config/rbac with the same names as config/default
namespace: site-operator-system
namePrefix: site-operator-
resources:
- namespace.yaml
- ../rbac
EOF
cat > config/lab-rbac/namespace.yaml <<'EOF'
apiVersion: v1
kind: Namespace
metadata:
  name: system
EOF

name: system is not a typo: the prefix and the namespace transformer turn it into site-operator-system, just like in config/default.

Save your progress? Sign up for free

Drill 2: The operator as a ServiceAccount

So far the manager ran with your admin kubeconfig (system:masters) – it was allowed to do everything, and a missing marker never showed up. Now it gets the identity it would have in the cluster.

Terminal 1: start envtest with the CRD (runs until the end of the day):

source ~/op-lab/env.sh
cd ~/op-lab/tag11-site-operator && envtest-up --crds config/crd/bases

Terminal 2 (working terminal; today you start the operator in the background because you restart it often – Terminal 3 stays free):

kubectl apply -k config/lab-rbac | grep -E 'namespace/|serviceaccount/|manager-role '
SA=system:serviceaccount:site-operator-system:site-operator-controller-manager
kubectl auth can-i list deployments --as=$SA
kubectl auth can-i delete deployments --as=$SA
kubectl auth can-i get secrets --as=$SA
kubectl auth can-i create leases -n site-operator-system --as=$SA

Expected:

namespace/site-operator-system created
serviceaccount/site-operator-controller-manager created
clusterrole.rbac.authorization.k8s.io/site-operator-manager-role created
yes
no
no
yes

delete is missing on purpose: the garbage collector cleans up children via the owner reference, the operator itself deletes nothing. The lease role applies only in its own namespace. Now a kubeconfig with the ServiceAccount’s token:

cp ~/op-lab/envtest.kubeconfig ~/op-lab/sa.kubeconfig
TOKEN=$(kubectl create token site-operator-controller-manager \
  -n site-operator-system --duration=4h)
export KUBECONFIG=~/op-lab/sa.kubeconfig
kubectl config set-credentials operator --token="$TOKEN" >/dev/null
kubectl config set-context operator --cluster=envtest --user=operator >/dev/null
kubectl config use-context operator
kubectl auth whoami | head -2 | cut -c1-80
kubectl get secrets 2>&1 | cut -c1-80
export KUBECONFIG=~/op-lab/envtest.kubeconfig

Expected:

Switched to context "operator".
ATTRIBUTE                                           VALUE
Username                                            system:serviceaccount:site-operat
Error from server (Forbidden): secrets is forbidden: User "system:serviceaccount

Security: The token is a real lab credential with a lifetime of 4 hours. sa.kubeconfig sits next to envtest.kubeconfig and is valid only for envtest; when it stops, envtest-up deletes only its own file – you clean up sa.kubeconfig yourself at the end.

Today you build the manager as a binary – two instances and kill work more reliably that way than with go run. Webhooks stay off, as on Day 10:

export ENABLE_WEBHOOKS=false
make build
bin/manager --kubeconfig ~/op-lab/sa.kubeconfig -health-probe-bind-address=:8081 \
  > lab/run.log 2>&1 &
echo $! > lab/manager.pid
cat > lab/site.yaml <<'EOF'
apiVersion: web.lab.example/v1
kind: StaticSite
metadata:
  name: demo
spec:
  content: "<h1>Hallo vom site-operator</h1>"
EOF
kubectl apply -f lab/site.yaml
sleep 3; kubectl get site demo; grep -c ERROR lab/run.log

Expected:

staticsite.web.lab.example/demo created
NAME   READY   AVAILABLE   URL                       AGE
demo           False       http://demo.default.svc   3s
0

controller-runtime registers --kubeconfig itself; ctrl.GetConfigOrDie() reads the flag before KUBECONFIG. With the permissions from the markers, everything runs – that is the real test of your RBAC.

Save your progress? Sign up for free

Drill 3: A missing marker – and a readiness that lies

Take away the operator’s permission to watch Services and apply the role again:

cp internal/controller/staticsite_controller.go lab/controller.go.bak
sed 's#configmaps;services,verbs=get;list;watch#configmaps,verbs=get;list;watch#' \
  lab/controller.go.bak > internal/controller/staticsite_controller.go
make manifests build >/dev/null && kubectl apply -k config/lab-rbac | grep configured
kill "$(cat lab/manager.pid)"
bin/manager --kubeconfig ~/op-lab/sa.kubeconfig -health-probe-bind-address=:8081 \
  > lab/run.log 2>&1 &
echo $! > lab/manager.pid
sleep 5; grep -m1 forbidden lab/run.log | grep -o 'failed to list[^:]*: [^:]*'
curl -s -w ' %{http_code}\n' localhost:8081/readyz

Expected:

clusterrole.rbac.authorization.k8s.io/site-operator-manager-role configured
failed to list *v1.Service: services is forbidden
ok 200

The informer for Services never gets its first list, the controller never starts its workers – and /readyz still reports ok. The Kubebuilder default only checks healthz.Ping: “The HTTP server answers.” After CacheSyncTimeout (2 minutes), the manager then exits with timed out waiting for cache to be synced, Kubernetes restarts it – a CrashLoop without a clear hint.

Save your progress? Sign up for free

Drill 4: Readiness that means something

A custom check asks the cache whether all informers are synced. It goes into its own package, not into cmd/: make build compiles only the file cmd/main.go (go build -o bin/manager cmd/main.go), so the compiler would not find a second file next to it.

mkdir -p internal/health
cat > internal/health/health.go <<'EOF'
// Package health contains readiness checks for the manager.
package health

import (
    "context"
    "errors"
    "net/http"
    "time"

    "sigs.k8s.io/controller-runtime/pkg/cache"
    "sigs.k8s.io/controller-runtime/pkg/healthz"
)

// CacheSynced reports "ready" only once all informers have their first list.
// If, e.g., RBAC is missing for a watched type, the pod stays NotReady.
func CacheSynced(c cache.Cache) healthz.Checker {
    return func(req *http.Request) error {
        ctx, cancel := context.WithTimeout(req.Context(), time.Second)
        defer cancel()
        if !c.WaitForCacheSync(ctx) {
            return errors.New("informer-cache nicht synchron")
        }
        return nil
    }
}
EOF

In cmd/main.go, the check replaces the ping for readiness (liveness stays with the ping – a missing permission is no reason for a restart). Import "example.com/site-operator/internal/health":

    ready := health.CacheSynced(mgr.GetCache())
    if err := mgr.AddReadyzCheck("informers", ready); err != nil {
        setupLog.Error(err, "Failed to set up ready check")
        os.Exit(1)
    }
make build >/dev/null && kill "$(cat lab/manager.pid)"
bin/manager --kubeconfig ~/op-lab/sa.kubeconfig -health-probe-bind-address=:8081 \
  > lab/run.log 2>&1 &
echo $! > lab/manager.pid
sleep 5; curl -s -w ' %{http_code}\n' 'localhost:8081/readyz?verbose'

Expected:

[-]informers failed: reason withheld
healthz check failed
 500

“reason withheld” is intentional: error details are in the manager’s log, not in the HTTP response. Now give the permission back:

cp lab/controller.go.bak internal/controller/staticsite_controller.go
make manifests build >/dev/null && kubectl apply -k config/lab-rbac | grep configured
kill "$(cat lab/manager.pid)"
bin/manager --kubeconfig ~/op-lab/sa.kubeconfig -health-probe-bind-address=:8081 \
  > lab/run.log 2>&1 &
echo $! > lab/manager.pid
sleep 4; curl -s -w ' %{http_code}\n' 'localhost:8081/readyz?verbose'

Expected:

clusterrole.rbac.authorization.k8s.io/site-operator-manager-role configured
[+]informers ok
healthz check passed
 200

Why? A pod that stays NotReady stands out immediately during a rollout (kubectl rollout status hangs, the old ReplicaSet stays). A pod that reports “Ready” and does nothing only stands out when customers complain.

Save your progress? Sign up for free

Drill 5: Leader election with two instances

In the cluster, the manager runs with --leader-elect. Outside the cluster, however, it does not know which namespace the lease should live in:

bin/manager --kubeconfig ~/op-lab/sa.kubeconfig -health-probe-bind-address=0 \
  --leader-elect 2>&1 | grep -o '"error": "[^"]*"' | cut -c1-80

Expected:

"error": "unable to find leader election namespace: not running in-cluster, pl

A flag fixes this. In cmd/main.go, with the other variables and flags:

    var leaderElectionNS string
    flag.StringVar(&leaderElectionNS, "leader-election-namespace", "",
        "Namespace der Leader-Lease (nötig außerhalb des Clusters)")

and in ctrl.Options directly below LeaderElectionID:

        LeaderElectionNamespace: leaderElectionNS,

Now two instances side by side – different probe ports, same lease:

make build >/dev/null && kill "$(cat lab/manager.pid)"
for n in 1 2; do
  bin/manager --kubeconfig ~/op-lab/sa.kubeconfig \
    -health-probe-bind-address=:808$n --leader-elect \
    --leader-election-namespace=site-operator-system > lab/le-$n.log 2>&1 &
  echo $! > lab/le-$n.pid
  sleep 3
done
grep -l 'Successfully acquired' lab/le-*.log
kubectl get lease -n site-operator-system

Expected:

lab/le-1.log
NAME                   HOLDER                                       AGE
b986d597.lab.example   mac_dc48cb4f-d75f-4f28-a27d-58727f08ae88       6s

Instance 1 is the leader (<hostname>_<uuid>), instance 2 waits in Attempting to acquire leader lease.... Now the leader crashes – kill -9, with no chance to clean up:

date +%T; kill -9 "$(cat lab/le-1.pid)"
until grep -q 'Successfully acquired' lab/le-2.log; do sleep 0.5; done; date +%T

Expected:

11:44:16
11:44:33

17 seconds without an active controller: the lease is valid for 15 seconds (LeaseDuration), after that instance 2 tries every 2 seconds (RetryPeriod). Even a clean kill (SIGTERM) takes this long without an extra option – you fix that on Day 13.

kill "$(cat lab/le-2.pid)"

Warning: Leader election protects only runnables that ask for it. Controllers do, while the webhook server and the metrics and probe servers run on every replica. Webhooks must therefore work without a leader.

Save your progress? Sign up for free

Drill 6: Metrics – protected by token and RBAC

The metrics server is turned off in the scaffold (--metrics-bind-address=0). In the cluster, config/default sets :8443 – with HTTPS (self-signed if there is no certificate) and the filter filters.WithAuthenticationAndAuthorization: every call sends a bearer token, the manager checks it via TokenReview and asks via SubjectAccessReview whether the caller may get the non-resource /metrics. For this, the manager itself needs the role metrics-auth-role – kubectl apply -k has already created it.

bin/manager --kubeconfig ~/op-lab/sa.kubeconfig -health-probe-bind-address=:8081 \
  --metrics-bind-address=:8443 > lab/run.log 2>&1 &
echo $! > lab/manager.pid
sleep 4
kubectl create serviceaccount prometheus
T=$(kubectl create token prometheus)
curl -sk -o /dev/null -w '%{http_code}\n' https://localhost:8443/metrics
curl -sk -w ' %{http_code}\n' -H "Authorization: Bearer $T" \
  https://localhost:8443/metrics

Expected:

serviceaccount/prometheus created
401
Authorization denied for user system:serviceaccount:default:prometheus
 403

401: no token. 403: valid token, but no permission. The prepared ClusterRole metrics-reader grants the permission:

kubectl create clusterrolebinding prometheus-metrics \
  --clusterrole=site-operator-metrics-reader --serviceaccount=default:prometheus
curl -sk -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $T" \
  https://localhost:8443/metrics
sleep 30
curl -sk -H "Authorization: Bearer $T" https://localhost:8443/metrics \
  | grep -E '^controller_runtime_(reconcile_total|max_concurrent|active_workers)'

Expected:

clusterrolebinding.rbac.authorization.k8s.io/prometheus-metrics created
403
controller_runtime_active_workers{controller="staticsite"} 0
controller_runtime_max_concurrent_reconciles{controller="staticsite"} 1
controller_runtime_reconcile_total{controller="staticsite",result="error"} 0
controller_runtime_reconcile_total{controller="staticsite",result="requeue"} 0
controller_runtime_reconcile_total{controller="staticsite",result="requeue_after"} 0
controller_runtime_reconcile_total{controller="staticsite",result="success"} 1

Why sleep 30? The filter caches decisions: a “no” for 30 seconds, a “yes” for 5 minutes (DenyCacheTTL/AllowCacheTTL in pkg/metrics/filters). That is why you get another 403 right after the binding – a popular trap when setting up Prometheus.

Tip: More useful standard metrics: controller_runtime_reconcile_time_seconds (histogram per controller), workqueue_depth and workqueue_longest_running_processor_seconds (is a worker stuck?), rest_client_requests_total (load on the API server).

Save your progress? Sign up for free

Drill 7: Custom metrics

The standard metrics do not answer two questions: “Does the operator keep fiddling with its child objects?” and “Which sites are unavailable right now?”

cat > internal/controller/metrics.go <<'EOF'
package controller

import (
    "github.com/prometheus/client_golang/prometheus"
    "k8s.io/apimachinery/pkg/api/meta"
    "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
    "sigs.k8s.io/controller-runtime/pkg/metrics"

    webv1 "example.com/site-operator/api/v1"
)

var (
    // childWrites counts what CreateOrUpdate did with the child objects.
    // If "updated" keeps rising, someone is fighting the operator (Day 13).
    childWrites = prometheus.NewCounterVec(prometheus.CounterOpts{
        Name: "site_operator_child_writes_total",
        Help: "Abgleich der Kindobjekte nach Art und Ergebnis",
    }, []string{"kind", "operation"})

    // siteAvailable is 1 if the condition Available=True, otherwise 0.
    siteAvailable = prometheus.NewGaugeVec(prometheus.GaugeOpts{
        Name: "site_operator_site_available",
        Help: "Available-Condition je StaticSite (1 = True)",
    }, []string{"namespace", "name"})
)

func init() {
    // The manager's registry – everything in it shows up under /metrics.
    metrics.Registry.MustRegister(childWrites, siteAvailable)
}

func countWrite(kind string, op controllerutil.OperationResult) {
    childWrites.WithLabelValues(kind, string(op)).Inc()
}

func setAvailableMetric(site *webv1.StaticSite) {
    v := 0.0
    if meta.IsStatusConditionTrue(site.Status.Conditions,
        webv1.ConditionAvailable) {
        v = 1
    }
    siteAvailable.WithLabelValues(site.Namespace, site.Name).Set(v)
}
EOF

Three small additions. In reconcileConfigMap, reconcileDeployment and reconcileService, each directly after the log line … abgeglichen (with "ConfigMap", "Deployment", "Service"):

    countWrite("Deployment", op)

In updateStatus (status.go), before the comparison with the old status:

    setAvailableMetric(site)
    if equality.Semantic.DeepEqual(orig.Status, site.Status) {

And in Reconcile: if the site is gone, its time series must go too – otherwise the dashboard reports a deleted site forever:

    if err := r.Get(ctx, req.NamespacedName, &site); err != nil {
        if apierrors.IsNotFound(err) { // gelöscht: Zeitreihe entfernen
            siteAvailable.DeleteLabelValues(req.Namespace, req.Name)
        }
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }
make build >/dev/null && kill "$(cat lab/manager.pid)"
bin/manager --kubeconfig ~/op-lab/sa.kubeconfig -health-probe-bind-address=:8081 \
  --metrics-bind-address=:8443 > lab/run.log 2>&1 &
echo $! > lab/manager.pid
sleep 4
kubectl patch staticsite demo --type=merge -p '{"spec":{"replicas":2}}'
sleep 2
curl -sk -H "Authorization: Bearer $T" https://localhost:8443/metrics \
  | grep '^site_operator'

Expected:

staticsite.web.lab.example/demo patched
site_operator_child_writes_total{kind="ConfigMap",operation="unchanged"} 3
site_operator_child_writes_total{kind="Deployment",operation="unchanged"} 2
site_operator_child_writes_total{kind="Deployment",operation="updated"} 1
site_operator_child_writes_total{kind="Service",operation="unchanged"} 3
site_operator_site_available{name="demo",namespace="default"} 0

Exactly one updated for the Deployment (replicas 1 → 2), everything else unchanged. available stays 0 – no pod starts in envtest.

Warning: Every label combination is a separate time series. namespace/name are fine for a few hundred sites, not for tens of thousands. Never use free text (error messages, URLs) as a label.

Save your progress? Sign up for free

Drill 8: Structured logs

The scaffold sets zap.Options{Development: true}: readable console, debug level. In production you want JSON – Loki, Elasticsearch or jq can filter on it. opts.BindFlags already brings the flags for this:

kill "$(cat lab/manager.pid)"
bin/manager --kubeconfig ~/op-lab/sa.kubeconfig -health-probe-bind-address=:8081 \
  --zap-encoder=json --zap-log-level=info > lab/run.json 2>&1 &
echo $! > lab/manager.pid
sleep 3
kubectl patch staticsite demo --type=merge -p '{"spec":{"replicas":3}}'
sleep 2
jq -c 'select(.msg == "Deployment abgeglichen") | {name, operation, reconcileID}' \
  lab/run.json

Expected (IDs shortened):

staticsite.web.lab.example/demo patched
{"name":"demo","operation":"unchanged","reconcileID":"e9b645a8-3b59-40a1-b84e-..."}
{"name":"demo","operation":"updated","reconcileID":"d405209b-a0e4-475f-92f4-..."}
{"name":"demo","operation":"unchanged","reconcileID":"85afde9d-dc91-4d76-aa03-..."}

Every line automatically carries controller, namespace, name and a reconcileID – controller-runtime attaches them to the logger in the context (logf.FromContext(ctx)). You find all lines of one reconcile run via the ID. Your own keys are added as pairs, never via fmt.Sprintf in the message:

    log := logf.FromContext(ctx).WithValues("generation", site.Generation)
    log.Info("Deployment abgeglichen", "operation", op)  // gut: filterbar
    log.Info(fmt.Sprintf("Deployment %s: %s", site.Name, op)) // schlecht

--zap-log-level=info hides log.V(1); --zap-log-level=debug (or 2, 3 …) shows more. The flags belong in the Deployment’s args.

Tracing outlook: controller-runtime has no built-in tracing for Reconcile. With OpenTelemetry (go.opentelemetry.io/otel), you start a span in Reconcile, pass the ctx along and instrument the registrar’s HTTP client (otelhttp.NewTransport) – then you can see whether a slow reconcile is stuck on the API server or on the external service. The kube-apiserver can export spans itself (--tracing-config-file); both together give you end-to-end traces.

Save your progress? Sign up for free

Checkpoint

The manager runs with the ServiceAccount token, metrics on :8443, probes on :8081:

kill "$(cat lab/manager.pid)"
bin/manager --kubeconfig ~/op-lab/sa.kubeconfig -health-probe-bind-address=:8081 \
  --metrics-bind-address=:8443 > lab/run.log 2>&1 &
echo $! > lab/manager.pid
cat > lab/check.sh <<'EOF'
ok()  { echo "✔ $1"; }
nok() { echo "✘ $1"; }
SA=system:serviceaccount:site-operator-system:site-operator-controller-manager
sleep 4
[ "$(kubectl auth can-i watch services --as=$SA)" = yes ] \
  && ok "RBAC: Services beobachten" || nok "RBAC fehlt"
[ "$(kubectl auth can-i delete deployments --as=$SA)" = no ] \
  && ok "RBAC: kein delete" || nok "RBAC zu weit"
curl -s 'localhost:8081/readyz?verbose' | grep -q '\[+\]informers ok' \
  && ok "readyz mit Cache-Check" || nok "readyz"
T=$(kubectl create token prometheus)
curl -sk -H "Authorization: Bearer $T" https://localhost:8443/metrics \
  | grep -q '^site_operator_site_available' && ok "eigene Metriken" || nok "Metriken"
[ "$(curl -sk -o /dev/null -w '%{http_code}' https://localhost:8443/metrics)" = 401 ] \
  && ok "Metriken ohne Token gesperrt" || nok "Metriken offen"
grep -q 'leader-election-namespace' cmd/main.go && ok "Lease-Namespace-Flag" \
  || nok "Flag fehlt"
EOF
bash lab/check.sh

Expected:

✔ RBAC: Services beobachten
✔ RBAC: kein delete
✔ readyz mit Cache-Check
✔ eigene Metriken
✔ Metriken ohne Token gesperrt
✔ Lease-Namespace-Flag

Challenge

Measure the calls to the DNS registrar (Day 7): a histogram site_operator_registrar_request_duration_seconds with the labels method (PUT/DELETE) and result (ok/error). Check it with registrar running and --registrar-url.

Hint: All calls go through Client.do in internal/registrar/client.go – a defer with time.Since is enough there.

Solution

cat > internal/registrar/metrics.go <<'EOF'
package registrar

import (
    "github.com/prometheus/client_golang/prometheus"
    "sigs.k8s.io/controller-runtime/pkg/metrics"
)

var requestDuration = prometheus.NewHistogramVec(prometheus.HistogramOpts{
    Name:    "site_operator_registrar_request_duration_seconds",
    Help:    "Dauer der Aufrufe beim DNS-Registrar",
    Buckets: []float64{.005, .025, .1, .5, 1, 5},
}, []string{"method", "result"})

func init() { metrics.Registry.MustRegister(requestDuration) }
EOF

In Client.do, name the return value and measure at the start:

func (c *Client) do(ctx context.Context, method, name string, body []byte,
    ok ...int) (err error) {
    defer func(start time.Time) {
        result := "ok"
        if err != nil {
            result = "error"
        }
        requestDuration.WithLabelValues(method, result).
            Observe(time.Since(start).Seconds())
    }(time.Now())
registrar 2> lab/registrar.log &   # Tag 7, Port 18090
echo $! > lab/registrar.pid
make build >/dev/null && kill "$(cat lab/manager.pid)"
bin/manager --kubeconfig ~/op-lab/sa.kubeconfig -health-probe-bind-address=:8081 \
  --metrics-bind-address=:8443 --registrar-url=http://127.0.0.1:18090 \
  > lab/run.log 2>&1 &
echo $! > lab/manager.pid
sleep 4
curl -sk -H "Authorization: Bearer $T" https://localhost:8443/metrics \
  | grep 'registrar_request_duration_seconds_count'

Expected:

site_operator_registrar_request_duration_seconds_count{method="PUT",result="ok"} 1

The code state in code/tag11-site-operator does not include the challenge.

Save your progress? Sign up for free

Enterprise Note

  • RBAC markers are code and belong in review: every new verb, especially on secrets, * or cluster-wide resources. With kubebuilder edit --namespaced (or rbac.namespaced in the Helm chart), the ClusterRole becomes a Role if the operator manages only one namespace.
  • In the cluster, Prometheus reads the metrics via a ServiceMonitor (config/prometheus) with the token of its ServiceAccount; that ServiceAccount needs the role metrics-reader. For production, use cert-manager (--metrics-cert-path) instead of self-signed certificates.
  • Lease times determine the downtime during a leader change; on overloaded API servers, times that are too short can lead to constant leader changes (Day 13).
  • Readiness: check only what the pod can fix itself or what makes it useless (cache, webhook server). External services belong in conditions and metrics, not in liveness – otherwise an outage of the registrar restarts all operator pods.
  • At many customers, logs go into a central SIEM: do not log tokens, Secrets or complete spec contents; JSON with stable keys (namespace, name, reconcileID) makes correlation easier.