Day 11: RBAC, Leader Election, Metrics, Health
Goal: The site-operator runs with exactly the permissions from its RBAC markers, elects a leader, delivers protected Prometheus metrics (including its own), reports health and readiness and logs in a structured way · Duration: approx. 150 min · Prerequisite: Day 10
Briefing
- RBAC comes from
+kubebuilder:rbacmarkers:make manifestswritesconfig/rbac/role.yaml. If a marker is missing, you only notice it with a real ServiceAccount. - Leader election: several replicas, but only one reconciles. Whoever holds the Lease is the leader.
- The manager serves metrics at
/metrics– over HTTPS, with token checking (TokenReview) and RBAC checking (SubjectAccessReview). /healthz(liveness) and/readyz(readiness) on port 8081, your own checks viaAddHealthzCheck/AddReadyzCheck.- Logs are structured (zap/logr): key-value pairs instead of free text, as JSON in production.
ServiceAccount site-operator-controller-manager
| ClusterRole manager-role (from markers)
envtest <-----------+ Role leader-election (Lease)
^ TokenReview/SAR manager :8443 /metrics <-- curl + token
'--------------------------- :8081 /healthz /readyz
Drill 1: RBAC from markers
source ~/op-lab/env.sh
cp -r ~/op-lab/tag10-site-operator ~/op-lab/tag11-site-operator
cd ~/op-lab/tag11-site-operator
grep -h 'kubebuilder:rbac' internal/controller/*.go | cut -c1-80Expected (shortened):
// +kubebuilder:rbac:groups=web.lab.example,resources=staticsites,verbs=get;list;wa
// +kubebuilder:rbac:groups=web.lab.example,resources=staticsites/status,verbs=get;
// +kubebuilder:rbac:groups="",resources=configmaps;services,verbs=get;list;watch
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch
// +kubebuilder:rbac:groups=events.k8s.io,resources=events,verbs=create;patch
controller-gen collects all markers in the package and
merges them into one ClusterRole
manager-role:
make manifests
grep -A3 'apps' config/rbac/role.yamlExpected:
- apps
resources:
- deployments
verbs:
Next to role.yaml, config/rbac/ holds the
bindings, the ServiceAccount, the role for the leader lease and the
roles for metrics protection. config/default puts the
namespace site-operator-system and the prefix
site-operator- in front. For the lab you build a small
overlay that creates only these RBAC objects – with
exactly the names the real Deployment will use later:
mkdir -p config/lab-rbac
cat > config/lab-rbac/kustomization.yaml <<'EOF'
# Lab only: RBAC from config/rbac with the same names as config/default
namespace: site-operator-system
namePrefix: site-operator-
resources:
- namespace.yaml
- ../rbac
EOF
cat > config/lab-rbac/namespace.yaml <<'EOF'
apiVersion: v1
kind: Namespace
metadata:
name: system
EOFname: system is not a typo: the prefix and the namespace
transformer turn it into site-operator-system, just like in
config/default.
Save your progress? Sign up for free
Drill 2: The operator as a ServiceAccount
So far the manager ran with your admin kubeconfig
(system:masters) – it was allowed to do everything, and a
missing marker never showed up. Now it gets the identity it would have
in the cluster.
Terminal 1: start envtest with the CRD (runs until the end of the day):
source ~/op-lab/env.sh
cd ~/op-lab/tag11-site-operator && envtest-up --crds config/crd/basesTerminal 2 (working terminal; today you start the operator in the background because you restart it often – Terminal 3 stays free):
kubectl apply -k config/lab-rbac | grep -E 'namespace/|serviceaccount/|manager-role '
SA=system:serviceaccount:site-operator-system:site-operator-controller-manager
kubectl auth can-i list deployments --as=$SA
kubectl auth can-i delete deployments --as=$SA
kubectl auth can-i get secrets --as=$SA
kubectl auth can-i create leases -n site-operator-system --as=$SAExpected:
namespace/site-operator-system created
serviceaccount/site-operator-controller-manager created
clusterrole.rbac.authorization.k8s.io/site-operator-manager-role created
yes
no
no
yes
delete is missing on purpose: the garbage collector
cleans up children via the owner reference, the operator itself deletes
nothing. The lease role applies only in its own namespace. Now a
kubeconfig with the ServiceAccount’s token:
cp ~/op-lab/envtest.kubeconfig ~/op-lab/sa.kubeconfig
TOKEN=$(kubectl create token site-operator-controller-manager \
-n site-operator-system --duration=4h)
export KUBECONFIG=~/op-lab/sa.kubeconfig
kubectl config set-credentials operator --token="$TOKEN" >/dev/null
kubectl config set-context operator --cluster=envtest --user=operator >/dev/null
kubectl config use-context operator
kubectl auth whoami | head -2 | cut -c1-80
kubectl get secrets 2>&1 | cut -c1-80
export KUBECONFIG=~/op-lab/envtest.kubeconfigExpected:
Switched to context "operator".
ATTRIBUTE VALUE
Username system:serviceaccount:site-operat
Error from server (Forbidden): secrets is forbidden: User "system:serviceaccount
Security: The token is a real lab credential with a lifetime of 4 hours.
sa.kubeconfigsits next toenvtest.kubeconfigand is valid only for envtest; when it stops,envtest-updeletes only its own file – you clean upsa.kubeconfigyourself at the end.
Today you build the manager as a binary – two instances and
kill work more reliably that way than with
go run. Webhooks stay off, as on Day 10:
export ENABLE_WEBHOOKS=false
make build
bin/manager --kubeconfig ~/op-lab/sa.kubeconfig -health-probe-bind-address=:8081 \
> lab/run.log 2>&1 &
echo $! > lab/manager.pid
cat > lab/site.yaml <<'EOF'
apiVersion: web.lab.example/v1
kind: StaticSite
metadata:
name: demo
spec:
content: "<h1>Hallo vom site-operator</h1>"
EOF
kubectl apply -f lab/site.yaml
sleep 3; kubectl get site demo; grep -c ERROR lab/run.logExpected:
staticsite.web.lab.example/demo created
NAME READY AVAILABLE URL AGE
demo False http://demo.default.svc 3s
0
controller-runtime registers --kubeconfig itself;
ctrl.GetConfigOrDie() reads the flag before
KUBECONFIG. With the permissions from the markers,
everything runs – that is the real test of your RBAC.
Save your progress? Sign up for free
Drill 3: A missing marker – and a readiness that lies
Take away the operator’s permission to watch Services and apply the role again:
cp internal/controller/staticsite_controller.go lab/controller.go.bak
sed 's#configmaps;services,verbs=get;list;watch#configmaps,verbs=get;list;watch#' \
lab/controller.go.bak > internal/controller/staticsite_controller.go
make manifests build >/dev/null && kubectl apply -k config/lab-rbac | grep configured
kill "$(cat lab/manager.pid)"
bin/manager --kubeconfig ~/op-lab/sa.kubeconfig -health-probe-bind-address=:8081 \
> lab/run.log 2>&1 &
echo $! > lab/manager.pid
sleep 5; grep -m1 forbidden lab/run.log | grep -o 'failed to list[^:]*: [^:]*'
curl -s -w ' %{http_code}\n' localhost:8081/readyzExpected:
clusterrole.rbac.authorization.k8s.io/site-operator-manager-role configured
failed to list *v1.Service: services is forbidden
ok 200
The informer for Services never gets its first list, the controller
never starts its workers – and /readyz still reports
ok. The Kubebuilder default only checks
healthz.Ping: “The HTTP server answers.” After
CacheSyncTimeout (2 minutes), the manager then exits with
timed out waiting for cache to be synced, Kubernetes
restarts it – a CrashLoop without a clear hint.
Save your progress? Sign up for free
Drill 4: Readiness that means something
A custom check asks the cache whether all informers are synced. It
goes into its own package, not into cmd/:
make build compiles only the file cmd/main.go
(go build -o bin/manager cmd/main.go), so the compiler
would not find a second file next to it.
mkdir -p internal/health
cat > internal/health/health.go <<'EOF'
// Package health contains readiness checks for the manager.
package health
import (
"context"
"errors"
"net/http"
"time"
"sigs.k8s.io/controller-runtime/pkg/cache"
"sigs.k8s.io/controller-runtime/pkg/healthz"
)
// CacheSynced reports "ready" only once all informers have their first list.
// If, e.g., RBAC is missing for a watched type, the pod stays NotReady.
func CacheSynced(c cache.Cache) healthz.Checker {
return func(req *http.Request) error {
ctx, cancel := context.WithTimeout(req.Context(), time.Second)
defer cancel()
if !c.WaitForCacheSync(ctx) {
return errors.New("informer-cache nicht synchron")
}
return nil
}
}
EOFIn cmd/main.go, the check replaces the ping for
readiness (liveness stays with the ping – a missing permission is no
reason for a restart). Import
"example.com/site-operator/internal/health":
ready := health.CacheSynced(mgr.GetCache())
if err := mgr.AddReadyzCheck("informers", ready); err != nil {
setupLog.Error(err, "Failed to set up ready check")
os.Exit(1)
}make build >/dev/null && kill "$(cat lab/manager.pid)"
bin/manager --kubeconfig ~/op-lab/sa.kubeconfig -health-probe-bind-address=:8081 \
> lab/run.log 2>&1 &
echo $! > lab/manager.pid
sleep 5; curl -s -w ' %{http_code}\n' 'localhost:8081/readyz?verbose'Expected:
[-]informers failed: reason withheld
healthz check failed
500
“reason withheld” is intentional: error details are in the manager’s log, not in the HTTP response. Now give the permission back:
cp lab/controller.go.bak internal/controller/staticsite_controller.go
make manifests build >/dev/null && kubectl apply -k config/lab-rbac | grep configured
kill "$(cat lab/manager.pid)"
bin/manager --kubeconfig ~/op-lab/sa.kubeconfig -health-probe-bind-address=:8081 \
> lab/run.log 2>&1 &
echo $! > lab/manager.pid
sleep 4; curl -s -w ' %{http_code}\n' 'localhost:8081/readyz?verbose'Expected:
clusterrole.rbac.authorization.k8s.io/site-operator-manager-role configured
[+]informers ok
healthz check passed
200
Why? A pod that stays NotReady stands out immediately during a
rollout (kubectl rollout status hangs, the old ReplicaSet
stays). A pod that reports “Ready” and does nothing only stands out when
customers complain.
Save your progress? Sign up for free
Drill 5: Leader election with two instances
In the cluster, the manager runs with --leader-elect.
Outside the cluster, however, it does not know which namespace the lease
should live in:
bin/manager --kubeconfig ~/op-lab/sa.kubeconfig -health-probe-bind-address=0 \
--leader-elect 2>&1 | grep -o '"error": "[^"]*"' | cut -c1-80Expected:
"error": "unable to find leader election namespace: not running in-cluster, pl
A flag fixes this. In cmd/main.go, with the other
variables and flags:
var leaderElectionNS string
flag.StringVar(&leaderElectionNS, "leader-election-namespace", "",
"Namespace der Leader-Lease (nötig außerhalb des Clusters)")and in ctrl.Options directly below
LeaderElectionID:
LeaderElectionNamespace: leaderElectionNS,Now two instances side by side – different probe ports, same lease:
make build >/dev/null && kill "$(cat lab/manager.pid)"
for n in 1 2; do
bin/manager --kubeconfig ~/op-lab/sa.kubeconfig \
-health-probe-bind-address=:808$n --leader-elect \
--leader-election-namespace=site-operator-system > lab/le-$n.log 2>&1 &
echo $! > lab/le-$n.pid
sleep 3
done
grep -l 'Successfully acquired' lab/le-*.log
kubectl get lease -n site-operator-systemExpected:
lab/le-1.log
NAME HOLDER AGE
b986d597.lab.example mac_dc48cb4f-d75f-4f28-a27d-58727f08ae88 6s
Instance 1 is the leader
(<hostname>_<uuid>), instance 2 waits in
Attempting to acquire leader lease.... Now the leader
crashes – kill -9, with no chance to clean up:
date +%T; kill -9 "$(cat lab/le-1.pid)"
until grep -q 'Successfully acquired' lab/le-2.log; do sleep 0.5; done; date +%TExpected:
11:44:16
11:44:33
17 seconds without an active controller: the lease is valid for 15
seconds (LeaseDuration), after that instance 2 tries every
2 seconds (RetryPeriod). Even a clean kill
(SIGTERM) takes this long without an extra option – you fix that on Day
13.
kill "$(cat lab/le-2.pid)"Warning: Leader election protects only runnables that ask for it. Controllers do, while the webhook server and the metrics and probe servers run on every replica. Webhooks must therefore work without a leader.
Save your progress? Sign up for free
Drill 6: Metrics – protected by token and RBAC
The metrics server is turned off in the scaffold
(--metrics-bind-address=0). In the cluster,
config/default sets :8443 – with HTTPS
(self-signed if there is no certificate) and the filter
filters.WithAuthenticationAndAuthorization: every call
sends a bearer token, the manager checks it via
TokenReview and asks via
SubjectAccessReview whether the caller may
get the non-resource /metrics. For this, the
manager itself needs the role metrics-auth-role –
kubectl apply -k has already created it.
bin/manager --kubeconfig ~/op-lab/sa.kubeconfig -health-probe-bind-address=:8081 \
--metrics-bind-address=:8443 > lab/run.log 2>&1 &
echo $! > lab/manager.pid
sleep 4
kubectl create serviceaccount prometheus
T=$(kubectl create token prometheus)
curl -sk -o /dev/null -w '%{http_code}\n' https://localhost:8443/metrics
curl -sk -w ' %{http_code}\n' -H "Authorization: Bearer $T" \
https://localhost:8443/metricsExpected:
serviceaccount/prometheus created
401
Authorization denied for user system:serviceaccount:default:prometheus
403
401: no token. 403: valid token, but no permission. The prepared
ClusterRole metrics-reader grants the permission:
kubectl create clusterrolebinding prometheus-metrics \
--clusterrole=site-operator-metrics-reader --serviceaccount=default:prometheus
curl -sk -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $T" \
https://localhost:8443/metrics
sleep 30
curl -sk -H "Authorization: Bearer $T" https://localhost:8443/metrics \
| grep -E '^controller_runtime_(reconcile_total|max_concurrent|active_workers)'Expected:
clusterrolebinding.rbac.authorization.k8s.io/prometheus-metrics created
403
controller_runtime_active_workers{controller="staticsite"} 0
controller_runtime_max_concurrent_reconciles{controller="staticsite"} 1
controller_runtime_reconcile_total{controller="staticsite",result="error"} 0
controller_runtime_reconcile_total{controller="staticsite",result="requeue"} 0
controller_runtime_reconcile_total{controller="staticsite",result="requeue_after"} 0
controller_runtime_reconcile_total{controller="staticsite",result="success"} 1
Why sleep 30? The filter caches decisions: a “no” for 30
seconds, a “yes” for 5 minutes
(DenyCacheTTL/AllowCacheTTL in
pkg/metrics/filters). That is why you get another 403 right
after the binding – a popular trap when setting up Prometheus.
Tip: More useful standard metrics:
controller_runtime_reconcile_time_seconds(histogram per controller),workqueue_depthandworkqueue_longest_running_processor_seconds(is a worker stuck?),rest_client_requests_total(load on the API server).
Save your progress? Sign up for free
Drill 7: Custom metrics
The standard metrics do not answer two questions: “Does the operator keep fiddling with its child objects?” and “Which sites are unavailable right now?”
cat > internal/controller/metrics.go <<'EOF'
package controller
import (
"github.com/prometheus/client_golang/prometheus"
"k8s.io/apimachinery/pkg/api/meta"
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
"sigs.k8s.io/controller-runtime/pkg/metrics"
webv1 "example.com/site-operator/api/v1"
)
var (
// childWrites counts what CreateOrUpdate did with the child objects.
// If "updated" keeps rising, someone is fighting the operator (Day 13).
childWrites = prometheus.NewCounterVec(prometheus.CounterOpts{
Name: "site_operator_child_writes_total",
Help: "Abgleich der Kindobjekte nach Art und Ergebnis",
}, []string{"kind", "operation"})
// siteAvailable is 1 if the condition Available=True, otherwise 0.
siteAvailable = prometheus.NewGaugeVec(prometheus.GaugeOpts{
Name: "site_operator_site_available",
Help: "Available-Condition je StaticSite (1 = True)",
}, []string{"namespace", "name"})
)
func init() {
// The manager's registry – everything in it shows up under /metrics.
metrics.Registry.MustRegister(childWrites, siteAvailable)
}
func countWrite(kind string, op controllerutil.OperationResult) {
childWrites.WithLabelValues(kind, string(op)).Inc()
}
func setAvailableMetric(site *webv1.StaticSite) {
v := 0.0
if meta.IsStatusConditionTrue(site.Status.Conditions,
webv1.ConditionAvailable) {
v = 1
}
siteAvailable.WithLabelValues(site.Namespace, site.Name).Set(v)
}
EOFThree small additions. In reconcileConfigMap,
reconcileDeployment and reconcileService, each
directly after the log line … abgeglichen (with
"ConfigMap", "Deployment",
"Service"):
countWrite("Deployment", op)In updateStatus (status.go), before the
comparison with the old status:
setAvailableMetric(site)
if equality.Semantic.DeepEqual(orig.Status, site.Status) {And in Reconcile: if the site is gone, its time series
must go too – otherwise the dashboard reports a deleted site
forever:
if err := r.Get(ctx, req.NamespacedName, &site); err != nil {
if apierrors.IsNotFound(err) { // gelöscht: Zeitreihe entfernen
siteAvailable.DeleteLabelValues(req.Namespace, req.Name)
}
return ctrl.Result{}, client.IgnoreNotFound(err)
}make build >/dev/null && kill "$(cat lab/manager.pid)"
bin/manager --kubeconfig ~/op-lab/sa.kubeconfig -health-probe-bind-address=:8081 \
--metrics-bind-address=:8443 > lab/run.log 2>&1 &
echo $! > lab/manager.pid
sleep 4
kubectl patch staticsite demo --type=merge -p '{"spec":{"replicas":2}}'
sleep 2
curl -sk -H "Authorization: Bearer $T" https://localhost:8443/metrics \
| grep '^site_operator'Expected:
staticsite.web.lab.example/demo patched
site_operator_child_writes_total{kind="ConfigMap",operation="unchanged"} 3
site_operator_child_writes_total{kind="Deployment",operation="unchanged"} 2
site_operator_child_writes_total{kind="Deployment",operation="updated"} 1
site_operator_child_writes_total{kind="Service",operation="unchanged"} 3
site_operator_site_available{name="demo",namespace="default"} 0
Exactly one updated for the Deployment (replicas 1 → 2),
everything else unchanged. available stays 0 –
no pod starts in envtest.
Warning: Every label combination is a separate time series.
namespace/nameare fine for a few hundred sites, not for tens of thousands. Never use free text (error messages, URLs) as a label.
Save your progress? Sign up for free
Drill 8: Structured logs
The scaffold sets zap.Options{Development: true}:
readable console, debug level. In production you want JSON – Loki,
Elasticsearch or jq can filter on it.
opts.BindFlags already brings the flags for this:
kill "$(cat lab/manager.pid)"
bin/manager --kubeconfig ~/op-lab/sa.kubeconfig -health-probe-bind-address=:8081 \
--zap-encoder=json --zap-log-level=info > lab/run.json 2>&1 &
echo $! > lab/manager.pid
sleep 3
kubectl patch staticsite demo --type=merge -p '{"spec":{"replicas":3}}'
sleep 2
jq -c 'select(.msg == "Deployment abgeglichen") | {name, operation, reconcileID}' \
lab/run.jsonExpected (IDs shortened):
staticsite.web.lab.example/demo patched
{"name":"demo","operation":"unchanged","reconcileID":"e9b645a8-3b59-40a1-b84e-..."}
{"name":"demo","operation":"updated","reconcileID":"d405209b-a0e4-475f-92f4-..."}
{"name":"demo","operation":"unchanged","reconcileID":"85afde9d-dc91-4d76-aa03-..."}
Every line automatically carries controller,
namespace, name and a reconcileID
– controller-runtime attaches them to the logger in the context
(logf.FromContext(ctx)). You find all lines of one
reconcile run via the ID. Your own keys are added as pairs, never via
fmt.Sprintf in the message:
log := logf.FromContext(ctx).WithValues("generation", site.Generation)
log.Info("Deployment abgeglichen", "operation", op) // gut: filterbar
log.Info(fmt.Sprintf("Deployment %s: %s", site.Name, op)) // schlecht--zap-log-level=info hides log.V(1);
--zap-log-level=debug (or 2, 3 …)
shows more. The flags belong in the Deployment’s args.
Tracing outlook: controller-runtime has no built-in
tracing for Reconcile. With OpenTelemetry
(go.opentelemetry.io/otel), you start a span in
Reconcile, pass the ctx along and instrument
the registrar’s HTTP client (otelhttp.NewTransport) – then
you can see whether a slow reconcile is stuck on the API server or on
the external service. The kube-apiserver can export spans itself
(--tracing-config-file); both together give you end-to-end
traces.
Save your progress? Sign up for free
Checkpoint
The manager runs with the ServiceAccount token, metrics on
:8443, probes on :8081:
kill "$(cat lab/manager.pid)"
bin/manager --kubeconfig ~/op-lab/sa.kubeconfig -health-probe-bind-address=:8081 \
--metrics-bind-address=:8443 > lab/run.log 2>&1 &
echo $! > lab/manager.pid
cat > lab/check.sh <<'EOF'
ok() { echo "✔ $1"; }
nok() { echo "✘ $1"; }
SA=system:serviceaccount:site-operator-system:site-operator-controller-manager
sleep 4
[ "$(kubectl auth can-i watch services --as=$SA)" = yes ] \
&& ok "RBAC: Services beobachten" || nok "RBAC fehlt"
[ "$(kubectl auth can-i delete deployments --as=$SA)" = no ] \
&& ok "RBAC: kein delete" || nok "RBAC zu weit"
curl -s 'localhost:8081/readyz?verbose' | grep -q '\[+\]informers ok' \
&& ok "readyz mit Cache-Check" || nok "readyz"
T=$(kubectl create token prometheus)
curl -sk -H "Authorization: Bearer $T" https://localhost:8443/metrics \
| grep -q '^site_operator_site_available' && ok "eigene Metriken" || nok "Metriken"
[ "$(curl -sk -o /dev/null -w '%{http_code}' https://localhost:8443/metrics)" = 401 ] \
&& ok "Metriken ohne Token gesperrt" || nok "Metriken offen"
grep -q 'leader-election-namespace' cmd/main.go && ok "Lease-Namespace-Flag" \
|| nok "Flag fehlt"
EOF
bash lab/check.shExpected:
✔ RBAC: Services beobachten
✔ RBAC: kein delete
✔ readyz mit Cache-Check
✔ eigene Metriken
✔ Metriken ohne Token gesperrt
✔ Lease-Namespace-Flag
Challenge
Measure the calls to the DNS registrar (Day 7): a histogram
site_operator_registrar_request_duration_seconds with the
labels method (PUT/DELETE) and result
(ok/error). Check it with
registrar running and --registrar-url.
Hint: All calls go through Client.do in
internal/registrar/client.go – a defer with
time.Since is enough there.
Solution
cat > internal/registrar/metrics.go <<'EOF'
package registrar
import (
"github.com/prometheus/client_golang/prometheus"
"sigs.k8s.io/controller-runtime/pkg/metrics"
)
var requestDuration = prometheus.NewHistogramVec(prometheus.HistogramOpts{
Name: "site_operator_registrar_request_duration_seconds",
Help: "Dauer der Aufrufe beim DNS-Registrar",
Buckets: []float64{.005, .025, .1, .5, 1, 5},
}, []string{"method", "result"})
func init() { metrics.Registry.MustRegister(requestDuration) }
EOFIn Client.do, name the return value and measure at the
start:
func (c *Client) do(ctx context.Context, method, name string, body []byte,
ok ...int) (err error) {
defer func(start time.Time) {
result := "ok"
if err != nil {
result = "error"
}
requestDuration.WithLabelValues(method, result).
Observe(time.Since(start).Seconds())
}(time.Now())registrar 2> lab/registrar.log & # Tag 7, Port 18090
echo $! > lab/registrar.pid
make build >/dev/null && kill "$(cat lab/manager.pid)"
bin/manager --kubeconfig ~/op-lab/sa.kubeconfig -health-probe-bind-address=:8081 \
--metrics-bind-address=:8443 --registrar-url=http://127.0.0.1:18090 \
> lab/run.log 2>&1 &
echo $! > lab/manager.pid
sleep 4
curl -sk -H "Authorization: Bearer $T" https://localhost:8443/metrics \
| grep 'registrar_request_duration_seconds_count'Expected:
site_operator_registrar_request_duration_seconds_count{method="PUT",result="ok"} 1
The code state in code/tag11-site-operator does not
include the challenge.
Save your progress? Sign up for free
Enterprise Note
- RBAC markers are code and belong in review: every new verb,
especially on
secrets,*or cluster-wide resources. Withkubebuilder edit --namespaced(orrbac.namespacedin the Helm chart), the ClusterRole becomes a Role if the operator manages only one namespace. - In the cluster, Prometheus reads the metrics via a
ServiceMonitor(config/prometheus) with the token of its ServiceAccount; that ServiceAccount needs the rolemetrics-reader. For production, use cert-manager (--metrics-cert-path) instead of self-signed certificates. - Lease times determine the downtime during a leader change; on overloaded API servers, times that are too short can lead to constant leader changes (Day 13).
- Readiness: check only what the pod can fix itself or what makes it useless (cache, webhook server). External services belong in conditions and metrics, not in liveness – otherwise an outage of the registrar restarts all operator pods.
- At many customers, logs go into a central SIEM: do not log tokens,
Secrets or complete spec contents; JSON with stable keys
(
namespace,name,reconcileID) makes correlation easier.