Day 16: Webhooks, APIServices and stuck namespaces
Goal: You find the webhook behind a rejected request, recognize an unavailable aggregated API and get a namespace out of
Terminatingwithout orphaning objects in etcd. · Duration: approx. 60 min · Prerequisite: Day 15
Briefing
- The API server calls admission webhooks on every matching write
request.
failurePolicy: Failrejects when the webhook does not respond,Ignorelets the request through – if the backend hangs, only aftertimeoutSeconds. - Aggregated APIs (
APIService) forward entire API groups to a Service. If the backend is missing, the cluster’s discovery is incomplete. - A namespace only disappears once everything in it has been deleted. What it is stuck on is in its conditions.
| Fault | Symptom | Where the answer is |
|---|---|---|
| Webhook without backend | failed calling webhook |
Webhook configuration, endpoints |
| Slow webhook | Every request takes 10 s | timeoutSeconds, API server log |
| APIService gone | False (MissingEndpoints) |
kubectl get apiservice |
| Namespace stuck | Terminating forever |
.status.conditions |
Drill 1: A webhook without a backend
Break it: A validating webhook for ConfigMaps in
tag16 whose Service has no Pod:
# on cp1
mkdir -p ~/debug/tag16 && cd ~/debug/tag16
kubectl create namespace tag16
cat > pruefer.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata: {name: pruefer, namespace: tag16}
spec:
selector: {app: pruefer}
ports: [{port: 443, targetPort: 8443}]
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata: {name: tag16-pruefung}
webhooks:
- name: pruefung.tag16.labor.test
admissionReviewVersions: [v1]
sideEffects: None
failurePolicy: Fail
timeoutSeconds: 5
clientConfig:
service: {name: pruefer, namespace: tag16, path: /validate}
namespaceSelector:
matchLabels: {kubernetes.io/metadata.name: tag16}
rules:
- apiGroups: [""]
apiVersions: [v1]
operations: [CREATE, UPDATE]
resources: [configmaps]
EOF
kubectl apply -f pruefer.yamlExpected:
namespace/tag16 created
service/pruefer created
validatingwebhookconfiguration.admissionregistration.k8s.io/tag16-pruefung created
The namespaceSelector limits the damage to
tag16 – without it, every namespace would be affected.
Symptom:
# on cp1
kubectl -n tag16 create configmap test --from-literal=a=bExpected (line wrapped):
error: failed to create configmap: Internal error occurred: failed calling webhook
"pruefung.tag16.labor.test": failed to call webhook: Post
"https://pruefer.tag16.svc:443/validate?timeout=5s": no endpoints available for
service "pruefer"
Investigate: The message names the webhook, not the configuration. You find both in the list:
# on cp1
kubectl get validatingwebhookconfigurations
J='{range .items[*]}{.metadata.name}{range .webhooks[*]} {.name} {.failurePolicy}'
kubectl get validatingwebhookconfigurations -o jsonpath="$J{end}{\"\n\"}{end}" \
| grep tag16
kubectl get validatingwebhookconfiguration tag16-pruefung \
-o jsonpath='{.webhooks[0].clientConfig.service}{"\n"}'
kubectl -n tag16 get endpointsliceExpected (abridged):
NAME WEBHOOKS AGE
kyverno-cel-exception-validating-webhook-cfg 1 163m
...
kyverno-ttl-validating-webhook-cfg 1 164m
tag16-pruefung 1 7s
tag16-pruefung pruefung.tag16.labor.test Fail
{"name":"pruefer","namespace":"tag16","path":"/validate","port":443}
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
pruefer-kpccn IPv4 <unset> <unset> 8s
Cause: The API server has to ask the webhook for
every ConfigMap in tag16. The Service has no endpoints (Day
7), the call fails, and because of failurePolicy: Fail the
API server rejects the request. You know the same pattern from Kyverno
(Day 12) – here without Kyverno.
Fix: The right fix is to get the backend running again. If it is gone for good, the configuration should be deleted – that comes in Drill 2. First the backend: a Pod with the matching label.
# on cp1
kubectl -n tag16 run pruefer --image=nicolaka/netshoot:v0.15 --labels=app=pruefer \
-- nc -lk -p 8443
kubectl -n tag16 wait --for=condition=Ready pod/pruefer --timeout=60s
kubectl -n tag16 get endpointsliceExpected:
pod/pruefer created
pod/pruefer condition met
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
pruefer-kpccn IPv4 8443 10.42.1.133 13s
Verify: The webhook is reachable – and the request still fails, just differently:
# on cp1
time kubectl -n tag16 create configmap test --from-literal=a=bExpected (line wrapped):
error: failed to create configmap: Internal error occurred: failed calling webhook
"pruefung.tag16.labor.test": failed to call webhook: Post
"https://pruefer.tag16.svc:443/validate?timeout=5s": context deadline exceeded
real 0m5.236s
nc accepts the connection but does not speak TLS and
never answers. A backend that “is running” is not yet a working webhook.
Drill 2 shows what happens when someone switches to Ignore
because of this.
Save your progress? Sign up for free
Drill 2: The slow webhook
Break it: Someone “defuses” the webhook:
Ignore instead of Fail, and more patience in
return.
# on cp1
kubectl patch validatingwebhookconfiguration tag16-pruefung --type=json -p '[
{"op": "replace", "path": "/webhooks/0/failurePolicy", "value": "Ignore"},
{"op": "replace", "path": "/webhooks/0/timeoutSeconds", "value": 10}]'Symptom:
# on cp1
time kubectl -n tag16 create configmap test2 --from-literal=a=bExpected:
configmap/test2 created
real 0m10.210s
The ConfigMap is created – after ten seconds. Every change in
tag16 now takes that long; with Pods as the target, this
slows down every deployment and every restart.
Investigate:
# on cp1
kubectl -n kube-system logs kube-apiserver-cp1 --since=1m \
| grep -m1 'failing open' | cut -c1-200 | fold -w 86Expected:
W1120 12:01:52.403866 1 dispatcher.go:205] Failed calling webhook, failing open
pruefung.tag16.labor.test: failed calling webhook "pruefung.tag16.labor.test": failed
to call webhook: Post "https
Cause: Ignore does not heal the
cluster, it only makes it sluggish: on every request the API server
waits timeoutSeconds and then lets it through (“failing
open”). The check never actually takes place.
Fix: The backend is not a webhook and is not coming back – the configuration gets deleted. Webhook configurations themselves never pass through webhooks, so you can always change or delete them:
# on cp1
kubectl delete validatingwebhookconfiguration tag16-pruefung
kubectl -n tag16 delete pod prueferVerify:
# on cp1
time kubectl -n tag16 create configmap test3 --from-literal=a=bExpected:
configmap/test3 created
real 0m0.264s
Save your progress? Sign up for free
Drill 3: The aggregated API is missing
Break it: RKE2’s metrics-server is switched off “temporarily”:
# on cp1
kubectl -n kube-system scale deploy rke2-metrics-server --replicas=0Symptom (after about 20 seconds):
# on cp1
sleep 20
kubectl top nodesExpected:
error: Metrics API not available
Investigate: kubectl top does not ask
the metrics-server directly but the API group
metrics.k8s.io, which the API server forwards via an
APIService:
# on cp1
kubectl get apiservice | grep -v Local | tr -s ' '
kubectl get apiservice v1beta1.metrics.k8s.io \
-o jsonpath='{.status.conditions[0].message}{"\n"}' | fold -w 86Expected:
NAME SERVICE AVAILABLE AGE
v1beta1.metrics.k8s.io kube-system/rke2-metrics-server False (MissingEndpoints) 5h30m
endpointslices for service/rke2-metrics-server in "kube-system" have no addresses with
port name "https"
All other APIServices show Local – the API server serves
them itself.
Cause: The APIService points to a Service without
endpoints. This does not only affect kubectl top and the
HPA (Day 17): the discovery of the entire cluster is now incomplete –
with consequences that Drill 4 shows. That is why the metrics-server
stays off until then.
Save your progress? Sign up for free
Drill 4: The namespace stays in Terminating
Break it: A ConfigMap with a finalizer that no controller will ever remove:
# on cp1
kubectl -n tag16 patch configmap test3 --type=merge \
-p '{"metadata":{"finalizers":["labor.test/aufraeumen"]}}'
kubectl delete namespace tag16 --wait=falseExpected:
configmap/test3 patched
namespace "tag16" deleted
--wait=false, because otherwise kubectl
waits forever for the deletion to finish.
Symptom:
# on cp1
sleep 15
kubectl get ns tag16Expected:
NAME STATUS AGE
tag16 Terminating 2m13s
AGE counts from creation in Drill 1, not from
deletion.
Investigate: The namespace controller writes what it is stuck on into the conditions:
# on cp1
kubectl get ns tag16 -o json | jq -r '.status.conditions[]
| select(.status=="True") | "\(.type): \(.message)"' | fold -w 86Expected:
NamespaceDeletionDiscoveryFailure: Discovery failed for some groups, 1 failing: unable
to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: stale GroupVers
ion discovery: metrics.k8s.io/v1beta1
NamespaceContentRemaining: Some resources are remaining: configmaps. has 1 resource in
stances
NamespaceFinalizersRemaining: Some content in the namespace has finalizers remaining:
labor.test/aufraeumen in 1 resource instances
Cause: Two things hold the namespace. The finalizer
on the ConfigMap – nobody removes it. And the incomplete discovery: the
controller deletes everything it finds, but does not know whether
objects in tag16 still exist in metrics.k8s.io
– and to be safe it does not release the namespace. Because the
APIService applies cluster-wide, every namespace you
delete during this time gets stuck.
Fix: First only the finalizer – with a real finalizer, you check beforehand which controller sets it and why it does not remove it:
# on cp1
kubectl -n tag16 patch configmap test3 --type=merge \
-p '{"metadata":{"finalizers":null}}'
sleep 20
kubectl -n tag16 get configmaps
kubectl get ns tag16
kubectl get ns tag16 -o json | jq -r '.status.conditions[]
| select(.status=="True") | .type'Expected:
configmap/test3 patched
No resources found in tag16 namespace.
NAME STATUS AGE
tag16 Terminating 2m41s
NamespaceDeletionDiscoveryFailure
The ConfigMap is gone, but the namespace is still stuck – on discovery. So repair the aggregated API:
# on cp1
kubectl -n kube-system scale deploy rke2-metrics-server --replicas=1
kubectl -n kube-system rollout status deploy/rke2-metrics-server
T0=$(date +%s)
until ! kubectl get ns tag16 >/dev/null 2>&1; do sleep 2; done
echo "weg nach $(( $(date +%s) - T0 )) s"Expected:
deployment.apps/rke2-metrics-server scaled
deployment "rke2-metrics-server" successfully rolled out
weg nach 35 s
The last line means “gone after 35 s”.
Verify:
# on cp1
kubectl get apiservice v1beta1.metrics.k8s.io
kubectl get ns tag16Expected:
NAME SERVICE AVAILABLE AGE
v1beta1.metrics.k8s.io kube-system/rke2-metrics-server True 5h32m
Error from server (NotFound): namespaces "tag16" not found
The namespace controller retries with backoff; hence the good half minute.
Warning: A tip circulating online is to remove the
kubernetesfinalizer on the namespace itself via/finalize. The namespace is then gone, but its objects are not necessarily: they remain orphaned in etcd and reappear as soon as someone creates a namespace with the same name. Always fix the cause.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show all validating and mutating webhooks with their
failurePolicy. Target time: 60 s - Show all APIServices that are not available. Target time: 20 s
- Show all namespaces in
Terminating. Target time: 20 s
Solution
# on cp1
J='{range .items[*]}{range .webhooks[*]}{.name} {.failurePolicy}{"\n"}{end}{end}'
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations \
-o jsonpath="$J" | sort | uniq -c | sort -rn
kubectl get apiservice | grep False; echo "rc=$?"
kubectl get ns --field-selector=status.phase=TerminatingExpected:
3 kyverno-svc.kyverno.svc Fail
1 validate-policy.kyverno.svc Fail
1 protect-generation-labels.kyverno.svc Fail
1 mutate-policy.kyverno.svc Fail
1 monitor-webhooks.kyverno.svc Ignore
1 kyverno-cleanup-controller.kyverno.svc Ignore
1 kyverno-cleanup-controller.kyverno.svc Fail
rc=1
No resources found
All remaining webhooks come from Kyverno (Day 12) – almost all of
them with Fail. Only monitor-webhooks
(Kyverno’s self-monitoring) and one of the two cleanup webhooks are set
to Ignore; if Kyverno fails, the others block.
Save your progress? Sign up for free
Checkpoint
# on cp1
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations \
| grep -c tag16
kubectl get apiservice | grep -c False
kubectl top nodes | head -2Expected:
0
0
NAME CPU(cores) CPU(%) MEMORY(bytes) MEMORY(%)
cp1 1137m 28% 1928Mi 16%
Challenge
A colleague set up a sidecar injector and labeled the namespace
tag16-app for it. Since then, the Deployment
web stays at 0/1. Find the reason and get
web running without switching off the injector for other
namespaces.
# on cp1
cat > injektor.yaml <<'EOF'
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata: {name: sidecar-injektor}
webhooks:
- name: inject.sidecar.labor.test
admissionReviewVersions: [v1]
sideEffects: None
failurePolicy: Fail
clientConfig:
service: {name: injektor, namespace: tag16-app, path: /mutate}
namespaceSelector:
matchLabels: {sidecar: an}
rules:
- apiGroups: [""]
apiVersions: [v1]
operations: [CREATE]
resources: [pods]
EOF
kubectl apply -f injektor.yaml
kubectl create namespace tag16-app
kubectl label namespace tag16-app sidecar=an
kubectl -n tag16-app create deployment web --image=traefik/whoami:v1.11Hint: No Pod, so look one level up – and not every webhook is a validating webhook.
Solution
# on cp1
sleep 5
kubectl -n tag16-app get deploy web
kubectl -n tag16-app events --types=Warning | tail -1 | fold -w 86
kubectl get mutatingwebhookconfigurations
kubectl get ns tag16-app --show-labelsExpected (abridged):
NAME READY UP-TO-DATE AVAILABLE AGE
web 0/1 0 0 8s
3s (x11 over 8s) Warning FailedCreate ReplicaSet/web-589cc584cd Error creating
: Internal error occurred: failed calling webhook "inject.sidecar.labor.test": failed
to call webhook: Post "https://injektor.tag16-app.svc:443/mutate?timeout=10s": service
"injektor" not found
NAME WEBHOOKS AGE
kyverno-policy-mutating-webhook-cfg 1 167m
kyverno-resource-mutating-webhook-cfg 0 167m
kyverno-verify-mutating-webhook-cfg 1 167m
sidecar-injektor 1 9s
NAME STATUS AGE LABELS
tag16-app Active 10s kubernetes.io/metadata.name=tag16-app,sidecar=an
The injector matches via the namespaceSelector
sidecar: an (‘on’), and its Service does not exist. For
tag16-app, removing the label is enough; then roll out
again, because the ReplicaSet controller is stuck in backoff:
# on cp1
kubectl label namespace tag16-app sidecar-
kubectl -n tag16-app rollout restart deploy/web
kubectl -n tag16-app rollout status deploy/webExpected (abridged):
namespace/tag16-app unlabeled
deployment.apps/web restarted
deployment "web" successfully rolled out
Save your progress? Sign up for free
Cleanup
# on cp1
kubectl delete mutatingwebhookconfiguration sidecar-injektor
kubectl delete namespace tag16-app
kubectl get ns | grep -c tag16Expected:
mutatingwebhookconfiguration.admissionregistration.k8s.io "sidecar-injektor" deleted
namespace "tag16-app" deleted
0
Interview Note
failed calling webhook– what first? Look up the webhook name from the message in the validating and mutating configurations, then check the target’s Service and endpoints.FailorIgnore?Failfor security-relevant checks (better to block than to wave through),Ignoreonly with a shorttimeoutSeconds– otherwise the cluster becomes sluggish.- Namespace stuck in
Terminating? Read.status.conditions:NamespaceDeletionDiscoveryFailure→ repair the APIService,NamespaceFinalizersRemaining→ find the finalizer’s controller.
Enterprise Note
- Every webhook gets a
namespaceSelectorthat excludeskube-systemand your own platform namespaces; otherwise an outage blocks its own repair. - Run webhook backends with multiple replicas, a PodDisruptionBudget
and monitoring
(
apiserver_admission_webhook_rejection_count,apiserver_admission_webhook_admission_duration_seconds). - When removing operators and the metrics-server, also clean up their APIServices and CRDs – otherwise the next namespace deletion gets stuck.
- Never remove finalizers in bulk via a script; find the responsible controller first.