Skip to content
Hands-on IT IT Drills

Day 16: Webhooks, APIServices and stuck namespaces

Goal: You find the webhook behind a rejected request, recognize an unavailable aggregated API and get a namespace out of Terminating without orphaning objects in etcd. · Duration: approx. 60 min · Prerequisite: Day 15

Briefing

  • The API server calls admission webhooks on every matching write request. failurePolicy: Fail rejects when the webhook does not respond, Ignore lets the request through – if the backend hangs, only after timeoutSeconds.
  • Aggregated APIs (APIService) forward entire API groups to a Service. If the backend is missing, the cluster’s discovery is incomplete.
  • A namespace only disappears once everything in it has been deleted. What it is stuck on is in its conditions.
Fault Symptom Where the answer is
Webhook without backend failed calling webhook Webhook configuration, endpoints
Slow webhook Every request takes 10 s timeoutSeconds, API server log
APIService gone False (MissingEndpoints) kubectl get apiservice
Namespace stuck Terminating forever .status.conditions

Drill 1: A webhook without a backend

Break it: A validating webhook for ConfigMaps in tag16 whose Service has no Pod:

# on cp1
mkdir -p ~/debug/tag16 && cd ~/debug/tag16
kubectl create namespace tag16
cat > pruefer.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata: {name: pruefer, namespace: tag16}
spec:
  selector: {app: pruefer}
  ports: [{port: 443, targetPort: 8443}]
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata: {name: tag16-pruefung}
webhooks:
- name: pruefung.tag16.labor.test
  admissionReviewVersions: [v1]
  sideEffects: None
  failurePolicy: Fail
  timeoutSeconds: 5
  clientConfig:
    service: {name: pruefer, namespace: tag16, path: /validate}
  namespaceSelector:
    matchLabels: {kubernetes.io/metadata.name: tag16}
  rules:
  - apiGroups: [""]
    apiVersions: [v1]
    operations: [CREATE, UPDATE]
    resources: [configmaps]
EOF
kubectl apply -f pruefer.yaml

Expected:

namespace/tag16 created
service/pruefer created
validatingwebhookconfiguration.admissionregistration.k8s.io/tag16-pruefung created

The namespaceSelector limits the damage to tag16 – without it, every namespace would be affected.

Symptom:

# on cp1
kubectl -n tag16 create configmap test --from-literal=a=b

Expected (line wrapped):

error: failed to create configmap: Internal error occurred: failed calling webhook
  "pruefung.tag16.labor.test": failed to call webhook: Post
  "https://pruefer.tag16.svc:443/validate?timeout=5s": no endpoints available for
  service "pruefer"

Investigate: The message names the webhook, not the configuration. You find both in the list:

# on cp1
kubectl get validatingwebhookconfigurations
J='{range .items[*]}{.metadata.name}{range .webhooks[*]} {.name} {.failurePolicy}'
kubectl get validatingwebhookconfigurations -o jsonpath="$J{end}{\"\n\"}{end}" \
  | grep tag16
kubectl get validatingwebhookconfiguration tag16-pruefung \
  -o jsonpath='{.webhooks[0].clientConfig.service}{"\n"}'
kubectl -n tag16 get endpointslice

Expected (abridged):

NAME                                            WEBHOOKS   AGE
kyverno-cel-exception-validating-webhook-cfg    1          163m
...
kyverno-ttl-validating-webhook-cfg              1          164m
tag16-pruefung                                  1          7s
tag16-pruefung pruefung.tag16.labor.test Fail
{"name":"pruefer","namespace":"tag16","path":"/validate","port":443}
NAME            ADDRESSTYPE   PORTS     ENDPOINTS   AGE
pruefer-kpccn   IPv4          <unset>   <unset>     8s

Cause: The API server has to ask the webhook for every ConfigMap in tag16. The Service has no endpoints (Day 7), the call fails, and because of failurePolicy: Fail the API server rejects the request. You know the same pattern from Kyverno (Day 12) – here without Kyverno.

Fix: The right fix is to get the backend running again. If it is gone for good, the configuration should be deleted – that comes in Drill 2. First the backend: a Pod with the matching label.

# on cp1
kubectl -n tag16 run pruefer --image=nicolaka/netshoot:v0.15 --labels=app=pruefer \
  -- nc -lk -p 8443
kubectl -n tag16 wait --for=condition=Ready pod/pruefer --timeout=60s
kubectl -n tag16 get endpointslice

Expected:

pod/pruefer created
pod/pruefer condition met
NAME            ADDRESSTYPE   PORTS   ENDPOINTS     AGE
pruefer-kpccn   IPv4          8443    10.42.1.133   13s

Verify: The webhook is reachable – and the request still fails, just differently:

# on cp1
time kubectl -n tag16 create configmap test --from-literal=a=b

Expected (line wrapped):

error: failed to create configmap: Internal error occurred: failed calling webhook
  "pruefung.tag16.labor.test": failed to call webhook: Post
  "https://pruefer.tag16.svc:443/validate?timeout=5s": context deadline exceeded

real    0m5.236s

nc accepts the connection but does not speak TLS and never answers. A backend that “is running” is not yet a working webhook. Drill 2 shows what happens when someone switches to Ignore because of this.

Save your progress? Sign up for free

Drill 2: The slow webhook

Break it: Someone “defuses” the webhook: Ignore instead of Fail, and more patience in return.

# on cp1
kubectl patch validatingwebhookconfiguration tag16-pruefung --type=json -p '[
  {"op": "replace", "path": "/webhooks/0/failurePolicy", "value": "Ignore"},
  {"op": "replace", "path": "/webhooks/0/timeoutSeconds", "value": 10}]'

Symptom:

# on cp1
time kubectl -n tag16 create configmap test2 --from-literal=a=b

Expected:

configmap/test2 created

real    0m10.210s

The ConfigMap is created – after ten seconds. Every change in tag16 now takes that long; with Pods as the target, this slows down every deployment and every restart.

Investigate:

# on cp1
kubectl -n kube-system logs kube-apiserver-cp1 --since=1m \
  | grep -m1 'failing open' | cut -c1-200 | fold -w 86

Expected:

W1120 12:01:52.403866       1 dispatcher.go:205] Failed calling webhook, failing open
pruefung.tag16.labor.test: failed calling webhook "pruefung.tag16.labor.test": failed
to call webhook: Post "https

Cause: Ignore does not heal the cluster, it only makes it sluggish: on every request the API server waits timeoutSeconds and then lets it through (“failing open”). The check never actually takes place.

Fix: The backend is not a webhook and is not coming back – the configuration gets deleted. Webhook configurations themselves never pass through webhooks, so you can always change or delete them:

# on cp1
kubectl delete validatingwebhookconfiguration tag16-pruefung
kubectl -n tag16 delete pod pruefer

Verify:

# on cp1
time kubectl -n tag16 create configmap test3 --from-literal=a=b

Expected:

configmap/test3 created

real    0m0.264s

Save your progress? Sign up for free

Drill 3: The aggregated API is missing

Break it: RKE2’s metrics-server is switched off “temporarily”:

# on cp1
kubectl -n kube-system scale deploy rke2-metrics-server --replicas=0

Symptom (after about 20 seconds):

# on cp1
sleep 20
kubectl top nodes

Expected:

error: Metrics API not available

Investigate: kubectl top does not ask the metrics-server directly but the API group metrics.k8s.io, which the API server forwards via an APIService:

# on cp1
kubectl get apiservice | grep -v Local | tr -s ' '
kubectl get apiservice v1beta1.metrics.k8s.io \
  -o jsonpath='{.status.conditions[0].message}{"\n"}' | fold -w 86

Expected:

NAME SERVICE AVAILABLE AGE
v1beta1.metrics.k8s.io kube-system/rke2-metrics-server False (MissingEndpoints) 5h30m
endpointslices for service/rke2-metrics-server in "kube-system" have no addresses with
 port name "https"

All other APIServices show Local – the API server serves them itself.

Cause: The APIService points to a Service without endpoints. This does not only affect kubectl top and the HPA (Day 17): the discovery of the entire cluster is now incomplete – with consequences that Drill 4 shows. That is why the metrics-server stays off until then.

Save your progress? Sign up for free

Drill 4: The namespace stays in Terminating

Break it: A ConfigMap with a finalizer that no controller will ever remove:

# on cp1
kubectl -n tag16 patch configmap test3 --type=merge \
  -p '{"metadata":{"finalizers":["labor.test/aufraeumen"]}}'
kubectl delete namespace tag16 --wait=false

Expected:

configmap/test3 patched
namespace "tag16" deleted

--wait=false, because otherwise kubectl waits forever for the deletion to finish.

Symptom:

# on cp1
sleep 15
kubectl get ns tag16

Expected:

NAME    STATUS        AGE
tag16   Terminating   2m13s

AGE counts from creation in Drill 1, not from deletion.

Investigate: The namespace controller writes what it is stuck on into the conditions:

# on cp1
kubectl get ns tag16 -o json | jq -r '.status.conditions[]
  | select(.status=="True") | "\(.type): \(.message)"' | fold -w 86

Expected:

NamespaceDeletionDiscoveryFailure: Discovery failed for some groups, 1 failing: unable
 to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: stale GroupVers
ion discovery: metrics.k8s.io/v1beta1
NamespaceContentRemaining: Some resources are remaining: configmaps. has 1 resource in
stances
NamespaceFinalizersRemaining: Some content in the namespace has finalizers remaining:
labor.test/aufraeumen in 1 resource instances

Cause: Two things hold the namespace. The finalizer on the ConfigMap – nobody removes it. And the incomplete discovery: the controller deletes everything it finds, but does not know whether objects in tag16 still exist in metrics.k8s.io – and to be safe it does not release the namespace. Because the APIService applies cluster-wide, every namespace you delete during this time gets stuck.

Fix: First only the finalizer – with a real finalizer, you check beforehand which controller sets it and why it does not remove it:

# on cp1
kubectl -n tag16 patch configmap test3 --type=merge \
  -p '{"metadata":{"finalizers":null}}'
sleep 20
kubectl -n tag16 get configmaps
kubectl get ns tag16
kubectl get ns tag16 -o json | jq -r '.status.conditions[]
  | select(.status=="True") | .type'

Expected:

configmap/test3 patched
No resources found in tag16 namespace.
NAME    STATUS        AGE
tag16   Terminating   2m41s
NamespaceDeletionDiscoveryFailure

The ConfigMap is gone, but the namespace is still stuck – on discovery. So repair the aggregated API:

# on cp1
kubectl -n kube-system scale deploy rke2-metrics-server --replicas=1
kubectl -n kube-system rollout status deploy/rke2-metrics-server
T0=$(date +%s)
until ! kubectl get ns tag16 >/dev/null 2>&1; do sleep 2; done
echo "weg nach $(( $(date +%s) - T0 )) s"

Expected:

deployment.apps/rke2-metrics-server scaled
deployment "rke2-metrics-server" successfully rolled out
weg nach 35 s

The last line means “gone after 35 s”.

Verify:

# on cp1
kubectl get apiservice v1beta1.metrics.k8s.io
kubectl get ns tag16

Expected:

NAME                     SERVICE                           AVAILABLE   AGE
v1beta1.metrics.k8s.io   kube-system/rke2-metrics-server   True        5h32m
Error from server (NotFound): namespaces "tag16" not found

The namespace controller retries with backoff; hence the good half minute.

Warning: A tip circulating online is to remove the kubernetes finalizer on the namespace itself via /finalize. The namespace is then gone, but its objects are not necessarily: they remain orphaned in etcd and reappear as soon as someone creates a namespace with the same name. Always fix the cause.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show all validating and mutating webhooks with their failurePolicy. Target time: 60 s
  2. Show all APIServices that are not available. Target time: 20 s
  3. Show all namespaces in Terminating. Target time: 20 s

Solution

# on cp1
J='{range .items[*]}{range .webhooks[*]}{.name} {.failurePolicy}{"\n"}{end}{end}'
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations \
  -o jsonpath="$J" | sort | uniq -c | sort -rn
kubectl get apiservice | grep False; echo "rc=$?"
kubectl get ns --field-selector=status.phase=Terminating

Expected:

      3 kyverno-svc.kyverno.svc Fail
      1 validate-policy.kyverno.svc Fail
      1 protect-generation-labels.kyverno.svc Fail
      1 mutate-policy.kyverno.svc Fail
      1 monitor-webhooks.kyverno.svc Ignore
      1 kyverno-cleanup-controller.kyverno.svc Ignore
      1 kyverno-cleanup-controller.kyverno.svc Fail
rc=1
No resources found

All remaining webhooks come from Kyverno (Day 12) – almost all of them with Fail. Only monitor-webhooks (Kyverno’s self-monitoring) and one of the two cleanup webhooks are set to Ignore; if Kyverno fails, the others block.

Save your progress? Sign up for free

Checkpoint

# on cp1
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations \
  | grep -c tag16
kubectl get apiservice | grep -c False
kubectl top nodes | head -2

Expected:

0
0
NAME      CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)
cp1       1137m        28%      1928Mi          16%

Challenge

A colleague set up a sidecar injector and labeled the namespace tag16-app for it. Since then, the Deployment web stays at 0/1. Find the reason and get web running without switching off the injector for other namespaces.

# on cp1
cat > injektor.yaml <<'EOF'
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata: {name: sidecar-injektor}
webhooks:
- name: inject.sidecar.labor.test
  admissionReviewVersions: [v1]
  sideEffects: None
  failurePolicy: Fail
  clientConfig:
    service: {name: injektor, namespace: tag16-app, path: /mutate}
  namespaceSelector:
    matchLabels: {sidecar: an}
  rules:
  - apiGroups: [""]
    apiVersions: [v1]
    operations: [CREATE]
    resources: [pods]
EOF
kubectl apply -f injektor.yaml
kubectl create namespace tag16-app
kubectl label namespace tag16-app sidecar=an
kubectl -n tag16-app create deployment web --image=traefik/whoami:v1.11

Hint: No Pod, so look one level up – and not every webhook is a validating webhook.

Solution

# on cp1
sleep 5
kubectl -n tag16-app get deploy web
kubectl -n tag16-app events --types=Warning | tail -1 | fold -w 86
kubectl get mutatingwebhookconfigurations
kubectl get ns tag16-app --show-labels

Expected (abridged):

NAME   READY   UP-TO-DATE   AVAILABLE   AGE
web    0/1     0            0           8s
3s (x11 over 8s)   Warning   FailedCreate   ReplicaSet/web-589cc584cd   Error creating
: Internal error occurred: failed calling webhook "inject.sidecar.labor.test": failed
to call webhook: Post "https://injektor.tag16-app.svc:443/mutate?timeout=10s": service
 "injektor" not found
NAME                                    WEBHOOKS   AGE
kyverno-policy-mutating-webhook-cfg     1          167m
kyverno-resource-mutating-webhook-cfg   0          167m
kyverno-verify-mutating-webhook-cfg     1          167m
sidecar-injektor                        1          9s
NAME        STATUS   AGE   LABELS
tag16-app   Active   10s   kubernetes.io/metadata.name=tag16-app,sidecar=an

The injector matches via the namespaceSelector sidecar: an (‘on’), and its Service does not exist. For tag16-app, removing the label is enough; then roll out again, because the ReplicaSet controller is stuck in backoff:

# on cp1
kubectl label namespace tag16-app sidecar-
kubectl -n tag16-app rollout restart deploy/web
kubectl -n tag16-app rollout status deploy/web

Expected (abridged):

namespace/tag16-app unlabeled
deployment.apps/web restarted
deployment "web" successfully rolled out

Save your progress? Sign up for free

Cleanup

# on cp1
kubectl delete mutatingwebhookconfiguration sidecar-injektor
kubectl delete namespace tag16-app
kubectl get ns | grep -c tag16

Expected:

mutatingwebhookconfiguration.admissionregistration.k8s.io "sidecar-injektor" deleted
namespace "tag16-app" deleted
0

Interview Note

  • failed calling webhook – what first? Look up the webhook name from the message in the validating and mutating configurations, then check the target’s Service and endpoints.
  • Fail or Ignore? Fail for security-relevant checks (better to block than to wave through), Ignore only with a short timeoutSeconds – otherwise the cluster becomes sluggish.
  • Namespace stuck in Terminating? Read .status.conditions: NamespaceDeletionDiscoveryFailure → repair the APIService, NamespaceFinalizersRemaining → find the finalizer’s controller.

Enterprise Note

  • Every webhook gets a namespaceSelector that excludes kube-system and your own platform namespaces; otherwise an outage blocks its own repair.
  • Run webhook backends with multiple replicas, a PodDisruptionBudget and monitoring (apiserver_admission_webhook_rejection_count, apiserver_admission_webhook_admission_duration_seconds).
  • When removing operators and the metrics-server, also clean up their APIServices and CRDs – otherwise the next namespace deletion gets stuck.
  • Never remove finalizers in bulk via a script; find the responsible controller first.