Day 10: Plaintext Secrets
Goal: You pull a password out of a Pod spec, move it into a Secret, and check encryption at rest. · Duration: approx. 65 min · Prerequisite: Day 7
Briefing
Secrets rarely leak through cryptography and almost always through
carelessness: a password as a plaintext env in the Pod
spec, readable by anyone with get pods or via the kubelet
proxy (Day 7). Add to that the question of how Secrets are actually
stored in etcd – and whether a stolen etcd snapshot gives away all
credentials at once. Today you plug both leaks: env →
secretKeyRef and encryption at rest.
Drill 1: The misconfiguration – plaintext in the Pod spec
kubectl create namespace vault-lab
kubectl -n vault-lab apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: app, namespace: vault-lab}
spec:
replicas: 1
selector: {matchLabels: {app: app}}
template:
metadata: {labels: {app: app}}
spec:
containers:
- name: c
image: nginxinc/nginx-unprivileged:stable
env:
- {name: API_KEY, value: sk-lab-PLAINTEXT-123}
EOF
kubectl -n vault-lab rollout status deploy/app --timeout=90sSave your progress? Sign up for free
Drill 2: Exploit – the value sits openly in the spec
kubectl -n vault-lab get pod -l app=app \
-o jsonpath='{.items[0].spec.containers[0].env[0]}'; echoExpected:
{"name":"API_KEY","value":"sk-lab-PLAINTEXT-123"}
Anyone allowed to read the Pod spec – via get pods,
describe, a GitOps report, or nodes/proxy (Day
7) – reads the key. The password is also in plaintext in the Deployment
manifest, so usually in Git as well.
Save your progress? Sign up for free
Drill 3: Fix – into the Secret, out of the spec
kubectl -n vault-lab create secret generic app-secret \
--from-literal=api-key=sk-lab-PLAINTEXT-123
kubectl -n vault-lab patch deploy app --type=json -p='[{"op":"replace",
"path":"/spec/template/spec/containers/0/env",
"value":[{"name":"API_KEY","valueFrom":{"secretKeyRef":
{"name":"app-secret","key":"api-key"}}}]}]'
kubectl -n vault-lab rollout status deploy/app --timeout=90sSave your progress? Sign up for free
Drill 4: Proof – a reference instead of the value
kubectl -n vault-lab get pod -l app=app \
-o jsonpath='{.items[0].spec.containers[0].env[0]}'; echoExpected:
{"name":"API_KEY","valueFrom":{"secretKeyRef":{"key":"api-key","name":"app-secret"}}}
The spec now contains only the reference. Inside the container the value is still there – the application runs:
P=$(kubectl -n vault-lab get pod -l app=app -o jsonpath='{.items[0].metadata.name}')
kubectl -n vault-lab exec $P -- printenv API_KEYExpected:
sk-lab-PLAINTEXT-123
Why is this better? The value is no longer in the Pod spec, not in the manifest, not in Git, and not visible via
nodes/proxy. Access now depends on the RBAC permissions on the Secret (Day 2) – and on encryption at rest.
Save your progress? Sign up for free
Drill 5: Check encryption at rest
Secrets are stored in etcd. If etcd is unencrypted, a stolen snapshot (or access to the etcd files, Day 6) is enough to read all Secrets – Base64 is not encryption. RKE2 ships with encryption at rest; check the status on the server node:
sudo rke2 secrets-encrypt statusExpected (abridged):
Encryption Status: Enabled
Current Rotation Stage: start
Server Encryption Hashes: All hashes match
Enabled means: Secrets are stored in etcd encrypted with
an AES key. If it says Disabled, you turn it on and
re-encrypt the existing data:
sudo rke2 secrets-encrypt enable
# then restart the server node in a controlled way, then:
sudo rke2 secrets-encrypt reencrypt
Warning:
reencryptrewrites all Secrets and requires a clean restart sequence of the server nodes – in production only in a maintenance window and following the docs.
Save your progress? Sign up for free
Checkpoint
No Deployment carries a password as a plaintext env (the
most common finding):
kubectl get deploy -A -o json | python3 -c '
import sys,json
for d in json.load(sys.stdin)["items"]:
for c in d["spec"]["template"]["spec"].get("containers",[]):
for e in c.get("env",[]) or []:
if "value" in e and any(x in e["name"].upper()
for x in ("PASS","SECRET","KEY","TOKEN")):
print(d["metadata"]["namespace"]+"/"+d["metadata"]["name"], e["name"])'Expected:
shop/db POSTGRES_PASSWORD
The single hit is the database from Day 0 – switch it to
secretKeyRef too, and the list stays empty. Every hit is a
plaintext Secret in a manifest.
Encryption is active:
sudo rke2 secrets-encrypt status | head -1Expected: Encryption Status: Enabled
Challenge
secretKeyRef protects the spec, but the value still ends
up as an environment variable in the container – and
printenv or a crash dump shows it. Describe when a
mounted Secret (as a file) is better, and mount
app-secret under /etc/secret as a test.
Hint: volumes with secret:
and a volumeMounts entry with
readOnly: true.
Solution
kubectl -n vault-lab patch deploy app --type=json -p='[
{"op":"add","path":"/spec/template/spec/volumes","value":[
{"name":"sec","secret":{"secretName":"app-secret"}}]},
{"op":"add","path":"/spec/template/spec/containers/0/volumeMounts","value":[
{"name":"sec","mountPath":"/etc/secret","readOnly":true}]}]'
P=$(kubectl -n vault-lab get pod -l app=app -o jsonpath='{.items[0].metadata.name}')
kubectl -n vault-lab exec $P -- cat /etc/secret/api-key; echoMounted Secrets do not show up in printenv, can be
rotated (with projected tokens) without restarting the Pod, and end up
in logs or crash dumps less often than environment variables. For highly
sensitive values they are the better choice.
Finally, clean up:
kubectl delete namespace vault-labSave your progress? Sign up for free
Enterprise Note
- Never put Secrets in a plaintext
envor in the manifest; usesecretKeyRefor mount them, and keep manifests in Git only with encrypted Secrets (SOPS, Sealed Secrets) or an external store (OpenBao/Vault). - Make encryption at rest mandatory and rotate the key regularly
(
rke2 secrets-encrypt rotate+reencrypt). - etcd snapshots are as sensitive as etcd itself – store them encrypted and with restricted access.
- For dynamic, short-lived credentials (DB logins, PKI), use a secret store like OpenBao instead of static Kubernetes Secrets – the topic of the course “OpenBao hands-on”.