Skip to content
Hands-on IT IT Drills

Day 10: Plaintext Secrets

Goal: You pull a password out of a Pod spec, move it into a Secret, and check encryption at rest. · Duration: approx. 65 min · Prerequisite: Day 7

Briefing

Secrets rarely leak through cryptography and almost always through carelessness: a password as a plaintext env in the Pod spec, readable by anyone with get pods or via the kubelet proxy (Day 7). Add to that the question of how Secrets are actually stored in etcd – and whether a stolen etcd snapshot gives away all credentials at once. Today you plug both leaks: env → secretKeyRef and encryption at rest.

Drill 1: The misconfiguration – plaintext in the Pod spec

kubectl create namespace vault-lab

kubectl -n vault-lab apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: app, namespace: vault-lab}
spec:
  replicas: 1
  selector: {matchLabels: {app: app}}
  template:
    metadata: {labels: {app: app}}
    spec:
      containers:
      - name: c
        image: nginxinc/nginx-unprivileged:stable
        env:
        - {name: API_KEY, value: sk-lab-PLAINTEXT-123}
EOF
kubectl -n vault-lab rollout status deploy/app --timeout=90s

Save your progress? Sign up for free

Drill 2: Exploit – the value sits openly in the spec

kubectl -n vault-lab get pod -l app=app \
  -o jsonpath='{.items[0].spec.containers[0].env[0]}'; echo

Expected:

{"name":"API_KEY","value":"sk-lab-PLAINTEXT-123"}

Anyone allowed to read the Pod spec – via get pods, describe, a GitOps report, or nodes/proxy (Day 7) – reads the key. The password is also in plaintext in the Deployment manifest, so usually in Git as well.

Save your progress? Sign up for free

Drill 3: Fix – into the Secret, out of the spec

kubectl -n vault-lab create secret generic app-secret \
  --from-literal=api-key=sk-lab-PLAINTEXT-123

kubectl -n vault-lab patch deploy app --type=json -p='[{"op":"replace",
  "path":"/spec/template/spec/containers/0/env",
  "value":[{"name":"API_KEY","valueFrom":{"secretKeyRef":
    {"name":"app-secret","key":"api-key"}}}]}]'
kubectl -n vault-lab rollout status deploy/app --timeout=90s

Save your progress? Sign up for free

Drill 4: Proof – a reference instead of the value

kubectl -n vault-lab get pod -l app=app \
  -o jsonpath='{.items[0].spec.containers[0].env[0]}'; echo

Expected:

{"name":"API_KEY","valueFrom":{"secretKeyRef":{"key":"api-key","name":"app-secret"}}}

The spec now contains only the reference. Inside the container the value is still there – the application runs:

P=$(kubectl -n vault-lab get pod -l app=app -o jsonpath='{.items[0].metadata.name}')
kubectl -n vault-lab exec $P -- printenv API_KEY

Expected:

sk-lab-PLAINTEXT-123

Why is this better? The value is no longer in the Pod spec, not in the manifest, not in Git, and not visible via nodes/proxy. Access now depends on the RBAC permissions on the Secret (Day 2) – and on encryption at rest.

Save your progress? Sign up for free

Drill 5: Check encryption at rest

Secrets are stored in etcd. If etcd is unencrypted, a stolen snapshot (or access to the etcd files, Day 6) is enough to read all Secrets – Base64 is not encryption. RKE2 ships with encryption at rest; check the status on the server node:

sudo rke2 secrets-encrypt status

Expected (abridged):

Encryption Status: Enabled
Current Rotation Stage: start
Server Encryption Hashes: All hashes match

Enabled means: Secrets are stored in etcd encrypted with an AES key. If it says Disabled, you turn it on and re-encrypt the existing data:

sudo rke2 secrets-encrypt enable
# then restart the server node in a controlled way, then:
sudo rke2 secrets-encrypt reencrypt

Warning: reencrypt rewrites all Secrets and requires a clean restart sequence of the server nodes – in production only in a maintenance window and following the docs.

Save your progress? Sign up for free

Checkpoint

No Deployment carries a password as a plaintext env (the most common finding):

kubectl get deploy -A -o json | python3 -c '
import sys,json
for d in json.load(sys.stdin)["items"]:
    for c in d["spec"]["template"]["spec"].get("containers",[]):
        for e in c.get("env",[]) or []:
            if "value" in e and any(x in e["name"].upper()
                                    for x in ("PASS","SECRET","KEY","TOKEN")):
                print(d["metadata"]["namespace"]+"/"+d["metadata"]["name"], e["name"])'

Expected:

shop/db POSTGRES_PASSWORD

The single hit is the database from Day 0 – switch it to secretKeyRef too, and the list stays empty. Every hit is a plaintext Secret in a manifest.

Encryption is active:

sudo rke2 secrets-encrypt status | head -1

Expected: Encryption Status: Enabled

Challenge

secretKeyRef protects the spec, but the value still ends up as an environment variable in the container – and printenv or a crash dump shows it. Describe when a mounted Secret (as a file) is better, and mount app-secret under /etc/secret as a test.

Hint: volumes with secret: and a volumeMounts entry with readOnly: true.

Solution

kubectl -n vault-lab patch deploy app --type=json -p='[
  {"op":"add","path":"/spec/template/spec/volumes","value":[
    {"name":"sec","secret":{"secretName":"app-secret"}}]},
  {"op":"add","path":"/spec/template/spec/containers/0/volumeMounts","value":[
    {"name":"sec","mountPath":"/etc/secret","readOnly":true}]}]'
P=$(kubectl -n vault-lab get pod -l app=app -o jsonpath='{.items[0].metadata.name}')
kubectl -n vault-lab exec $P -- cat /etc/secret/api-key; echo

Mounted Secrets do not show up in printenv, can be rotated (with projected tokens) without restarting the Pod, and end up in logs or crash dumps less often than environment variables. For highly sensitive values they are the better choice.

Finally, clean up:

kubectl delete namespace vault-lab

Save your progress? Sign up for free

Enterprise Note

  • Never put Secrets in a plaintext env or in the manifest; use secretKeyRef or mount them, and keep manifests in Git only with encrypted Secrets (SOPS, Sealed Secrets) or an external store (OpenBao/Vault).
  • Make encryption at rest mandatory and rotate the key regularly (rke2 secrets-encrypt rotate + reencrypt).
  • etcd snapshots are as sensitive as etcd itself – store them encrypted and with restricted access.
  • For dynamic, short-lived credentials (DB logins, PKI), use a secret store like OpenBao instead of static Kubernetes Secrets – the topic of the course “OpenBao hands-on”.