Skip to content
Hands-on IT IT Drills

Day 4: Configuration – ConfigMaps and Secrets

Goal: No more key and no more URL in the manifest – configuration comes from a ConfigMap, the API key from a Secret, and you rotate it without downtime. · Duration: approx. 75 min · Prerequisite: Day 3

Briefing

  • A ConfigMap holds key-value pairs or whole files. A Secret is the same for confidential values – stored only base64-encoded, not encrypted.
  • A container gets both as an environment variable (envFrom, valueFrom) or as a file via a volume.
  • The kubelet updates files from a volume after a change, but not variables. The notes services only read their configuration at startup anyway.
  • Secrets never end up as YAML in the repo; they are created by the script k8s/secrets.sh.
 ConfigMap notes-config ──envFrom────────────► worker  (NOTES_API_URL, INTERVAL)
                        └─valueFrom──────────► web     (NOTES_API_URL)
 Secret notes-api ──volume /etc/notes/api-key► api, worker (*_FILE)
                  └─secretKeyRef─────────────► web     (NOTES_API_KEY)

Drill 1: The Key in Plain Text

Start the day and fetch the state of Day 3:

source ~/k8s-lab/setup/env.sh
mkdir -p ~/k8s-lab/tag04 && cd ~/k8s-lab/tag04
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag03/notes/." .

Where is the API key today?

grep -rn 'lab-api-key' k8s/
c='NAME:.metadata.name,ENV:.spec.template.spec.containers[0].env[*].name'
kubectl get deploy -o custom-columns="$c"

Expected:

k8s/worker.yaml:27:              value: lab-api-key-123
k8s/api.yaml:31:              value: lab-api-key-123
k8s/web.yaml:27:              value: lab-api-key-123
NAME     ENV
api      NOTES_API_KEY
web      NOTES_API_URL,NOTES_API_KEY
worker   NOTES_API_URL,NOTES_API_KEY

The same value three times, three times in plain text; anyone with read access to the repo knows it. And NOTES_API_URL appears twice. Today you pull both out.

Save your progress? Sign up for free

Drill 2: ConfigMap Three Ways

kubectl create configmap builds a ConfigMap from literals or files. With --dry-run=client -o yaml you only see the result, without creating anything:

kubectl create configmap demo --from-literal=INTERVAL=10 --from-literal=MODE=lab \
  --dry-run=client -o yaml
printf 'level=debug\nformat=text\n' > app.properties
kubectl create configmap demo --from-file=app.properties --dry-run=client -o yaml
kubectl create configmap demo --from-env-file=app.properties --dry-run=client -o yaml
rm app.properties

Expected:

apiVersion: v1
data:
  INTERVAL: "10"
  MODE: lab
kind: ConfigMap
metadata:
  name: demo
apiVersion: v1
data:
  app.properties: |
    level=debug
    format=text
kind: ConfigMap
metadata:
  name: demo
apiVersion: v1
data:
  format: text
  level: debug
kind: ConfigMap
metadata:
  name: demo

--from-file stores the whole file under its name, --from-env-file splits it into keys. For the notes app the third way is enough – YAML directly, versioned in the repo:

cat > k8s/config.yaml <<'EOF'
# k8s/config.yaml – state at end of Day 4
apiVersion: v1
kind: ConfigMap
metadata:
  name: notes-config
  labels:
    app.kubernetes.io/part-of: notes
data:
  NOTES_API_URL: http://api:8080
  INTERVAL: "10"
EOF
kubectl apply --dry-run=server -f k8s/config.yaml
kubectl apply -f k8s/config.yaml
kubectl get configmap notes-config -o jsonpath='{.data}'; echo

Expected:

configmap/notes-config created (server dry run)
configmap/notes-config created
{"INTERVAL":"10","NOTES_API_URL":"http://api:8080"}

"10" is in quotes: values of a ConfigMap are always strings, and the API server rejects a bare 10.

Save your progress? Sign up for free

Drill 3: Secret via Script

cat > k8s/secrets.sh <<'EOF'
#!/usr/bin/env bash
# k8s/secrets.sh – state at end of Day 4: creates the Secrets (idempotent, lab values only)
set -euo pipefail
kubectl -n notes create secret generic notes-api \
  --from-literal=api-key="${NOTES_API_KEY:-lab-api-key-123}" \
  --dry-run=client -o yaml | kubectl apply -f -
EOF
chmod +x k8s/secrets.sh
bash -n k8s/secrets.sh && ./k8s/secrets.sh
./k8s/secrets.sh
kubectl get secret notes-api

Expected (abridged, names, IPs and times will differ on your machine):

secret/notes-api created
secret/notes-api unchanged
NAME        TYPE     DATA   AGE
notes-api   Opaque   1      1s

create … --dry-run=client -o yaml only generates the object, kubectl apply -f - creates it or reconciles it. That is why the script can run any number of times – create alone would fail the second time with AlreadyExists. The value comes from the environment; the repo only contains the lab default.

How secret is a Secret?

kubectl get secret notes-api -o yaml | grep -A1 '^data'
kubectl get secret notes-api -o jsonpath='{.data.api-key}' | base64 -d; echo

Expected:

data:
  api-key: bGFiLWFwaS1rZXktMTIz
lab-api-key-123

Security: base64 is an encoding, not encryption. Anyone allowed to get secret knows the value. And kubectl apply additionally creates the annotation kubectl.kubernetes.io/last-applied-configuration with the same value. Only RBAC and encryption in etcd offer protection (Enterprise Note).

Save your progress? Sign up for free

Drill 4: envFrom, valueFrom and a Secret as a File

The three Deployments now get their values from the ConfigMap and the Secret. Take the complete files from the companion code:

cp "$LAB/code/tag04/notes/k8s/"{api,web,worker}.yaml k8s/

Three excerpts show the differences. worker uses envFrom to fetch all keys of the ConfigMap, and the key as a file:

# k8s/worker.yaml (excerpt) – state at end of Day 4
    spec:
      containers:
        - name: worker
          image: notes-worker:0.1.0
          envFrom:
            - configMapRef:
                name: notes-config
          env:
            - name: NOTES_API_KEY_FILE
              value: /etc/notes/api-key
          volumeMounts:
            - name: api-key
              mountPath: /etc/notes
              readOnly: true
      volumes:
        - name: api-key
          secret:
            secretName: notes-api

api gets the same volume and NOTES_API_KEY_FILE, but no ConfigMap. web uses valueFrom to fetch individual keys selectively – from the Secret too, because notes-web only knows NOTES_API_KEY:

# k8s/web.yaml (excerpt) – state at end of Day 4
          env:
            - name: NOTES_API_URL
              valueFrom:
                configMapKeyRef:
                  name: notes-config
                  key: NOTES_API_URL
            - name: NOTES_API_KEY
              valueFrom:
                secretKeyRef:
                  name: notes-api
                  key: api-key
grep -rn 'lab-api-key' k8s/*.yaml; echo "Exit: $?"
kubectl apply --dry-run=server -f k8s/ >/dev/null && kubectl apply -f k8s/
for d in api web worker; do kubectl rollout status deploy/$d >/dev/null; done
kubectl logs deploy/api | head -2
kubectl exec deploy/worker -- env | grep -E '^(NOTES|INTERVAL)' | sort
kubectl exec deploy/web -- printenv NOTES_API_URL NOTES_API_KEY

Expected (abridged):

Exit: 1
deployment.apps/api configured
service/api unchanged
configmap/notes-config unchanged
namespace/notes unchanged
deployment.apps/web configured
service/web unchanged
deployment.apps/worker configured
time=… level=INFO msg="api key loaded" source=file
time=… level=INFO msg="store ready" store=memory
INTERVAL=10
NOTES_API_KEY_FILE=/etc/notes/api-key
NOTES_API_URL=http://api:8080
http://api:8080
lab-api-key-123

source=file – api read the key from the file, not from a variable. In worker the key appears nowhere in the environment; in web it does. What does the file look like?

kubectl exec deploy/worker -- ls -la /etc/notes
curl -s -X POST localhost:30080/api/notes -H 'Content-Type: application/json' \
  -d '{"text":"Key kommt aus dem Secret"}'; echo

Expected (abridged):

drwxrwxrwt 3 root root  100 Oct 30 14:34 .
drwxr-xr-x 1 root root 4096 Oct 30 14:34 ..
drwxr-xr-x 2 root root   60 Oct 30 14:34 ..2026_10_30_14_34_29.420101691
lrwxrwxrwx 1 root root   31 Oct 30 14:34 ..data -> ..2026_10_30_14_34_29.420101691
lrwxrwxrwx 1 root root   14 Oct 30 14:34 api-key -> ..data/api-key
{"id":"13bf6e3a-…","text":"Key kommt aus dem Secret"}

api-key is a symlink via ..data to a timestamped directory. On a change, the kubelet writes a new directory and repoints ..data in a single step – readers never see half a file. The volume is a tmpfs; the key never lands on the node’s disk.

Way Field Updates itself?
all keys as variables envFrom no
one key as a variable valueFrom (configMapKeyRef, secretKeyRef) no
keys as files volumes (configMap, secret) yes, after approx. 60–90 s

Save your progress? Sign up for free

Drill 5: Change the Configuration

The worker should count every 5 seconds instead of every 10:

kubectl patch configmap notes-config --type merge -p '{"data":{"INTERVAL":"5"}}'
sleep 20
kubectl exec deploy/worker -- printenv INTERVAL
kubectl logs deploy/worker --timestamps | tail -2

Expected (abridged):

configmap/notes-config patched
10
2026-10-30T14:34:57.64…Z summary: 1 notes, 5 words
2026-10-30T14:35:07.70…Z summary: 1 notes, 5 words

Nothing happens. An environment variable is set when the container starts and stays. You need new Pods:

kubectl rollout restart deploy/worker
kubectl rollout status deploy/worker >/dev/null
kubectl exec deploy/worker -- printenv INTERVAL
sleep 11; kubectl logs deploy/worker --timestamps | tail -3
kubectl get deploy worker -o jsonpath='{.spec.template.metadata.annotations}'; echo

Expected (abridged):

deployment.apps/worker restarted
5
2026-10-30T14:35:18.96…Z worker started, api=http://api:8080, interval=5s
2026-10-30T14:35:19.75…Z summary: 0 notes, 0 words
2026-10-30T14:35:24.78…Z summary: 1 notes, 5 words
{"kubectl.kubernetes.io/restartedAt":"2026-10-30T14:35:12Z"}

rollout restart only puts an annotation with the time into the Pod template. The template changes, so the Deployment rolls out as it would for a new image – with the same strategy, without downtime. By the way, the 0 notes are the in-memory replicas from Day 3: the worker hit a different api.

Back to the state in the repo:

kubectl apply -f k8s/config.yaml
kubectl rollout restart deploy/worker && kubectl rollout status deploy/worker

Expected (abridged):

configmap/notes-config configured
deployment.apps/worker restarted
deployment "worker" successfully rolled out

Save your progress? Sign up for free

Drill 6: Rotate the Key

The key has leaked; a new one is needed. Create the new Secret and watch when the file in the worker switches over:

NOTES_API_KEY=lab-api-key-456 ./k8s/secrets.sh
s=$SECONDS
until [ "$(kubectl exec deploy/worker -- cat /etc/notes/api-key)" \
  = lab-api-key-456 ]; do sleep 2; done
echo "Datei nach $((SECONDS - s)) s aktualisiert"
kubectl exec deploy/web -- printenv NOTES_API_KEY
curl -s -X POST localhost:30080/api/notes -H 'Content-Type: application/json' \
  -d '{"text":"vor dem Neustart"}'; echo

Expected (abridged, the seconds differ):

secret/notes-api configured
Datei nach 72 s aktualisiert
lab-api-key-123
{"id":"7ba3655d-…","text":"vor dem Neustart"}

The file follows after about a minute (the kubelet’s sync interval plus cache); the variable in web never does. The POST still works: api read the old key at startup, and web sends the old one along. Now the trap – restart only api:

kubectl rollout restart deploy/api && kubectl rollout status deploy/api >/dev/null
curl -s -X POST localhost:30080/api/notes -H 'Content-Type: application/json' \
  -d '{"text":"nur api neu"}'; echo

Expected:

deployment.apps/api restarted
{"error":"invalid api key"}

api now knows …-456, web still sends …-123. A rotation is only finished when all readers have been restarted:

kubectl rollout restart deploy/web deploy/worker
kubectl rollout status deploy/web >/dev/null
kubectl rollout status deploy/worker >/dev/null
sleep 3
curl -s -X POST localhost:30080/api/notes -H 'Content-Type: application/json' \
  -d '{"text":"alle neu"}'; echo

Expected (abridged):

deployment.apps/web restarted
deployment.apps/worker restarted
{"id":"819e6086-…","text":"alle neu"}

Warning: Without the sleep 3, curl occasionally reports nothing right after the rollout of web (exit 56, connection reset). An old web Pod was still getting traffic while it was already being terminated. Day 5 shows what helps against that.

End of the drill – back to the lab key so that the final state is right:

./k8s/secrets.sh
kubectl rollout restart deploy/api deploy/web deploy/worker
for d in api web worker; do kubectl rollout status deploy/$d >/dev/null; done
sleep 3; kubectl exec deploy/web -- printenv NOTES_API_KEY

Expected:

secret/notes-api configured
deployment.apps/api restarted
deployment.apps/web restarted
deployment.apps/worker restarted
lab-api-key-123

Save your progress? Sign up for free

Checkpoint

kubectl exec deploy/worker -- printenv INTERVAL NOTES_API_URL

Expected: both values from the ConfigMap notes-config.

10
http://api:8080
kubectl logs deploy/api | grep 'api key'

Expected: source=file; the key comes from the Secret volume.

Found 3 pods, using pod/api-9bcdb694-zx2cx
time=… level=INFO msg="api key loaded" source=file
grep -rl lab-api-key k8s/

Expected: only secrets.sh; no manifest contains the key.

k8s/secrets.sh
curl -s -X POST localhost:30080/api/notes -H 'Content-Type: application/json' \
  -d '{"text":"Kontrollpunkt Tag 4"}'

Expected: web passes the key on, and the api creates the note.

{"id":"dbba8751-…","text":"Kontrollpunkt Tag 4"}

Challenge

A colleague makes a typo in the key name of the Secret for web (apikey instead of api-key) and applies it with kubectl patch. What do you see on the Pods, why does the page stay reachable, and how do you find the cause?

Hint: Look at the events of the new Pod – and think about what the RollingUpdate strategy does with the old ones.

Solution

kubectl patch deploy web --type json -p '[{"op":"replace",
  "path":"/spec/template/spec/containers/0/env/1/valueFrom/secretKeyRef/key",
  "value":"apikey"}]'
sleep 15
kubectl get pods -l app.kubernetes.io/name=web
kubectl rollout status deploy/web --timeout 10s; echo "Exit: $?"
kubectl get events --field-selector type=Warning,reason=Failed \
  -o custom-columns=MESSAGE:.message | tail -1
curl -s -o /dev/null -w '%{http_code}\n' localhost:30080/api/notes

Expected (abridged):

deployment.apps/web patched
NAME                   READY   STATUS                       RESTARTS   AGE
web-5f7797cd47-88d5f   1/1     Running                      0          33s
web-5f7797cd47-qmp95   1/1     Running                      0          36s
web-8495c45c4d-wnjxn   0/1     CreateContainerConfigError   0          16s
Waiting for deployment "web" rollout to finish: 1 out of 2 new replicas have been…
error: timed out waiting for the condition
Exit: 1
Error: couldn't find key apikey in Secret notes/notes-api
200

If a key or a whole Secret is missing, the kubelet can’t build the environment: CreateContainerConfigError. The old Pods keep running because the strategy first wants to see a new Pod ready before it removes an old one – the rollout hangs, the app doesn’t. Fix it with the file from the repo:

kubectl apply -f k8s/web.yaml && kubectl rollout status deploy/web

Expected:

deployment.apps/web configured
service/web unchanged
Waiting for deployment "web" rollout to finish: 1 old replicas are pending…
deployment "web" successfully rolled out

kubectl rollout undo deploy/web would work too, but warns that the annotation last-applied-configuration no longer matches the cluster afterwards.

Save your progress? Sign up for free

Enterprise Note

  • Encrypt Secrets in etcd (EncryptionConfiguration, ideally with a KMS provider) and restrict get/list on Secrets tightly with RBAC – list returns all values as well.
  • Secrets don’t belong in Git, not even base64-encoded. Common choices are the External Secrets Operator with Vault/OpenBao (“OpenBao secrets hands-on”), Sealed Secrets or SOPS.
  • Automatic restarts after a config change: a checksum annotation in the Pod template (Helm, Day 10) or a controller like Reloader.
  • immutable: true for ConfigMaps and Secrets that never change takes load off the API server and prevents accidental changes; new values then mean a new name.
  • subPath mounts don’t receive updates – if you rely on live reload, mount the whole directory.