Day 4: Configuration – ConfigMaps and Secrets
Goal: No more key and no more URL in the manifest – configuration comes from a ConfigMap, the API key from a Secret, and you rotate it without downtime. · Duration: approx. 75 min · Prerequisite: Day 3
Briefing
- A ConfigMap holds key-value pairs or whole files. A Secret is the same for confidential values – stored only base64-encoded, not encrypted.
- A container gets both as an environment variable
(
envFrom,valueFrom) or as a file via a volume. - The kubelet updates files from a volume after a change, but not variables. The notes services only read their configuration at startup anyway.
- Secrets never end up as YAML in the repo; they are created by the
script
k8s/secrets.sh.
ConfigMap notes-config ──envFrom────────────► worker (NOTES_API_URL, INTERVAL)
└─valueFrom──────────► web (NOTES_API_URL)
Secret notes-api ──volume /etc/notes/api-key► api, worker (*_FILE)
└─secretKeyRef─────────────► web (NOTES_API_KEY)
Drill 1: The Key in Plain Text
Start the day and fetch the state of Day 3:
source ~/k8s-lab/setup/env.sh
mkdir -p ~/k8s-lab/tag04 && cd ~/k8s-lab/tag04
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag03/notes/." .Where is the API key today?
grep -rn 'lab-api-key' k8s/
c='NAME:.metadata.name,ENV:.spec.template.spec.containers[0].env[*].name'
kubectl get deploy -o custom-columns="$c"Expected:
k8s/worker.yaml:27: value: lab-api-key-123
k8s/api.yaml:31: value: lab-api-key-123
k8s/web.yaml:27: value: lab-api-key-123
NAME ENV
api NOTES_API_KEY
web NOTES_API_URL,NOTES_API_KEY
worker NOTES_API_URL,NOTES_API_KEY
The same value three times, three times in plain text; anyone with
read access to the repo knows it. And NOTES_API_URL appears
twice. Today you pull both out.
Save your progress? Sign up for free
Drill 2: ConfigMap Three Ways
kubectl create configmap builds a ConfigMap from
literals or files. With --dry-run=client -o yaml you only
see the result, without creating anything:
kubectl create configmap demo --from-literal=INTERVAL=10 --from-literal=MODE=lab \
--dry-run=client -o yaml
printf 'level=debug\nformat=text\n' > app.properties
kubectl create configmap demo --from-file=app.properties --dry-run=client -o yaml
kubectl create configmap demo --from-env-file=app.properties --dry-run=client -o yaml
rm app.propertiesExpected:
apiVersion: v1
data:
INTERVAL: "10"
MODE: lab
kind: ConfigMap
metadata:
name: demo
apiVersion: v1
data:
app.properties: |
level=debug
format=text
kind: ConfigMap
metadata:
name: demo
apiVersion: v1
data:
format: text
level: debug
kind: ConfigMap
metadata:
name: demo
--from-file stores the whole file under its name,
--from-env-file splits it into keys. For the notes app the
third way is enough – YAML directly, versioned in the repo:
cat > k8s/config.yaml <<'EOF'
# k8s/config.yaml – state at end of Day 4
apiVersion: v1
kind: ConfigMap
metadata:
name: notes-config
labels:
app.kubernetes.io/part-of: notes
data:
NOTES_API_URL: http://api:8080
INTERVAL: "10"
EOF
kubectl apply --dry-run=server -f k8s/config.yaml
kubectl apply -f k8s/config.yaml
kubectl get configmap notes-config -o jsonpath='{.data}'; echoExpected:
configmap/notes-config created (server dry run)
configmap/notes-config created
{"INTERVAL":"10","NOTES_API_URL":"http://api:8080"}
"10" is in quotes: values of a ConfigMap are always
strings, and the API server rejects a bare 10.
Save your progress? Sign up for free
Drill 3: Secret via Script
cat > k8s/secrets.sh <<'EOF'
#!/usr/bin/env bash
# k8s/secrets.sh – state at end of Day 4: creates the Secrets (idempotent, lab values only)
set -euo pipefail
kubectl -n notes create secret generic notes-api \
--from-literal=api-key="${NOTES_API_KEY:-lab-api-key-123}" \
--dry-run=client -o yaml | kubectl apply -f -
EOF
chmod +x k8s/secrets.sh
bash -n k8s/secrets.sh && ./k8s/secrets.sh
./k8s/secrets.sh
kubectl get secret notes-apiExpected (abridged, names, IPs and times will differ on your machine):
secret/notes-api created
secret/notes-api unchanged
NAME TYPE DATA AGE
notes-api Opaque 1 1s
create … --dry-run=client -o yaml only generates the
object, kubectl apply -f - creates it or reconciles it.
That is why the script can run any number of times – create
alone would fail the second time with AlreadyExists. The
value comes from the environment; the repo only contains the lab
default.
How secret is a Secret?
kubectl get secret notes-api -o yaml | grep -A1 '^data'
kubectl get secret notes-api -o jsonpath='{.data.api-key}' | base64 -d; echoExpected:
data:
api-key: bGFiLWFwaS1rZXktMTIz
lab-api-key-123
Security: base64 is an encoding, not encryption. Anyone allowed to
get secretknows the value. Andkubectl applyadditionally creates the annotationkubectl.kubernetes.io/last-applied-configurationwith the same value. Only RBAC and encryption in etcd offer protection (Enterprise Note).
Save your progress? Sign up for free
Drill 4: envFrom,
valueFrom and a Secret as a File
The three Deployments now get their values from the ConfigMap and the Secret. Take the complete files from the companion code:
cp "$LAB/code/tag04/notes/k8s/"{api,web,worker}.yaml k8s/Three excerpts show the differences. worker uses envFrom
to fetch all keys of the ConfigMap, and the key as a
file:
# k8s/worker.yaml (excerpt) – state at end of Day 4
spec:
containers:
- name: worker
image: notes-worker:0.1.0
envFrom:
- configMapRef:
name: notes-config
env:
- name: NOTES_API_KEY_FILE
value: /etc/notes/api-key
volumeMounts:
- name: api-key
mountPath: /etc/notes
readOnly: true
volumes:
- name: api-key
secret:
secretName: notes-apiapi gets the same volume and NOTES_API_KEY_FILE, but no
ConfigMap. web uses valueFrom to fetch
individual keys selectively – from the Secret too,
because notes-web only knows NOTES_API_KEY:
# k8s/web.yaml (excerpt) – state at end of Day 4
env:
- name: NOTES_API_URL
valueFrom:
configMapKeyRef:
name: notes-config
key: NOTES_API_URL
- name: NOTES_API_KEY
valueFrom:
secretKeyRef:
name: notes-api
key: api-keygrep -rn 'lab-api-key' k8s/*.yaml; echo "Exit: $?"
kubectl apply --dry-run=server -f k8s/ >/dev/null && kubectl apply -f k8s/
for d in api web worker; do kubectl rollout status deploy/$d >/dev/null; done
kubectl logs deploy/api | head -2
kubectl exec deploy/worker -- env | grep -E '^(NOTES|INTERVAL)' | sort
kubectl exec deploy/web -- printenv NOTES_API_URL NOTES_API_KEYExpected (abridged):
Exit: 1
deployment.apps/api configured
service/api unchanged
configmap/notes-config unchanged
namespace/notes unchanged
deployment.apps/web configured
service/web unchanged
deployment.apps/worker configured
time=… level=INFO msg="api key loaded" source=file
time=… level=INFO msg="store ready" store=memory
INTERVAL=10
NOTES_API_KEY_FILE=/etc/notes/api-key
NOTES_API_URL=http://api:8080
http://api:8080
lab-api-key-123
source=file – api read the key from the file, not from a
variable. In worker the key appears nowhere in the environment; in web
it does. What does the file look like?
kubectl exec deploy/worker -- ls -la /etc/notes
curl -s -X POST localhost:30080/api/notes -H 'Content-Type: application/json' \
-d '{"text":"Key kommt aus dem Secret"}'; echoExpected (abridged):
drwxrwxrwt 3 root root 100 Oct 30 14:34 .
drwxr-xr-x 1 root root 4096 Oct 30 14:34 ..
drwxr-xr-x 2 root root 60 Oct 30 14:34 ..2026_10_30_14_34_29.420101691
lrwxrwxrwx 1 root root 31 Oct 30 14:34 ..data -> ..2026_10_30_14_34_29.420101691
lrwxrwxrwx 1 root root 14 Oct 30 14:34 api-key -> ..data/api-key
{"id":"13bf6e3a-…","text":"Key kommt aus dem Secret"}
api-key is a symlink via ..data to a
timestamped directory. On a change, the kubelet writes a new directory
and repoints ..data in a single step – readers never see
half a file. The volume is a tmpfs; the key never lands on
the node’s disk.
| Way | Field | Updates itself? |
|---|---|---|
| all keys as variables | envFrom |
no |
| one key as a variable | valueFrom (configMapKeyRef,
secretKeyRef) |
no |
| keys as files | volumes (configMap,
secret) |
yes, after approx. 60–90 s |
Save your progress? Sign up for free
Drill 5: Change the Configuration
The worker should count every 5 seconds instead of every 10:
kubectl patch configmap notes-config --type merge -p '{"data":{"INTERVAL":"5"}}'
sleep 20
kubectl exec deploy/worker -- printenv INTERVAL
kubectl logs deploy/worker --timestamps | tail -2Expected (abridged):
configmap/notes-config patched
10
2026-10-30T14:34:57.64…Z summary: 1 notes, 5 words
2026-10-30T14:35:07.70…Z summary: 1 notes, 5 words
Nothing happens. An environment variable is set when the container starts and stays. You need new Pods:
kubectl rollout restart deploy/worker
kubectl rollout status deploy/worker >/dev/null
kubectl exec deploy/worker -- printenv INTERVAL
sleep 11; kubectl logs deploy/worker --timestamps | tail -3
kubectl get deploy worker -o jsonpath='{.spec.template.metadata.annotations}'; echoExpected (abridged):
deployment.apps/worker restarted
5
2026-10-30T14:35:18.96…Z worker started, api=http://api:8080, interval=5s
2026-10-30T14:35:19.75…Z summary: 0 notes, 0 words
2026-10-30T14:35:24.78…Z summary: 1 notes, 5 words
{"kubectl.kubernetes.io/restartedAt":"2026-10-30T14:35:12Z"}
rollout restart only puts an annotation with the time
into the Pod template. The template changes, so the Deployment rolls out
as it would for a new image – with the same strategy, without downtime.
By the way, the 0 notes are the in-memory replicas from Day
3: the worker hit a different api.
Back to the state in the repo:
kubectl apply -f k8s/config.yaml
kubectl rollout restart deploy/worker && kubectl rollout status deploy/workerExpected (abridged):
configmap/notes-config configured
deployment.apps/worker restarted
deployment "worker" successfully rolled out
Save your progress? Sign up for free
Drill 6: Rotate the Key
The key has leaked; a new one is needed. Create the new Secret and watch when the file in the worker switches over:
NOTES_API_KEY=lab-api-key-456 ./k8s/secrets.sh
s=$SECONDS
until [ "$(kubectl exec deploy/worker -- cat /etc/notes/api-key)" \
= lab-api-key-456 ]; do sleep 2; done
echo "Datei nach $((SECONDS - s)) s aktualisiert"
kubectl exec deploy/web -- printenv NOTES_API_KEY
curl -s -X POST localhost:30080/api/notes -H 'Content-Type: application/json' \
-d '{"text":"vor dem Neustart"}'; echoExpected (abridged, the seconds differ):
secret/notes-api configured
Datei nach 72 s aktualisiert
lab-api-key-123
{"id":"7ba3655d-…","text":"vor dem Neustart"}
The file follows after about a minute (the kubelet’s sync interval plus cache); the variable in web never does. The POST still works: api read the old key at startup, and web sends the old one along. Now the trap – restart only api:
kubectl rollout restart deploy/api && kubectl rollout status deploy/api >/dev/null
curl -s -X POST localhost:30080/api/notes -H 'Content-Type: application/json' \
-d '{"text":"nur api neu"}'; echoExpected:
deployment.apps/api restarted
{"error":"invalid api key"}
api now knows …-456, web still sends …-123.
A rotation is only finished when all readers have been
restarted:
kubectl rollout restart deploy/web deploy/worker
kubectl rollout status deploy/web >/dev/null
kubectl rollout status deploy/worker >/dev/null
sleep 3
curl -s -X POST localhost:30080/api/notes -H 'Content-Type: application/json' \
-d '{"text":"alle neu"}'; echoExpected (abridged):
deployment.apps/web restarted
deployment.apps/worker restarted
{"id":"819e6086-…","text":"alle neu"}
Warning: Without the
sleep 3,curloccasionally reports nothing right after the rollout of web (exit 56, connection reset). An old web Pod was still getting traffic while it was already being terminated. Day 5 shows what helps against that.
End of the drill – back to the lab key so that the final state is right:
./k8s/secrets.sh
kubectl rollout restart deploy/api deploy/web deploy/worker
for d in api web worker; do kubectl rollout status deploy/$d >/dev/null; done
sleep 3; kubectl exec deploy/web -- printenv NOTES_API_KEYExpected:
secret/notes-api configured
deployment.apps/api restarted
deployment.apps/web restarted
deployment.apps/worker restarted
lab-api-key-123
Save your progress? Sign up for free
Checkpoint
kubectl exec deploy/worker -- printenv INTERVAL NOTES_API_URLExpected: both values from the ConfigMap
notes-config.
10
http://api:8080
kubectl logs deploy/api | grep 'api key'Expected: source=file; the key comes from the Secret
volume.
Found 3 pods, using pod/api-9bcdb694-zx2cx
time=… level=INFO msg="api key loaded" source=file
grep -rl lab-api-key k8s/Expected: only secrets.sh; no manifest contains the
key.
k8s/secrets.sh
curl -s -X POST localhost:30080/api/notes -H 'Content-Type: application/json' \
-d '{"text":"Kontrollpunkt Tag 4"}'Expected: web passes the key on, and the api creates the note.
{"id":"dbba8751-…","text":"Kontrollpunkt Tag 4"}
Challenge
A colleague makes a typo in the key name of the Secret for web
(apikey instead of api-key) and applies it
with kubectl patch. What do you see on the Pods, why does
the page stay reachable, and how do you find the cause?
Hint: Look at the events of the new Pod – and think about what the RollingUpdate strategy does with the old ones.
Solution
kubectl patch deploy web --type json -p '[{"op":"replace",
"path":"/spec/template/spec/containers/0/env/1/valueFrom/secretKeyRef/key",
"value":"apikey"}]'
sleep 15
kubectl get pods -l app.kubernetes.io/name=web
kubectl rollout status deploy/web --timeout 10s; echo "Exit: $?"
kubectl get events --field-selector type=Warning,reason=Failed \
-o custom-columns=MESSAGE:.message | tail -1
curl -s -o /dev/null -w '%{http_code}\n' localhost:30080/api/notesExpected (abridged):
deployment.apps/web patched
NAME READY STATUS RESTARTS AGE
web-5f7797cd47-88d5f 1/1 Running 0 33s
web-5f7797cd47-qmp95 1/1 Running 0 36s
web-8495c45c4d-wnjxn 0/1 CreateContainerConfigError 0 16s
Waiting for deployment "web" rollout to finish: 1 out of 2 new replicas have been…
error: timed out waiting for the condition
Exit: 1
Error: couldn't find key apikey in Secret notes/notes-api
200
If a key or a whole Secret is missing, the kubelet can’t build the
environment: CreateContainerConfigError. The old Pods keep
running because the strategy first wants to see a new Pod ready before
it removes an old one – the rollout hangs, the app doesn’t. Fix it with
the file from the repo:
kubectl apply -f k8s/web.yaml && kubectl rollout status deploy/webExpected:
deployment.apps/web configured
service/web unchanged
Waiting for deployment "web" rollout to finish: 1 old replicas are pending…
deployment "web" successfully rolled out
kubectl rollout undo deploy/web would work too, but
warns that the annotation last-applied-configuration no
longer matches the cluster afterwards.
Save your progress? Sign up for free
Enterprise Note
- Encrypt Secrets in etcd (
EncryptionConfiguration, ideally with a KMS provider) and restrictget/liston Secrets tightly with RBAC –listreturns all values as well. - Secrets don’t belong in Git, not even base64-encoded. Common choices are the External Secrets Operator with Vault/OpenBao (“OpenBao secrets hands-on”), Sealed Secrets or SOPS.
- Automatic restarts after a config change: a checksum annotation in the Pod template (Helm, Day 10) or a controller like Reloader.
immutable: truefor ConfigMaps and Secrets that never change takes load off the API server and prevents accidental changes; new values then mean a new name.subPathmounts don’t receive updates – if you rely on live reload, mount the whole directory.