Day 10: Encrypting Secrets
Goal: You show that Secrets sit in plain text in etcd, enable encryption at rest in the API server, re-encrypt all existing Secrets and rotate the key. · Duration: approx. 60 min · Prerequisite: Day 6
Briefing
- CKS: Minimize Microservice Vulnerabilities – “Manage Kubernetes secrets”.
- A Secret is only base64-encoded. Anyone who can read etcd (backup,
snapshot, certificate from
/etc/kubernetes/pki) has all the passwords. - The
EncryptionConfigurationtells the API server which resources it encrypts before writing. The first provider writes, all providers read. - Existing objects stay unencrypted until they are written again.
k create secret --> kube-apiserver --(1st provider: aescbc, key2)--> etcd
^ k8s:enc:aescbc:v1:key2:...
+-- /etc/kubernetes/enc/enc.yaml (hostPath, root only)
mkdir -p ~/cks/tag10/backup && cd ~/cks/tag10
k create ns tag10
k config set-context --current --namespace=tag10Drill 1: Providing etcdctl
The etcd Pod ships etcdctl. For working on
cp1, a separate binary in the server’s version is more
convenient:
k -n kube-system get pod etcd-cp1 -o jsonpath='{.spec.containers[0].image}{"\n"}'
V=v3.7.0
B=https://github.com/etcd-io/etcd/releases/download/$V
ARCH=$(dpkg --print-architecture)
curl -sLO $B/etcd-$V-linux-${ARCH}.tar.gz
curl -sL $B/SHA256SUMS \
| grep linux-${ARCH}.tar.gz | sha256sum -c
tar xzf etcd-$V-linux-${ARCH}.tar.gz
sudo install etcd-$V-linux-${ARCH}/etcdctl /usr/local/bin/
etcdctl versionExpected:
registry.k8s.io/etcd:3.7.0-0
etcd-v3.7.0-linux-amd64.tar.gz: OK
etcdctl version: 3.7.0
API version: 3.7
You no longer need ETCDCTL_API=3. To log in, the client
certificate the API server uses works fine:
P=/etc/kubernetes/pki
E="--cacert=$P/etcd/ca.crt --cert=$P/apiserver-etcd-client.crt"
E="$E --key=$P/apiserver-etcd-client.key"Without a separate binary, it works inside the etcd Pod (there with
server.crt/server.key):
k -n kube-system exec etcd-cp1 -- etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key \
get /registry/secrets/kube-system --prefix --keys-only | head -1Expected: one line
/registry/secrets/kube-system/…
Save your progress? Sign up for free
Drill 2: A Secret in plain text in etcd
k create secret generic db-cred --from-literal=password=Lab-Geheim-42
sudo etcdctl $E get /registry/secrets/tag10/db-cred --print-value-only \
| hexdump -C | grep -A1 passwordExpected (abridged):
000000e0 65 22 3a 7b 7d 7d 42 00 12 19 0a 08 70 61 73 73 |e":{}}B.....pass|
000000f0 77 6f 72 64 12 0d 4c 61 62 2d 47 65 68 65 69 6d |word..Lab-Geheim|
00000100 2d 34 32 1a 06 4f 70 61 71 75 65 1a 00 22 00 0a |-42..Opaque.."..|
Lab-Geheim-42 (a pure lab value) sits unencrypted in the
Protobuf object – not even base64. Every etcd backup contains it just
the same.
Save your progress? Sign up for free
Drill 3: Creating the EncryptionConfiguration
A key for aescbc is 32 random bytes, base64-encoded:
sudo mkdir -p /etc/kubernetes/enc
K1=$(head -c 32 /dev/urandom | base64)
sudo tee /etc/kubernetes/enc/enc.yaml >/dev/null <<EOF
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: $K1
- identity: {}
EOF
sudo chmod 600 /etc/kubernetes/enc/enc.yaml
sudo ls -l /etc/kubernetes/encExpected:
-rw------- 1 root root 239 Nov 18 18:19 enc.yaml
Here <<EOF is deliberately unquoted so the shell
substitutes $K1. identity comes
last: it writes plain text but reads the old,
unencrypted Secrets. Instead of aescbc you can use
secretbox (same key length; the docs rate
aescbc as weak because of CBC and prefer
secretbox); aesgcm is fast but requires
rotation every 200,000 writes. For production the docs recommend
kms (v2) with an external key service.
Save your progress? Sign up for free
Drill 4: Enabling it in the API server
M=/etc/kubernetes/manifests
sudo cp $M/kube-apiserver.yaml ~/cks/tag10/backup/
sudo vim $M/kube-apiserver.yamlAdd the flag, mount and volume – the entries from Day 6 stay:
- --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
# under volumeMounts:
- mountPath: /etc/kubernetes/enc
name: enc
readOnly: true
# under volumes:
- hostPath:
path: /etc/kubernetes/enc
type: Directory
name: encsudo diff ~/cks/tag10/backup/kube-apiserver.yaml $M/kube-apiserver.yamlExpected:
43a44
> - --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
83a85,87
> - mountPath: /etc/kubernetes/enc
> name: enc
> readOnly: true
108a113,116
> - hostPath:
> path: /etc/kubernetes/enc
> type: Directory
> name: enc
After about a minute:
k -n kube-system get pod kube-apiserver-cp1
k get --raw /readyz; echoExpected:
NAME READY STATUS RESTARTS AGE
kube-apiserver-cp1 1/1 Running 0 37s
ok
If it doesn’t start,
sudo crictl ps -a --name kube-apiserver shows the container
and sudo crictl logs <id> the reason (typo in the
YAML, missing volume).
Save your progress? Sign up for free
Drill 5: Proof and rewriting
k create secret generic api-key --from-literal=key=Lab-Neu-99
for s in db-cred api-key; do
sudo etcdctl $E get /registry/secrets/tag10/$s --print-value-only \
| head -c 23 | hexdump -C | head -1
done
k get secret api-key -o jsonpath='{.data.key}' | base64 -d; echoExpected:
00000000 6b 38 73 00 0a 0c 0a 02 76 31 12 06 53 65 63 72 |k8s.....v1..Secr|
00000000 6b 38 73 3a 65 6e 63 3a 61 65 73 63 62 63 3a 76 |k8s:enc:aescbc:v|
Lab-Neu-99
The new Secret starts with k8s:enc:aescbc:v1:key1:, the
old one is still plain text. Nothing changes for k – the
API server decrypts transparently. Rewrite all Secrets:
k get secrets -A -o json | k replace -f - | tail -2
sudo etcdctl $E get /registry/secrets --prefix --keys-only | grep -c .
sudo etcdctl $E get /registry/secrets --prefix --print-value-only \
| grep -ac "^k8s:enc:aescbc"Expected:
secret/db-cred replaced
secret/sh.helm.release.v1.traefik.v1 replaced
9
9
As many keys as encrypted values: no Secret is left in plain text. The number depends on your cluster – what matters is that both values are equal.
Save your progress? Sign up for free
Drill 6: Rotating the key
Put the new key before the old one –
key2 writes, key1 still reads:
sudo cp /etc/kubernetes/enc/enc.yaml ~/cks/tag10/backup/enc-key1.yaml
K2=$(head -c 32 /dev/urandom | base64)
sudo sed -i "s| keys:|&\n - name: key2\n secret: $K2|" \
/etc/kubernetes/enc/enc.yaml
sudo grep -A5 "keys:" /etc/kubernetes/enc/enc.yaml | grep nameExpected:
- name: key2
- name: key1
The API server reads the file only at startup. Warning:
k delete pod kube-apiserver-cp1 deletes only the mirror
Pod, the container keeps running. Restart via the runtime:
sudo crictl stop $(sudo crictl ps --name kube-apiserver -q)
k -n kube-system get pod kube-apiserver-cp1Expected (after about a minute):
NAME READY STATUS RESTARTS AGE
kube-apiserver-cp1 1/1 Running 1 (51s ago) 2m4s
k get secrets -A -o json | k replace -f - >/dev/null
sudo etcdctl $E get /registry/secrets --prefix --print-value-only \
| grep -ac "k8s:enc:aescbc:v1:key2"Expected:
11
Only now, once everything is written with key2, remove
key1 and restart once more:
sudo sed -i "/name: key1/,+1d" /etc/kubernetes/enc/enc.yaml
sudo crictl stop $(sudo crictl ps --name kube-apiserver -q)Wait until k get --raw /readyz reports ok
again (about 1 min):
k get secret db-cred -o jsonpath='{.data.password}' | base64 -d; echoExpected:
Lab-Geheim-42
If you delete key1 before rewriting, every Secret
encrypted with it becomes unreadable. With several API servers there is
an intermediate step: first add the new key in second position
everywhere.
Save your progress? Sign up for free
Drill 7: Upgrade-proof, and Secrets in the Pod
As on Day 6, the flag belongs in kubeadm-config,
otherwise kubeadm upgrade removes it:
k -n kube-system edit cm kubeadm-configAdd under apiServer:
extraArgs:
# ... profiling, authentication-config stay
- name: encryption-provider-config
value: /etc/kubernetes/enc/enc.yaml
extraVolumes:
- hostPath: /etc/kubernetes/enc
mountPath: /etc/kubernetes/enc
name: enc
pathType: Directory
readOnly: truesudo kubeadm upgrade diff v1.37.2 | grep -c -- "--encryption-provider-config"Expected:
0
The flag is no longer removed (the rest of the diff is only ordering). And in the Pod? A Secret as an environment variable ends up in the container configuration:
cat > pod-secret.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: app
spec:
containers:
- name: app
image: docker.io/library/busybox:1.37
command: ["sleep", "3600"]
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: db-cred
key: password
volumeMounts:
- name: cred
mountPath: /etc/cred
readOnly: true
volumes:
- name: cred
secret:
secretName: db-cred
defaultMode: 0400
EOF
k apply -f pod-secret.yaml
k wait --for=condition=Ready pod/app
k get pod app -o jsonpath='{.spec.nodeName}{"\n"}'
k exec app -- sh -c "mount | grep /etc/cred; ls -lL /etc/cred"Expected (abridged):
worker2
tmpfs on /etc/cred type tmpfs (ro,relatime,size=3907440k,inode64,noswap)
-r-------- 1 root root 13 Nov 18 18:24 password
On worker2 (ssh [email protected]):
C=$(sudo crictl ps --name app -q --label io.kubernetes.pod.namespace=tag10)
sudo crictl inspect $C | grep DB_PASSWORDExpected:
"key": "DB_PASSWORD",
"DB_PASSWORD=Lab-Geheim-42",
The variable sits in plain text in the runtime configuration, is
inherited by every child process and shows up in crash dumps. The volume
lives in RAM (tmpfs), is readable only by the container and
is updated when the Secret changes – prefer it.
Save your progress? Sign up for free
Exercise: Speed Drills
- Which file does the API server use for encryption at rest? Target time: 15 s
- Which provider encrypts new Secrets? Target time: 20 s
- Is
tag10/api-keyencrypted in etcd, and with which key? Target time: 45 s - Generate a valid key for
aescbcorsecretbox. Target time: 10 s
Solution
sudo grep -- --encryption-provider /etc/kubernetes/manifests/kube-apiserver.yaml
sudo grep -A2 "providers:" /etc/kubernetes/enc/enc.yaml
sudo etcdctl $E get /registry/secrets/tag10/api-key --print-value-only \
| head -c 23; echo
head -c 32 /dev/urandom | base64Expected (key is random):
- --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
providers:
- aescbc:
keys:
k8s:enc:aescbc:v1:key2:
yjnOuN73vFOYidBvGl0BosfFvLnuA4287Zy1g5xNsKY=
Save your progress? Sign up for free
Checkpoint
k get --raw /readyz; echoExpected: ok
sudo grep -c -- "--encryption-provider-config" \
/etc/kubernetes/manifests/kube-apiserver.yamlExpected: 1
sudo etcdctl $E get /registry/secrets/tag10/db-cred --print-value-only \
| head -c 23; echoExpected: k8s:enc:aescbc:v1:key2:
k -n kube-system get cm kubeadm-config -o yaml | grep -c "/etc/kubernetes/enc"Expected: 3 (flag,
hostPath, mountPath)
Clean up after the challenge: k delete ns tag10 and
k config set-context --current --namespace=default. In the
lab, encryption with key2 stays active –
flag, volume, /etc/kubernetes/enc/enc.yaml and the entry in
kubeadm-config. Day 15 and Day 19 build on it.
Challenge
A colleague “just tidied up the order” in enc.yaml.
Reproduce it, prove that new Secrets land in plain text again, and fix
it.
sudo cp /etc/kubernetes/enc/enc.yaml ~/cks/tag10/backup/enc-key2.yaml
sudo sed -i -e "/ - identity: {}/d" -e "s|^ providers:|&\n - identity: {}|" \
/etc/kubernetes/enc/enc.yaml
sudo crictl stop $(sudo crictl ps --name kube-apiserver -q)Hint: Compare the number of Secret keys in etcd with the number of encrypted values.
Solution
Wait until k get --raw /readyz reports ok
again (about 1 min):
k create secret generic token --from-literal=t=Lab-Token-7
sudo etcdctl $E get /registry/secrets --prefix --keys-only | grep -c .
sudo etcdctl $E get /registry/secrets --prefix --print-value-only \
| grep -ac "^k8s:enc:aescbc"
sudo grep -A1 "providers:" /etc/kubernetes/enc/enc.yamlExpected:
secret/token created
12
11
providers:
- identity: {}
One Secret too few is encrypted; identity is in front
and writes plain text. The old Secrets stay readable because
aescbc still serves as a read provider – which is why
nobody notices the error otherwise. Fix it:
sudo cp ~/cks/tag10/backup/enc-key2.yaml /etc/kubernetes/enc/enc.yaml
sudo crictl stop $(sudo crictl ps --name kube-apiserver -q)Wait until k get --raw /readyz reports ok
again (about 1 min):
k get --raw /readyz; echo
k get secrets -A -o json | k replace -f - >/dev/null
sudo etcdctl $E get /registry/secrets/tag10/token --print-value-only | head -c 23; echoExpected:
ok
k8s:enc:aescbc:v1:key2:
Save your progress? Sign up for free
Exam Tip
- Three parts, all required:
EncryptionConfiguration, the flag--encryption-provider-config, the hostPath volume plus mount. Afterwardsk get secrets -A -o json | k replace -f -, otherwise the existing Secrets stay plain text. - Typical trap:
identityin first position, or a key that is not 32 bytes long (head -c 32 /dev/urandom | base64). Wait for the Pod to be1/1 Ready. - Docs: kubernetes.io → Tasks → Administer a Cluster → “Encrypting Confidential Data at Rest” (example YAML and etcdctl proof to copy).
Enterprise Note
- Don’t leave the key next to etcd:
kmsv2 with Vault, cloud KMS or an HSM; the API server then only sees a data key. - etcd backups without
enc.yaml(or KMS access) are worthless – back up the keys separately and just as securely. - Managed Kubernetes (EKS, GKE, AKS) offers envelope encryption via cloud KMS as a switch; rotation there happens through the KMS policy.
- Don’t store Secrets in the cluster at all: External Secrets Operator or the Secrets Store CSI Driver fetch them at runtime from Vault or the cloud vault.
- Keep RBAC on
secretstight – anyone allowed toget secretsreads everything in plain text despite encryption.