Skip to content
Hands-on IT IT Drills

Day 10: Encrypting Secrets

Goal: You show that Secrets sit in plain text in etcd, enable encryption at rest in the API server, re-encrypt all existing Secrets and rotate the key. · Duration: approx. 60 min · Prerequisite: Day 6

Briefing

  • CKS: Minimize Microservice Vulnerabilities – “Manage Kubernetes secrets”.
  • A Secret is only base64-encoded. Anyone who can read etcd (backup, snapshot, certificate from /etc/kubernetes/pki) has all the passwords.
  • The EncryptionConfiguration tells the API server which resources it encrypts before writing. The first provider writes, all providers read.
  • Existing objects stay unencrypted until they are written again.
k create secret --> kube-apiserver --(1st provider: aescbc, key2)--> etcd
                         ^                                  k8s:enc:aescbc:v1:key2:...
                         +-- /etc/kubernetes/enc/enc.yaml (hostPath, root only)
mkdir -p ~/cks/tag10/backup && cd ~/cks/tag10
k create ns tag10
k config set-context --current --namespace=tag10

Drill 1: Providing etcdctl

The etcd Pod ships etcdctl. For working on cp1, a separate binary in the server’s version is more convenient:

k -n kube-system get pod etcd-cp1 -o jsonpath='{.spec.containers[0].image}{"\n"}'
V=v3.7.0
B=https://github.com/etcd-io/etcd/releases/download/$V
ARCH=$(dpkg --print-architecture)
curl -sLO $B/etcd-$V-linux-${ARCH}.tar.gz
curl -sL $B/SHA256SUMS \
  | grep linux-${ARCH}.tar.gz | sha256sum -c
tar xzf etcd-$V-linux-${ARCH}.tar.gz
sudo install etcd-$V-linux-${ARCH}/etcdctl /usr/local/bin/
etcdctl version

Expected:

registry.k8s.io/etcd:3.7.0-0
etcd-v3.7.0-linux-amd64.tar.gz: OK
etcdctl version: 3.7.0
API version: 3.7

You no longer need ETCDCTL_API=3. To log in, the client certificate the API server uses works fine:

P=/etc/kubernetes/pki
E="--cacert=$P/etcd/ca.crt --cert=$P/apiserver-etcd-client.crt"
E="$E --key=$P/apiserver-etcd-client.key"

Without a separate binary, it works inside the etcd Pod (there with server.crt/server.key):

k -n kube-system exec etcd-cp1 -- etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/kube-system --prefix --keys-only | head -1

Expected: one line /registry/secrets/kube-system/…

Save your progress? Sign up for free

Drill 2: A Secret in plain text in etcd

k create secret generic db-cred --from-literal=password=Lab-Geheim-42
sudo etcdctl $E get /registry/secrets/tag10/db-cred --print-value-only \
  | hexdump -C | grep -A1 password

Expected (abridged):

000000e0  65 22 3a 7b 7d 7d 42 00  12 19 0a 08 70 61 73 73  |e":{}}B.....pass|
000000f0  77 6f 72 64 12 0d 4c 61  62 2d 47 65 68 65 69 6d  |word..Lab-Geheim|
00000100  2d 34 32 1a 06 4f 70 61  71 75 65 1a 00 22 00 0a  |-42..Opaque.."..|

Lab-Geheim-42 (a pure lab value) sits unencrypted in the Protobuf object – not even base64. Every etcd backup contains it just the same.

Save your progress? Sign up for free

Drill 3: Creating the EncryptionConfiguration

A key for aescbc is 32 random bytes, base64-encoded:

sudo mkdir -p /etc/kubernetes/enc
K1=$(head -c 32 /dev/urandom | base64)
sudo tee /etc/kubernetes/enc/enc.yaml >/dev/null <<EOF
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - aescbc:
      keys:
      - name: key1
        secret: $K1
  - identity: {}
EOF
sudo chmod 600 /etc/kubernetes/enc/enc.yaml
sudo ls -l /etc/kubernetes/enc

Expected:

-rw------- 1 root root 239 Nov 18 18:19 enc.yaml

Here <<EOF is deliberately unquoted so the shell substitutes $K1. identity comes last: it writes plain text but reads the old, unencrypted Secrets. Instead of aescbc you can use secretbox (same key length; the docs rate aescbc as weak because of CBC and prefer secretbox); aesgcm is fast but requires rotation every 200,000 writes. For production the docs recommend kms (v2) with an external key service.

Save your progress? Sign up for free

Drill 4: Enabling it in the API server

M=/etc/kubernetes/manifests
sudo cp $M/kube-apiserver.yaml ~/cks/tag10/backup/
sudo vim $M/kube-apiserver.yaml

Add the flag, mount and volume – the entries from Day 6 stay:

    - --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
    # under volumeMounts:
    - mountPath: /etc/kubernetes/enc
      name: enc
      readOnly: true
  # under volumes:
  - hostPath:
      path: /etc/kubernetes/enc
      type: Directory
    name: enc
sudo diff ~/cks/tag10/backup/kube-apiserver.yaml $M/kube-apiserver.yaml

Expected:

43a44
>     - --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
83a85,87
>     - mountPath: /etc/kubernetes/enc
>       name: enc
>       readOnly: true
108a113,116
>   - hostPath:
>       path: /etc/kubernetes/enc
>       type: Directory
>     name: enc

After about a minute:

k -n kube-system get pod kube-apiserver-cp1
k get --raw /readyz; echo

Expected:

NAME                 READY   STATUS    RESTARTS   AGE
kube-apiserver-cp1   1/1     Running   0          37s
ok

If it doesn’t start, sudo crictl ps -a --name kube-apiserver shows the container and sudo crictl logs <id> the reason (typo in the YAML, missing volume).

Save your progress? Sign up for free

Drill 5: Proof and rewriting

k create secret generic api-key --from-literal=key=Lab-Neu-99
for s in db-cred api-key; do
  sudo etcdctl $E get /registry/secrets/tag10/$s --print-value-only \
    | head -c 23 | hexdump -C | head -1
done
k get secret api-key -o jsonpath='{.data.key}' | base64 -d; echo

Expected:

00000000  6b 38 73 00 0a 0c 0a 02  76 31 12 06 53 65 63 72  |k8s.....v1..Secr|
00000000  6b 38 73 3a 65 6e 63 3a  61 65 73 63 62 63 3a 76  |k8s:enc:aescbc:v|
Lab-Neu-99

The new Secret starts with k8s:enc:aescbc:v1:key1:, the old one is still plain text. Nothing changes for k – the API server decrypts transparently. Rewrite all Secrets:

k get secrets -A -o json | k replace -f - | tail -2
sudo etcdctl $E get /registry/secrets --prefix --keys-only | grep -c .
sudo etcdctl $E get /registry/secrets --prefix --print-value-only \
  | grep -ac "^k8s:enc:aescbc"

Expected:

secret/db-cred replaced
secret/sh.helm.release.v1.traefik.v1 replaced
9
9

As many keys as encrypted values: no Secret is left in plain text. The number depends on your cluster – what matters is that both values are equal.

Save your progress? Sign up for free

Drill 6: Rotating the key

Put the new key before the old one – key2 writes, key1 still reads:

sudo cp /etc/kubernetes/enc/enc.yaml ~/cks/tag10/backup/enc-key1.yaml
K2=$(head -c 32 /dev/urandom | base64)
sudo sed -i "s|      keys:|&\n      - name: key2\n        secret: $K2|" \
  /etc/kubernetes/enc/enc.yaml
sudo grep -A5 "keys:" /etc/kubernetes/enc/enc.yaml | grep name

Expected:

      - name: key2
      - name: key1

The API server reads the file only at startup. Warning: k delete pod kube-apiserver-cp1 deletes only the mirror Pod, the container keeps running. Restart via the runtime:

sudo crictl stop $(sudo crictl ps --name kube-apiserver -q)
k -n kube-system get pod kube-apiserver-cp1

Expected (after about a minute):

NAME                 READY   STATUS    RESTARTS      AGE
kube-apiserver-cp1   1/1     Running   1 (51s ago)   2m4s
k get secrets -A -o json | k replace -f - >/dev/null
sudo etcdctl $E get /registry/secrets --prefix --print-value-only \
  | grep -ac "k8s:enc:aescbc:v1:key2"

Expected:

11

Only now, once everything is written with key2, remove key1 and restart once more:

sudo sed -i "/name: key1/,+1d" /etc/kubernetes/enc/enc.yaml
sudo crictl stop $(sudo crictl ps --name kube-apiserver -q)

Wait until k get --raw /readyz reports ok again (about 1 min):

k get secret db-cred -o jsonpath='{.data.password}' | base64 -d; echo

Expected:

Lab-Geheim-42

If you delete key1 before rewriting, every Secret encrypted with it becomes unreadable. With several API servers there is an intermediate step: first add the new key in second position everywhere.

Save your progress? Sign up for free

Drill 7: Upgrade-proof, and Secrets in the Pod

As on Day 6, the flag belongs in kubeadm-config, otherwise kubeadm upgrade removes it:

k -n kube-system edit cm kubeadm-config

Add under apiServer:

      extraArgs:
      # ... profiling, authentication-config stay
      - name: encryption-provider-config
        value: /etc/kubernetes/enc/enc.yaml
      extraVolumes:
      - hostPath: /etc/kubernetes/enc
        mountPath: /etc/kubernetes/enc
        name: enc
        pathType: Directory
        readOnly: true
sudo kubeadm upgrade diff v1.37.2 | grep -c -- "--encryption-provider-config"

Expected:

0

The flag is no longer removed (the rest of the diff is only ordering). And in the Pod? A Secret as an environment variable ends up in the container configuration:

cat > pod-secret.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
  - name: app
    image: docker.io/library/busybox:1.37
    command: ["sleep", "3600"]
    env:
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: db-cred
          key: password
    volumeMounts:
    - name: cred
      mountPath: /etc/cred
      readOnly: true
  volumes:
  - name: cred
    secret:
      secretName: db-cred
      defaultMode: 0400
EOF
k apply -f pod-secret.yaml
k wait --for=condition=Ready pod/app
k get pod app -o jsonpath='{.spec.nodeName}{"\n"}'
k exec app -- sh -c "mount | grep /etc/cred; ls -lL /etc/cred"

Expected (abridged):

worker2
tmpfs on /etc/cred type tmpfs (ro,relatime,size=3907440k,inode64,noswap)
-r--------    1 root     root            13 Nov 18 18:24 password

On worker2 (ssh [email protected]):

C=$(sudo crictl ps --name app -q --label io.kubernetes.pod.namespace=tag10)
sudo crictl inspect $C | grep DB_PASSWORD

Expected:

          "key": "DB_PASSWORD",
          "DB_PASSWORD=Lab-Geheim-42",

The variable sits in plain text in the runtime configuration, is inherited by every child process and shows up in crash dumps. The volume lives in RAM (tmpfs), is readable only by the container and is updated when the Secret changes – prefer it.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Which file does the API server use for encryption at rest? Target time: 15 s
  2. Which provider encrypts new Secrets? Target time: 20 s
  3. Is tag10/api-key encrypted in etcd, and with which key? Target time: 45 s
  4. Generate a valid key for aescbc or secretbox. Target time: 10 s

Solution

sudo grep -- --encryption-provider /etc/kubernetes/manifests/kube-apiserver.yaml
sudo grep -A2 "providers:" /etc/kubernetes/enc/enc.yaml
sudo etcdctl $E get /registry/secrets/tag10/api-key --print-value-only \
  | head -c 23; echo
head -c 32 /dev/urandom | base64

Expected (key is random):

    - --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
  providers:
  - aescbc:
      keys:
k8s:enc:aescbc:v1:key2:
yjnOuN73vFOYidBvGl0BosfFvLnuA4287Zy1g5xNsKY=

Save your progress? Sign up for free

Checkpoint

k get --raw /readyz; echo

Expected: ok

sudo grep -c -- "--encryption-provider-config" \
  /etc/kubernetes/manifests/kube-apiserver.yaml

Expected: 1

sudo etcdctl $E get /registry/secrets/tag10/db-cred --print-value-only \
  | head -c 23; echo

Expected: k8s:enc:aescbc:v1:key2:

k -n kube-system get cm kubeadm-config -o yaml | grep -c "/etc/kubernetes/enc"

Expected: 3 (flag, hostPath, mountPath)

Clean up after the challenge: k delete ns tag10 and k config set-context --current --namespace=default. In the lab, encryption with key2 stays active – flag, volume, /etc/kubernetes/enc/enc.yaml and the entry in kubeadm-config. Day 15 and Day 19 build on it.

Challenge

A colleague “just tidied up the order” in enc.yaml. Reproduce it, prove that new Secrets land in plain text again, and fix it.

sudo cp /etc/kubernetes/enc/enc.yaml ~/cks/tag10/backup/enc-key2.yaml
sudo sed -i -e "/  - identity: {}/d" -e "s|^  providers:|&\n  - identity: {}|" \
  /etc/kubernetes/enc/enc.yaml
sudo crictl stop $(sudo crictl ps --name kube-apiserver -q)

Hint: Compare the number of Secret keys in etcd with the number of encrypted values.

Solution

Wait until k get --raw /readyz reports ok again (about 1 min):

k create secret generic token --from-literal=t=Lab-Token-7
sudo etcdctl $E get /registry/secrets --prefix --keys-only | grep -c .
sudo etcdctl $E get /registry/secrets --prefix --print-value-only \
  | grep -ac "^k8s:enc:aescbc"
sudo grep -A1 "providers:" /etc/kubernetes/enc/enc.yaml

Expected:

secret/token created
12
11
  providers:
  - identity: {}

One Secret too few is encrypted; identity is in front and writes plain text. The old Secrets stay readable because aescbc still serves as a read provider – which is why nobody notices the error otherwise. Fix it:

sudo cp ~/cks/tag10/backup/enc-key2.yaml /etc/kubernetes/enc/enc.yaml
sudo crictl stop $(sudo crictl ps --name kube-apiserver -q)

Wait until k get --raw /readyz reports ok again (about 1 min):

k get --raw /readyz; echo
k get secrets -A -o json | k replace -f - >/dev/null
sudo etcdctl $E get /registry/secrets/tag10/token --print-value-only | head -c 23; echo

Expected:

ok
k8s:enc:aescbc:v1:key2:

Save your progress? Sign up for free

Exam Tip

  • Three parts, all required: EncryptionConfiguration, the flag --encryption-provider-config, the hostPath volume plus mount. Afterwards k get secrets -A -o json | k replace -f -, otherwise the existing Secrets stay plain text.
  • Typical trap: identity in first position, or a key that is not 32 bytes long (head -c 32 /dev/urandom | base64). Wait for the Pod to be 1/1 Ready.
  • Docs: kubernetes.io → Tasks → Administer a Cluster → “Encrypting Confidential Data at Rest” (example YAML and etcdctl proof to copy).

Enterprise Note

  • Don’t leave the key next to etcd: kms v2 with Vault, cloud KMS or an HSM; the API server then only sees a data key.
  • etcd backups without enc.yaml (or KMS access) are worthless – back up the keys separately and just as securely.
  • Managed Kubernetes (EKS, GKE, AKS) offers envelope encryption via cloud KMS as a switch; rotation there happens through the KMS policy.
  • Don’t store Secrets in the cluster at all: External Secrets Operator or the Secrets Store CSI Driver fetch them at runtime from Vault or the cloud vault.
  • Keep RBAC on secrets tight – anyone allowed to get secrets reads everything in plain text despite encryption.