Skip to content
Hands-on IT IT Drills

Appendix B: Final Exam

Goal: You solve 14 tasks from across the book in 120 minutes – a new secrets repo from the first key to the rollout, plus team, Shamir, Transit, Sealed Secrets and troubleshooting – and grade yourself. · Duration: 120 min · Prerequisite: Days 0–16

Rules

  • Time: 120 minutes, set a timer. Each task has a weight in percent, 100 % in total.
  • There is no certificate. Your own bar: from 70 % on, you know the craft.
  • Allowed: getsops.io/docs, --help and your own files under ~/sops as templates. Not allowed: this book during the exam.
  • Everything runs in ~/sops/pruefung (exam; keys in keys/, repo in repo/) and in the namespace pruefung. The repo stays local, without Forgejo.
  • Work with SOPS_AGE_KEY_FILE=~/sops/pruefung/keys/admin.key – your own key from keys.txt has no business in this repo.
  • Tasks 2 to 13 build on each other. Task 13 has a preparation step that you only run once you get there.
  • The solutions are at the end. Read them only after the time is up.

Preparation

# on argo
mkdir -p ~/sops/pruefung/keys
kubectl create namespace pruefung
source ~/sops/tag07/bao.env && bao token lookup -format=json | jq -r .data.ttl

Expected:

namespace/pruefung created
25353

The OpenBao token from Day 7 must still be valid (TTL in seconds greater than 0). If not: set up OpenBao again as in Day 7, Drill 7 (Fix).

Task 1: Inventory (5 %)

For prod/notes-db.yaml and plattform/grafana-admin.yaml from ~/sops/repo, show in one line each which age recipients (the first 12 characters are enough) and which OpenBao address can open them.

Task 2: Two keys (5 %)

In ~/sops/pruefung/keys, create the age keys admin.key and ci.key, both readable only by you.

Task 3: The repo (10 %)

Create ~/sops/pruefung/repo as a Git repo with: rules for secrets/dev/*.yaml (admin + ci, only data/stringData), secrets/dev/*.env (admin + ci) and secrets/prod/*.yaml (admin + OpenBao Transit, only data/stringData), YAML with two spaces; plus .gitignore, readable diffs, .gitleaks.toml and a versioned hook that checks everything under secrets/. First commit “Gerüst” (scaffold).

Task 4: The dev Secret (6 %)

Create secrets/dev/db.yaml: Secret db in the namespace pruefung with password lab-pr-db-123. Encrypted, committed; kind stays readable.

Task 5: The env file (5 %)

Create secrets/dev/app.env with API_KEY=lab-pr-key-123, encrypt it and print the value without writing it to disk unencrypted for that.

Task 6: prod via OpenBao (8 %)

Create secrets/prod/db.yaml (Secret db, namespace pruefung-prod, password lab-pr-prod-123) and prove that it can be decrypted without any age key, only via OpenBao.

Task 7: The colleague (6 %)

The colleague (~/sops/tag01/kollegin.key) should be able to read secrets/dev/db.yaml. Prove it with her key alone.

Task 8: CI out (8 %)

The CI key is considered lost. Remove it from all dev rules and files so that the data key is new as well. Prove that ci.key no longer opens the file.

Task 9: The emergency safe (8 %)

secrets/notfall.yaml with root_password: lab-pr-root-123 should only open when two of three people come together: admin, the colleague, Ben (~/sops/tag06/ben.key). Prove: admin alone fails, admin and Ben succeed.

Task 10: The hook (5 %)

Prove that a plain-text file secrets/dev/klar.yaml with password: lab-pr-klar-123 cannot be committed. Afterwards, nothing of it may be left.

Task 11: Rolling out (8 %)

Roll out secrets/dev/db.yaml to the namespace pruefung. No annotation with the plain text may be attached to the Secret.

Task 12: A SealedSecret (8 %)

Create sealed/api.sealed.yaml in the repo: SealedSecret api in the namespace pruefung, scope strict, token=lab-pr-api-123. Check it against the controller before applying it, apply it and commit. Watch out for the hook.

Task 13: The night shift (8 %)

Preparation (run it only now):

# on argo
cd ~/sops/pruefung/repo
sed -i 's/namespace: pruefung$/namespace: pruefung-qa/' secrets/dev/db.yaml

A colleague moved the Secret to pruefung-qa – on purpose. Since then the file can no longer be opened. Repair it without losing the move, and prove that the colleague can read it again too.

Task 14: Short questions (10 %)

2 % each, one answer in one sentence:

  1. Why is it not enough to remove a recipient from .sops.yaml?
  2. What is the difference between MAC mismatch and Could not decrypt with AES_GCM?
  3. Why kubectl create instead of kubectl apply when restoring the Sealed Secrets keys?
  4. What does the list under Group 0: FAILED show?
  5. When do you choose Sealed Secrets over sops?

Solutions

Read them only after the time is up.

Solution 1

# on argo
cd ~/sops/repo
for f in prod/notes-db.yaml plattform/grafana-admin.yaml; do
  echo "$f: $(yq '[.sops.age[].recipient | .[0:12]]
    + [.sops.hc_vault[].vault_address] | join(" ")' "$f")"
done

Expected:

prod/notes-db.yaml: age1r025nv6k http://172.16.0.182:30820
plattform/grafana-admin.yaml: age1rnz4tqup http://172.16.0.182:30820

The recipients are in the file’s sops block, not in .sops.yaml (Day 2, 4).

Solution 2

# on argo
cd ~/sops/pruefung/keys
umask 077
age-keygen -o admin.key 2>/dev/null
age-keygen -o ci.key 2>/dev/null
stat -c '%a %n' *.key

Expected:

600 admin.key
600 ci.key

Solution 3

# on argo
cd ~/sops/pruefung && git init -q -b main repo && cd repo
A=$(age-keygen -y ../keys/admin.key)
C=$(age-keygen -y ../keys/ci.key)
cat > .sops.yaml <<EOF
creation_rules:
  - path_regex: ^secrets/dev/.*\.yaml\$
    encrypted_regex: ^(data|stringData)\$
    age: $A,$C
  - path_regex: ^secrets/dev/.*\.env\$
    age: $A,$C
  - path_regex: ^secrets/prod/.*\.yaml\$
    encrypted_regex: ^(data|stringData)\$
    age: $A
    hc_vault_transit_uri: http://172.16.0.182:30820/v1/transit/keys/sops
stores:
  yaml:
    indent: 2
EOF
printf '*.key\n*.token\nkeys.txt\n' > .gitignore
printf '*.yaml diff=sopsdiffer\n*.env diff=sopsdiffer\n' > .gitattributes
cp ~/sops/repo/.gitleaks.toml .
mkdir -p hooks secrets/dev secrets/prod
cp ~/sops/repo/hooks/pre-commit hooks/
sed -i 's#^    dev/\*.*)$#    secrets/*)#' hooks/pre-commit
grep -c 'secrets/\*)' hooks/pre-commit
git config core.hooksPath hooks
git config diff.sopsdiffer.textconv "sops decrypt"
git add -A && git commit -q -m Gerüst && git log --oneline

Expected:

1
3050209 Gerüst

Grading: 2 % each for rules, .gitignore, diff driver, .gitleaks.toml, hook with core.hooksPath (Day 4, 8, 9, 12, 15).

Solution 4

# on argo
cd ~/sops/pruefung/repo
export SOPS_AGE_KEY_FILE=~/sops/pruefung/keys/admin.key
kubectl create secret generic db -n pruefung --from-literal=password=lab-pr-db-123 \
  --dry-run=client -o yaml > secrets/dev/db.yaml
sops encrypt -i secrets/dev/db.yaml
yq '.kind, .data.password | .[0:20]' secrets/dev/db.yaml
git add secrets && git commit -q -m db && echo ok

Expected:

Secret
ENC[AES256_GCM,data:
ok

Solution 5

# on argo
cd ~/sops/pruefung/repo
export SOPS_AGE_KEY_FILE=~/sops/pruefung/keys/admin.key
printf 'API_KEY=lab-pr-key-123\n' > secrets/dev/app.env
sops encrypt -i secrets/dev/app.env
sops exec-env secrets/dev/app.env 'echo $API_KEY'
git add secrets && git commit -q -m env && echo ok

Expected:

lab-pr-key-123
ok

exec-env gives the values only to the child process (Day 8). printf with the plain text is the one moment in which it is on disk – encrypted right afterwards.

Solution 6

# on argo
cd ~/sops/pruefung/repo
source ~/sops/tag07/bao.env
export SOPS_AGE_KEY_FILE=~/sops/pruefung/keys/admin.key
kubectl create secret generic db -n pruefung-prod \
  --from-literal=password=lab-pr-prod-123 --dry-run=client -o yaml \
  > secrets/prod/db.yaml
sops encrypt -i secrets/prod/db.yaml
XDG_CONFIG_HOME=/nirgends SOPS_AGE_KEY_FILE=/dev/null \
  sops decrypt --extract '["data"]["password"]' secrets/prod/db.yaml | base64 -d; echo
git add secrets && git commit -q -m prod && echo ok

Expected:

lab-pr-prod-123
ok

XDG_CONFIG_HOME=/nirgends hides keys.txt, SOPS_AGE_KEY_FILE=/dev/null hides the admin key – what remains is Transit with the token (Day 7).

Solution 7

# on argo
cd ~/sops/pruefung/repo
export SOPS_AGE_KEY_FILE=~/sops/pruefung/keys/admin.key
K=$(age-keygen -y ~/sops/tag01/kollegin.key)
yq -i '(.creation_rules[] | select(.path_regex == "^secrets/dev/.*\\.yaml$")).age
  += ",'"$K"'"' .sops.yaml
sops updatekeys -y secrets/dev/db.yaml >/dev/null 2>&1
XDG_CONFIG_HOME=/nirgends SOPS_AGE_KEY_FILE=~/sops/tag01/kollegin.key \
  sops decrypt --extract '["metadata"]["name"]' secrets/dev/db.yaml; echo
git commit -q -am kollegin && echo ok

Expected:

db
ok

Solution 8

# on argo
cd ~/sops/pruefung/repo
export SOPS_AGE_KEY_FILE=~/sops/pruefung/keys/admin.key
C=$(age-keygen -y ~/sops/pruefung/keys/ci.key)
yq -i '(.creation_rules[] | select(.path_regex | test("^\\^secrets/dev"))).age
  |= sub(",'"$C"'"; "")' .sops.yaml
for f in secrets/dev/db.yaml secrets/dev/app.env; do
  sops updatekeys -y "$f" >/dev/null 2>&1
  sops rotate -i "$f"
done
XDG_CONFIG_HOME=/nirgends SOPS_AGE_KEY_FILE=~/sops/pruefung/keys/ci.key \
  sops decrypt secrets/dev/db.yaml 2>&1 | grep -c FAILED
git commit -q -am "ci raus" && echo ok

Expected:

3
ok

updatekeys takes the recipient out, rotate creates a new data key (Day 5). The old file in the Git history stays readable with ci.key – that is why a lost key also calls for a new password.

Solution 9

# on argo
cd ~/sops/pruefung/repo
export SOPS_AGE_KEY_FILE=~/sops/pruefung/keys/admin.key
A=$(age-keygen -y ../keys/admin.key)
K=$(age-keygen -y ~/sops/tag01/kollegin.key)
B=$(age-keygen -y ~/sops/tag06/ben.key)
yq -i '.creation_rules += [{"path_regex": "^secrets/notfall\\.yaml$",
  "shamir_threshold": 2, "key_groups": [{"age": ["'"$A"'"]},
  {"age": ["'"$K"'"]}, {"age": ["'"$B"'"]}]}]' .sops.yaml
printf 'root_password: lab-pr-root-123\n' > secrets/notfall.yaml
sops encrypt -i secrets/notfall.yaml
XDG_CONFIG_HOME=/nirgends sops decrypt secrets/notfall.yaml 2>&1 \
  | grep -o 'Recovery failed' | head -1
XDG_CONFIG_HOME=/nirgends SOPS_AGE_KEY="$(cat ~/sops/tag06/ben.key)" \
  sops decrypt secrets/notfall.yaml
git add -A && git commit -q -m notfall && echo ok

Expected:

Recovery failed
root_password: lab-pr-root-123
ok

Solution 10

# on argo
cd ~/sops/pruefung/repo
printf 'password: lab-pr-klar-123\n' > secrets/dev/klar.yaml
git add secrets/dev/klar.yaml
git commit -q -m klar; echo "rc=$?"
git restore --staged secrets/dev/klar.yaml && rm secrets/dev/klar.yaml
git status -s | wc -l

Expected (the hook says “PLAIN TEXT: … is not encrypted with sops”):

KLARTEXT: secrets/dev/klar.yaml ist nicht mit sops verschlüsselt
rc=1
0

The structure check catches what gitleaks does not detect (Day 9).

Solution 11

# on argo
cd ~/sops/pruefung/repo
export SOPS_AGE_KEY_FILE=~/sops/pruefung/keys/admin.key
sops decrypt secrets/dev/db.yaml | kubectl apply --server-side -f -
kubectl -n pruefung get secret db -o jsonpath='{.metadata.annotations}'; echo "|"
kubectl -n pruefung get secret db -o jsonpath='{.data.password}' | base64 -d; echo

Expected:

secret/db serverside-applied
|
lab-pr-db-123

Solution 12

# on argo
cd ~/sops/pruefung/repo
mkdir -p sealed
kubectl create secret generic api -n pruefung --from-literal=token=lab-pr-api-123 \
  --dry-run=client -o yaml | kubeseal -o yaml > sealed/api.sealed.yaml
kubeseal --validate < sealed/api.sealed.yaml && echo gültig
kubectl apply -f sealed/api.sealed.yaml >/dev/null
sleep 3
kubectl -n pruefung get secret api -o jsonpath='{.data.token}' | base64 -d; echo
git add sealed && git commit -q -m sealed; echo "rc=$?"

Expected (gültig means “valid”):

gültig
lab-pr-api-123
rc=0

sealed/ lies outside of secrets/, so the structure check does not apply, and the .gitleaks.toml from Day 12 lets sealed values through.

Solution 13

# on argo
cd ~/sops/pruefung/repo
export SOPS_AGE_KEY_FILE=~/sops/pruefung/keys/admin.key
sops decrypt secrets/dev/db.yaml 2>&1 | cut -c1-12
sops decrypt --ignore-mac secrets/dev/db.yaml \
  | sops encrypt --filename-override secrets/dev/db.yaml /dev/stdin > ../db.yaml
mv ../db.yaml secrets/dev/db.yaml
sops decrypt --extract '["metadata"]["namespace"]' secrets/dev/db.yaml; echo
XDG_CONFIG_HOME=/nirgends SOPS_AGE_KEY_FILE=~/sops/tag01/kollegin.key \
  sops decrypt --extract '["metadata"]["namespace"]' secrets/dev/db.yaml; echo
git commit -q -am qa && echo ok

Expected:

MAC mismatch
pruefung-qa
pruefung-qa
ok

When encrypting anew, the recipients from .sops.yaml apply – that is, admin and the colleague, without CI (Day 14).

Solution 14

  1. The file keeps its recipients until updatekeys, and the Git history knows the old data key – so updatekeys, rotate and change the secret.
  2. MAC mismatch: plain text changed outside of sops; AES_GCM: an encrypted value is damaged.
  3. apply writes the object including the private key as last-applied-configuration into an annotation, which ends up unencrypted in Git with the next backup.
  4. The recipients of the file with their result – that is, which keys would help.
  5. When only the cluster should see the values, only Kubernetes Secrets are involved and nobody wants to manage keys.

Scoring

Task Topic Weight Day
1 reading recipients 5 % 2, 4
2 age keys 5 % 1
3 repo scaffold 10 % 4, 8, 9, 12, 15
4 partial encryption 6 % 3
5 dotenv, exec-env 5 % 3, 8
6 Transit 8 % 7
7 adding a recipient 6 % 5
8 removing a recipient, rotating 8 % 5
9 Shamir 8 % 6
10 hook 5 % 9
11 applying safely 8 % 11, 15
12 Sealed Secrets 8 % 12
13 MAC repair 8 % 14
14 short questions 10 % all

Full points only with a proof command. From 70 %: you can set up and run a secrets repo for a team. 50–69 %: repeat the days of the tasks you missed. Below 50 %: work through Days 2 to 6 again.

Cleanup

# on argo
kubectl delete namespace pruefung
rm -rf ~/sops/pruefung

Expected:

namespace "pruefung" deleted

After that, the whole lab can go: sudo /usr/local/bin/k3s-uninstall.sh removes k3s including Forgejo, OpenBao and the controller. The tools in /usr/local/bin stay.

Checkpoint

# on argo
kubectl get ns pruefung 2>&1 | tail -1
ls ~/sops

Expected:

Error from server (NotFound): namespaces "pruefung" not found
repo
tag00
tag01
tag02
tag03
tag04
tag05
tag06
tag07
tag08
tag09
tag10
tag11
tag12
tag13
tag15