Appendix B: Final Exam
Goal: You solve 14 tasks from across the book in 120 minutes – a new secrets repo from the first key to the rollout, plus team, Shamir, Transit, Sealed Secrets and troubleshooting – and grade yourself. · Duration: 120 min · Prerequisite: Days 0–16
Rules
- Time: 120 minutes, set a timer. Each task has a weight in percent, 100 % in total.
- There is no certificate. Your own bar: from 70 % on, you know the craft.
- Allowed: getsops.io/docs,
--helpand your own files under~/sopsas templates. Not allowed: this book during the exam. - Everything runs in
~/sops/pruefung(exam; keys inkeys/, repo inrepo/) and in the namespacepruefung. The repo stays local, without Forgejo. - Work with
SOPS_AGE_KEY_FILE=~/sops/pruefung/keys/admin.key– your own key fromkeys.txthas no business in this repo. - Tasks 2 to 13 build on each other. Task 13 has a preparation step that you only run once you get there.
- The solutions are at the end. Read them only after the time is up.
Preparation
# on argo
mkdir -p ~/sops/pruefung/keys
kubectl create namespace pruefung
source ~/sops/tag07/bao.env && bao token lookup -format=json | jq -r .data.ttlExpected:
namespace/pruefung created
25353
The OpenBao token from Day 7 must still be valid (TTL in seconds greater than 0). If not: set up OpenBao again as in Day 7, Drill 7 (Fix).
Task 1: Inventory (5 %)
For prod/notes-db.yaml and
plattform/grafana-admin.yaml from ~/sops/repo,
show in one line each which age recipients (the first 12 characters are
enough) and which OpenBao address can open them.
Task 2: Two keys (5 %)
In ~/sops/pruefung/keys, create the age keys
admin.key and ci.key, both readable only by
you.
Task 3: The repo (10 %)
Create ~/sops/pruefung/repo as a Git repo with: rules
for secrets/dev/*.yaml (admin + ci, only
data/stringData),
secrets/dev/*.env (admin + ci) and
secrets/prod/*.yaml (admin + OpenBao Transit, only
data/stringData), YAML with two spaces; plus
.gitignore, readable diffs, .gitleaks.toml and
a versioned hook that checks everything under secrets/.
First commit “Gerüst” (scaffold).
Task 4: The dev Secret (6 %)
Create secrets/dev/db.yaml: Secret db in
the namespace pruefung with password
lab-pr-db-123. Encrypted, committed; kind
stays readable.
Task 5: The env file (5 %)
Create secrets/dev/app.env with
API_KEY=lab-pr-key-123, encrypt it and print the value
without writing it to disk unencrypted for that.
Task 6: prod via OpenBao (8 %)
Create secrets/prod/db.yaml (Secret db,
namespace pruefung-prod, password
lab-pr-prod-123) and prove that it can be decrypted
without any age key, only via OpenBao.
Task 7: The colleague (6 %)
The colleague (~/sops/tag01/kollegin.key) should be able
to read secrets/dev/db.yaml. Prove it with her key
alone.
Task 8: CI out (8 %)
The CI key is considered lost. Remove it from all dev rules and files
so that the data key is new as well. Prove that ci.key no
longer opens the file.
Task 9: The emergency safe (8 %)
secrets/notfall.yaml with
root_password: lab-pr-root-123 should only open when two of
three people come together: admin, the colleague, Ben
(~/sops/tag06/ben.key). Prove: admin alone fails, admin and
Ben succeed.
Task 10: The hook (5 %)
Prove that a plain-text file secrets/dev/klar.yaml with
password: lab-pr-klar-123 cannot be committed. Afterwards,
nothing of it may be left.
Task 11: Rolling out (8 %)
Roll out secrets/dev/db.yaml to the namespace
pruefung. No annotation with the plain text may be attached
to the Secret.
Task 12: A SealedSecret (8 %)
Create sealed/api.sealed.yaml in the repo: SealedSecret
api in the namespace pruefung, scope
strict, token=lab-pr-api-123. Check it against
the controller before applying it, apply it and commit. Watch out for
the hook.
Task 13: The night shift (8 %)
Preparation (run it only now):
# on argo
cd ~/sops/pruefung/repo
sed -i 's/namespace: pruefung$/namespace: pruefung-qa/' secrets/dev/db.yamlA colleague moved the Secret to pruefung-qa – on
purpose. Since then the file can no longer be opened. Repair it without
losing the move, and prove that the colleague can read it again too.
Task 14: Short questions (10 %)
2 % each, one answer in one sentence:
- Why is it not enough to remove a recipient from
.sops.yaml? - What is the difference between
MAC mismatchandCould not decrypt with AES_GCM? - Why
kubectl createinstead ofkubectl applywhen restoring the Sealed Secrets keys? - What does the list under
Group 0: FAILEDshow? - When do you choose Sealed Secrets over sops?
Solutions
Read them only after the time is up.
Solution 1
# on argo
cd ~/sops/repo
for f in prod/notes-db.yaml plattform/grafana-admin.yaml; do
echo "$f: $(yq '[.sops.age[].recipient | .[0:12]]
+ [.sops.hc_vault[].vault_address] | join(" ")' "$f")"
doneExpected:
prod/notes-db.yaml: age1r025nv6k http://172.16.0.182:30820
plattform/grafana-admin.yaml: age1rnz4tqup http://172.16.0.182:30820
The recipients are in the file’s sops block, not in
.sops.yaml (Day 2, 4).
Solution 2
# on argo
cd ~/sops/pruefung/keys
umask 077
age-keygen -o admin.key 2>/dev/null
age-keygen -o ci.key 2>/dev/null
stat -c '%a %n' *.keyExpected:
600 admin.key
600 ci.key
Solution 3
# on argo
cd ~/sops/pruefung && git init -q -b main repo && cd repo
A=$(age-keygen -y ../keys/admin.key)
C=$(age-keygen -y ../keys/ci.key)
cat > .sops.yaml <<EOF
creation_rules:
- path_regex: ^secrets/dev/.*\.yaml\$
encrypted_regex: ^(data|stringData)\$
age: $A,$C
- path_regex: ^secrets/dev/.*\.env\$
age: $A,$C
- path_regex: ^secrets/prod/.*\.yaml\$
encrypted_regex: ^(data|stringData)\$
age: $A
hc_vault_transit_uri: http://172.16.0.182:30820/v1/transit/keys/sops
stores:
yaml:
indent: 2
EOF
printf '*.key\n*.token\nkeys.txt\n' > .gitignore
printf '*.yaml diff=sopsdiffer\n*.env diff=sopsdiffer\n' > .gitattributes
cp ~/sops/repo/.gitleaks.toml .
mkdir -p hooks secrets/dev secrets/prod
cp ~/sops/repo/hooks/pre-commit hooks/
sed -i 's#^ dev/\*.*)$# secrets/*)#' hooks/pre-commit
grep -c 'secrets/\*)' hooks/pre-commit
git config core.hooksPath hooks
git config diff.sopsdiffer.textconv "sops decrypt"
git add -A && git commit -q -m Gerüst && git log --onelineExpected:
1
3050209 Gerüst
Grading: 2 % each for rules, .gitignore, diff driver,
.gitleaks.toml, hook with core.hooksPath (Day
4, 8, 9, 12, 15).
Solution 4
# on argo
cd ~/sops/pruefung/repo
export SOPS_AGE_KEY_FILE=~/sops/pruefung/keys/admin.key
kubectl create secret generic db -n pruefung --from-literal=password=lab-pr-db-123 \
--dry-run=client -o yaml > secrets/dev/db.yaml
sops encrypt -i secrets/dev/db.yaml
yq '.kind, .data.password | .[0:20]' secrets/dev/db.yaml
git add secrets && git commit -q -m db && echo okExpected:
Secret
ENC[AES256_GCM,data:
ok
Solution 5
# on argo
cd ~/sops/pruefung/repo
export SOPS_AGE_KEY_FILE=~/sops/pruefung/keys/admin.key
printf 'API_KEY=lab-pr-key-123\n' > secrets/dev/app.env
sops encrypt -i secrets/dev/app.env
sops exec-env secrets/dev/app.env 'echo $API_KEY'
git add secrets && git commit -q -m env && echo okExpected:
lab-pr-key-123
ok
exec-env gives the values only to the child process (Day
8). printf with the plain text is the one moment in which
it is on disk – encrypted right afterwards.
Solution 6
# on argo
cd ~/sops/pruefung/repo
source ~/sops/tag07/bao.env
export SOPS_AGE_KEY_FILE=~/sops/pruefung/keys/admin.key
kubectl create secret generic db -n pruefung-prod \
--from-literal=password=lab-pr-prod-123 --dry-run=client -o yaml \
> secrets/prod/db.yaml
sops encrypt -i secrets/prod/db.yaml
XDG_CONFIG_HOME=/nirgends SOPS_AGE_KEY_FILE=/dev/null \
sops decrypt --extract '["data"]["password"]' secrets/prod/db.yaml | base64 -d; echo
git add secrets && git commit -q -m prod && echo okExpected:
lab-pr-prod-123
ok
XDG_CONFIG_HOME=/nirgends hides keys.txt,
SOPS_AGE_KEY_FILE=/dev/null hides the admin key – what
remains is Transit with the token (Day 7).
Solution 7
# on argo
cd ~/sops/pruefung/repo
export SOPS_AGE_KEY_FILE=~/sops/pruefung/keys/admin.key
K=$(age-keygen -y ~/sops/tag01/kollegin.key)
yq -i '(.creation_rules[] | select(.path_regex == "^secrets/dev/.*\\.yaml$")).age
+= ",'"$K"'"' .sops.yaml
sops updatekeys -y secrets/dev/db.yaml >/dev/null 2>&1
XDG_CONFIG_HOME=/nirgends SOPS_AGE_KEY_FILE=~/sops/tag01/kollegin.key \
sops decrypt --extract '["metadata"]["name"]' secrets/dev/db.yaml; echo
git commit -q -am kollegin && echo okExpected:
db
ok
Solution 8
# on argo
cd ~/sops/pruefung/repo
export SOPS_AGE_KEY_FILE=~/sops/pruefung/keys/admin.key
C=$(age-keygen -y ~/sops/pruefung/keys/ci.key)
yq -i '(.creation_rules[] | select(.path_regex | test("^\\^secrets/dev"))).age
|= sub(",'"$C"'"; "")' .sops.yaml
for f in secrets/dev/db.yaml secrets/dev/app.env; do
sops updatekeys -y "$f" >/dev/null 2>&1
sops rotate -i "$f"
done
XDG_CONFIG_HOME=/nirgends SOPS_AGE_KEY_FILE=~/sops/pruefung/keys/ci.key \
sops decrypt secrets/dev/db.yaml 2>&1 | grep -c FAILED
git commit -q -am "ci raus" && echo okExpected:
3
ok
updatekeys takes the recipient out, rotate
creates a new data key (Day 5). The old file in the Git history stays
readable with ci.key – that is why a lost key also calls
for a new password.
Solution 9
# on argo
cd ~/sops/pruefung/repo
export SOPS_AGE_KEY_FILE=~/sops/pruefung/keys/admin.key
A=$(age-keygen -y ../keys/admin.key)
K=$(age-keygen -y ~/sops/tag01/kollegin.key)
B=$(age-keygen -y ~/sops/tag06/ben.key)
yq -i '.creation_rules += [{"path_regex": "^secrets/notfall\\.yaml$",
"shamir_threshold": 2, "key_groups": [{"age": ["'"$A"'"]},
{"age": ["'"$K"'"]}, {"age": ["'"$B"'"]}]}]' .sops.yaml
printf 'root_password: lab-pr-root-123\n' > secrets/notfall.yaml
sops encrypt -i secrets/notfall.yaml
XDG_CONFIG_HOME=/nirgends sops decrypt secrets/notfall.yaml 2>&1 \
| grep -o 'Recovery failed' | head -1
XDG_CONFIG_HOME=/nirgends SOPS_AGE_KEY="$(cat ~/sops/tag06/ben.key)" \
sops decrypt secrets/notfall.yaml
git add -A && git commit -q -m notfall && echo okExpected:
Recovery failed
root_password: lab-pr-root-123
ok
Solution 10
# on argo
cd ~/sops/pruefung/repo
printf 'password: lab-pr-klar-123\n' > secrets/dev/klar.yaml
git add secrets/dev/klar.yaml
git commit -q -m klar; echo "rc=$?"
git restore --staged secrets/dev/klar.yaml && rm secrets/dev/klar.yaml
git status -s | wc -lExpected (the hook says “PLAIN TEXT: … is not encrypted with sops”):
KLARTEXT: secrets/dev/klar.yaml ist nicht mit sops verschlüsselt
rc=1
0
The structure check catches what gitleaks does not detect (Day 9).
Solution 11
# on argo
cd ~/sops/pruefung/repo
export SOPS_AGE_KEY_FILE=~/sops/pruefung/keys/admin.key
sops decrypt secrets/dev/db.yaml | kubectl apply --server-side -f -
kubectl -n pruefung get secret db -o jsonpath='{.metadata.annotations}'; echo "|"
kubectl -n pruefung get secret db -o jsonpath='{.data.password}' | base64 -d; echoExpected:
secret/db serverside-applied
|
lab-pr-db-123
Solution 12
# on argo
cd ~/sops/pruefung/repo
mkdir -p sealed
kubectl create secret generic api -n pruefung --from-literal=token=lab-pr-api-123 \
--dry-run=client -o yaml | kubeseal -o yaml > sealed/api.sealed.yaml
kubeseal --validate < sealed/api.sealed.yaml && echo gültig
kubectl apply -f sealed/api.sealed.yaml >/dev/null
sleep 3
kubectl -n pruefung get secret api -o jsonpath='{.data.token}' | base64 -d; echo
git add sealed && git commit -q -m sealed; echo "rc=$?"Expected (gültig means “valid”):
gültig
lab-pr-api-123
rc=0
sealed/ lies outside of secrets/, so the
structure check does not apply, and the .gitleaks.toml from
Day 12 lets sealed values through.
Solution 13
# on argo
cd ~/sops/pruefung/repo
export SOPS_AGE_KEY_FILE=~/sops/pruefung/keys/admin.key
sops decrypt secrets/dev/db.yaml 2>&1 | cut -c1-12
sops decrypt --ignore-mac secrets/dev/db.yaml \
| sops encrypt --filename-override secrets/dev/db.yaml /dev/stdin > ../db.yaml
mv ../db.yaml secrets/dev/db.yaml
sops decrypt --extract '["metadata"]["namespace"]' secrets/dev/db.yaml; echo
XDG_CONFIG_HOME=/nirgends SOPS_AGE_KEY_FILE=~/sops/tag01/kollegin.key \
sops decrypt --extract '["metadata"]["namespace"]' secrets/dev/db.yaml; echo
git commit -q -am qa && echo okExpected:
MAC mismatch
pruefung-qa
pruefung-qa
ok
When encrypting anew, the recipients from .sops.yaml
apply – that is, admin and the colleague, without CI (Day 14).
Solution 14
- The file keeps its recipients until
updatekeys, and the Git history knows the old data key – soupdatekeys,rotateand change the secret. MAC mismatch: plain text changed outside of sops;AES_GCM: an encrypted value is damaged.applywrites the object including the private key aslast-applied-configurationinto an annotation, which ends up unencrypted in Git with the next backup.- The recipients of the file with their result – that is, which keys would help.
- When only the cluster should see the values, only Kubernetes Secrets are involved and nobody wants to manage keys.
Scoring
| Task | Topic | Weight | Day |
|---|---|---|---|
| 1 | reading recipients | 5 % | 2, 4 |
| 2 | age keys | 5 % | 1 |
| 3 | repo scaffold | 10 % | 4, 8, 9, 12, 15 |
| 4 | partial encryption | 6 % | 3 |
| 5 | dotenv, exec-env | 5 % | 3, 8 |
| 6 | Transit | 8 % | 7 |
| 7 | adding a recipient | 6 % | 5 |
| 8 | removing a recipient, rotating | 8 % | 5 |
| 9 | Shamir | 8 % | 6 |
| 10 | hook | 5 % | 9 |
| 11 | applying safely | 8 % | 11, 15 |
| 12 | Sealed Secrets | 8 % | 12 |
| 13 | MAC repair | 8 % | 14 |
| 14 | short questions | 10 % | all |
Full points only with a proof command. From 70 %: you can set up and run a secrets repo for a team. 50–69 %: repeat the days of the tasks you missed. Below 50 %: work through Days 2 to 6 again.
Cleanup
# on argo
kubectl delete namespace pruefung
rm -rf ~/sops/pruefungExpected:
namespace "pruefung" deleted
After that, the whole lab can go:
sudo /usr/local/bin/k3s-uninstall.sh removes k3s including
Forgejo, OpenBao and the controller. The tools in
/usr/local/bin stay.
Checkpoint
# on argo
kubectl get ns pruefung 2>&1 | tail -1
ls ~/sopsExpected:
Error from server (NotFound): namespaces "pruefung" not found
repo
tag00
tag01
tag02
tag03
tag04
tag05
tag06
tag07
tag08
tag09
tag10
tag11
tag12
tag13
tag15