Skip to content
Hands-on IT IT Drills

Day 16: Competition – the Leak in the Repo

Goal: In eight rounds you find and fix planted mishaps from across the book – sops, rules, hook, Sealed Secrets and Shamir – against the clock and with proof. · Duration: approx. 120 min · Prerequisite: Days 0–15

Briefing

  • You play in a fresh clone ~/sops/tag16/klon (nothing is pushed) and in the namespaces tag16 and tag16b.
  • Eight rounds, all layers: hook, MAC, recipients, rules, applying, scope, controller keys, key groups.
  • Tools: sops decrypt and its messages, sops filestatus, updatekeys, yq, git, kubeseal --validate, kubectl get sealedsecret.
  • Every round ends with a proof command.
Round  Symptom                                              Target
1      git commit: leaks found                              3 min
2      sops: MAC mismatch                                   4 min
3      Colleague: Group 0: FAILED                           4 min
4      sops: no matching creation rules found               3 min
5      kubectl: illegal base64 data                         3 min
6      SealedSecret: no key could decrypt secret (token)    4 min
7      Secret gone, SealedSecret cannot decrypt             6 min
8      Emergency: the safe will not open                    4 min

Rules of the Game

  • In each round, the referee runs the referee commands – a colleague, or you yourself without looking. Then start the clock.
  • Find the cause, fix it, prove it with one command. Only then stop the clock.
  • Fix the cause, not the symptom: if you switch off the hook (--no-verify), commit plain text, use --ignore-mac on someone else’s file or seal again instead of restoring, you get zero points.
  • Read the hint only after the target time has run out.

Playing Field

First the most important check: Round 7 deletes the controller keys. The backup in the repo must contain all keys that are currently in the cluster.

# on argo
cd ~/sops/repo
B=$(sops decrypt backup/sealed-secrets-keys.yaml \
  | yq '[.items[].metadata.name] | sort | join(",")')
C=$(kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o json | jq -r '[.items[].metadata.name] | sort | join(",")')
echo "$B" | tr ',' '\n' | wc -l
[ "$B" = "$C" ] && echo gleich || echo VERSCHIEDEN

Expected (your count may differ; gleich means “same”, VERSCHIEDEN means “different”):

4
gleich

Warning: If it says VERSCHIEDEN, update the backup first (Day 13, Checkpoint) – otherwise Round 7 means real data loss.

# on argo
mkdir -p ~/sops/tag16 && cd ~/sops/tag16
git clone -q http://172.16.0.182:30300/lab/secrets.git klon
cd klon && git config core.hooksPath hooks
kubectl create namespace tag16
kubectl create namespace tag16b

Expected:

namespace/tag16 created
namespace/tag16b created

Round 1: The hasty commit

Situation: “Just quickly check in the token for the webhook Secret.” The commit does not go through.

Referee commands:

# on argo
cd ~/sops/tag16/klon
kubectl create secret generic wk -n notes-dev --from-literal=token=lab-wk-123 \
  --dry-run=client -o yaml > dev/wk.yaml
git add dev/wk.yaml

Target time: 3 min

Hint: What is in the staged file – and what does filestatus say?

Solution Round 1

# on argo
cd ~/sops/tag16/klon
git commit -q -m "Webhook-Token"; echo "rc=$?"
sops filestatus dev/wk.yaml
sops encrypt -i dev/wk.yaml
git add dev/wk.yaml && git commit -q -m "Webhook-Token"; echo "rc=$?"

Expected:

6:35PM WRN leaks found: 2
rc=1
{"encrypted":false}
rc=0

gitleaks detects the Kubernetes Secret in plain text (Day 9). Encrypt, stage again – the hook checks the staged state, not the file on disk.

Save your progress? Sign up for free

Round 2: The move

Situation: “The Secret neu belongs in notes-qa from today on. I changed the namespace, now nothing works.”

Referee commands:

# on argo
cd ~/sops/tag16/klon
sed -i 's/namespace: notes-dev/namespace: notes-qa/' dev/neu.yaml

Target time: 4 min

Hint: The change is intended. Who computes the MAC again?

Solution Round 2

# on argo
cd ~/sops/tag16/klon
sops decrypt dev/neu.yaml 2>&1 | cut -c1-20
sops decrypt --ignore-mac dev/neu.yaml \
  | sops encrypt --filename-override dev/neu.yaml /dev/stdin > ~/sops/tag16/neu.yaml
mv ~/sops/tag16/neu.yaml dev/neu.yaml
sops decrypt --extract '["metadata"]["namespace"]' dev/neu.yaml; echo
git commit -q -am "neu nach notes-qa"; echo "rc=$?"

Expected:

MAC mismatch. File ha
notes-qa
rc=0

The MAC covers the plain text (Day 3, Day 14). --ignore-mac is fine here because the change is known and intended.

Save your progress? Sign up for free

Round 3: The forgotten colleague

Situation: “But I added the colleague to .sops.yaml. She still cannot open dev/notes-db.yaml.”

Referee commands:

# on argo
cd ~/sops/tag16/klon
K=$(age-keygen -y ~/sops/tag01/kollegin.key)
yq -i '(.creation_rules[] | select(.path_regex == "^dev/.*\\.yaml$")).age
  += ",'"$K"'"' .sops.yaml
git commit -q -am "Kollegin für dev"

Target time: 4 min

Hint: Whom does the error message list – the recipients of the rule or of the file?

Solution Round 3

# on argo
cd ~/sops/tag16/klon
XDG_CONFIG_HOME=/nirgends SOPS_AGE_KEY_FILE=~/sops/tag01/kollegin.key \
  sops decrypt dev/notes-db.yaml 2>&1 | grep -c ': FAILED'
sops updatekeys -y dev/notes-db.yaml >/dev/null 2>&1
XDG_CONFIG_HOME=/nirgends SOPS_AGE_KEY_FILE=~/sops/tag01/kollegin.key \
  sops decrypt --extract '["metadata"]["name"]' dev/notes-db.yaml; echo
git commit -q -am "Kollegin: updatekeys"; echo "rc=$?"

Expected:

3
notes-db
rc=0

Three FAILED: the group and the two recipients of the file (you and Argo CD). The colleague is only in the rule; updatekeys carries her over into the file (Day 5).

Save your progress? Sign up for free

Round 4: The extension

Situation: “The cache configuration cannot be encrypted. It works for all the other files.”

Referee commands:

# on argo
cd ~/sops/tag16/klon
kubectl create secret generic cache -n notes-dev --from-literal=pw=lab-cache-123 \
  --dry-run=client -o yaml > dev/cache.yml

Target time: 3 min

Hint: Read the path_regex of the dev rule character by character.

Solution Round 4

# on argo
cd ~/sops/tag16/klon
sops encrypt -i dev/cache.yml 2>&1 | tail -1
yq '.creation_rules[] | select(.path_regex | test("dev")) | .path_regex' .sops.yaml
yq -i '(.creation_rules[] | select(.path_regex == "^dev/.*\\.yaml$")).path_regex
  = "^dev/.*\\.ya?ml$"' .sops.yaml
sops encrypt -i dev/cache.yml && sops filestatus dev/cache.yml
git add .sops.yaml dev/cache.yml && git commit -q -m "cache"; echo "rc=$?"

Expected:

error loading config: no matching creation rules found
^dev/.*\.yaml$
^dev/.*\.env$
{"encrypted":true}
rc=0

.yml is not .yaml. Alternatively, rename the file – what matters is that rule and file name match (Day 4).

Save your progress? Sign up for free

Round 5: The direct apply

Situation: “I only wanted to test the Secret wk from Round 1 in tag16 for a moment.”

Referee commands: none – the colleague only tried it.

Target time: 3 min

Hint: What does kubectl get to see when you apply the file directly?

Solution Round 5

# on argo
cd ~/sops/tag16/klon
kubectl apply -f dev/wk.yaml 2>&1 | grep -o 'cannot be handled.*'
sops decrypt dev/wk.yaml | yq '.metadata.namespace = "tag16"' \
  | kubectl apply --server-side -f -
kubectl -n tag16 get secret wk -o jsonpath='{.data.token}' | base64 -d; echo

Expected:

cannot be handled as a Secret: illegal base64 data at input byte 3
secret/wk serverside-applied
lab-wk-123

The ENC[…] values are not base64 (Day 3). Decrypt in the pipe, set the namespace there (not in the file – that would break the MAC) and apply server-side (Day 11).

Save your progress? Sign up for free

Round 6: The move in the cluster

Situation: “I need the SealedSecret from tag16 in tag16b as well. I just changed the namespace.”

Referee commands:

# on argo
cd ~/sops/tag16
kubectl create secret generic app -n tag16 --from-literal=token=lab-app-16 \
  --dry-run=client -o yaml | kubeseal -o yaml > app.sealed.yaml
kubectl apply -f app.sealed.yaml >/dev/null
sed 's/namespace: tag16$/namespace: tag16b/' app.sealed.yaml | kubectl apply -f -
sleep 3

Target time: 4 min

Hint: Which scope is in the SealedSecret – and what does it mean?

Solution Round 6

# on argo
cd ~/sops/tag16
kubectl -n tag16b get sealedsecret app -o jsonpath='{.status.conditions[0].message}'
echo
yq '.metadata.annotations // "strict"' app.sealed.yaml
kubectl create secret generic app -n tag16b --from-literal=token=lab-app-16 \
  --dry-run=client -o yaml | kubeseal -o yaml | kubectl apply -f -
sleep 3
kubectl -n tag16b get secret app -o jsonpath='{.data.token}' | base64 -d; echo

Expected:

no key could decrypt secret (token)
strict
sealedsecret.bitnami.com/app configured
lab-app-16

Scope strict: namespace and name are part of the encryption (Day 12). For the second namespace, you seal again – or, if deliberately intended, --scope namespace-wide/cluster-wide.

Save your progress? Sign up for free

Round 7: The new controller

Situation: “Since the controller was rebuilt, the Secret app in tag16 is missing. But the SealedSecret is still there!”

Referee commands:

Warning: Only after a successful check in the Playing Field.

# on argo
kubectl -n kube-system delete secret -l sealedsecrets.bitnami.com/sealed-secrets-key \
  >/dev/null
kubectl -n kube-system delete pod -l app.kubernetes.io/name=sealed-secrets >/dev/null
kubectl -n kube-system rollout status deploy/sealed-secrets-controller >/dev/null
sleep 3
kubectl -n tag16 delete secret app >/dev/null
sleep 5

Target time: 6 min

Hint: How many keys does the controller know, and where does it get the old ones from?

Solution Round 7

# on argo
kubectl -n tag16 get sealedsecret app -o jsonpath='{.status.conditions[0].message}'
echo
kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key \
  --no-headers | wc -l
cd ~/sops/repo
sops decrypt backup/sealed-secrets-keys.yaml | kubectl create -f -
kubectl -n kube-system delete pod -l app.kubernetes.io/name=sealed-secrets >/dev/null
kubectl -n kube-system rollout status deploy/sealed-secrets-controller >/dev/null
sleep 6
kubectl -n tag16 get secret app -o jsonpath='{.data.token}' | base64 -d; echo

Expected:

no key could decrypt secret (token)
1
secret/sealed-secrets-keyfdv6z created
secret/sealed-secrets-keyh7bnt created
secret/sealed-secrets-keyk7bkd created
secret/sealed-secrets-keytbvdh created
lab-app-16

A single new key – the controller created one at startup. Restore from the backup (create, not apply, Day 13), restart, and the controller creates the Secret again.

Save your progress? Sign up for free

Round 8: The safe

Situation: Emergency. You need the root password from tresor/notfall.yaml.

Referee commands: none – the safe is built this way (Day 6).

Target time: 4 min

Hint: How many groups does the file need, and which group holds your key?

Solution Round 8

# on argo
cd ~/sops/tag16/klon
sops decrypt tresor/notfall.yaml 2>&1 | grep -oE 'Recovery failed|threshold.*' \
  | head -1
yq '.sops.shamir_threshold, (.sops.key_groups | length)' tresor/notfall.yaml
SOPS_AGE_KEY="$(cat ~/sops/tag06/ben.key)" sops decrypt tresor/notfall.yaml

Expected:

Recovery failed
2
3
root_password: lab-root-123

Threshold 2 of 3 groups. Your key opens one; Ben’s key from group 2 opens the second. keys.txt and SOPS_AGE_KEY are read together (Day 4).

Save your progress? Sign up for free

Scoring

Round Error Target time Points
1 plain text staged 3 min 2
2 plain text changed outside of sops 4 min 2
3 recipient only in .sops.yaml 4 min 2
4 path_regex does not match .yml 3 min 2
5 encrypted file applied directly 3 min 2
6 strict SealedSecret moved 4 min 2
7 controller keys lost 6 min 3
8 Shamir: too few groups 4 min 3

Full points only within the target time, without the hint and with a proof command; otherwise half (round down). Total target time: 31 min, maximum: 18 points.

Points Assessment
15–18 ready for the final exam (Appendix B)
10–14 good; repeat the days of the rounds you missed
0–9 work through Days 3, 4, 5, 12 and 13 again

The final measurement:

# on argo
cd ~/sops/tag16/klon
git status -s | wc -l
git log --oneline origin/main.. | wc -l
kubectl get sealedsecret -A --no-headers | awk '{print $1, $2, $3}'

Expected:

0
5
tag16 app True
tag16b app True

Challenge

After Round 7, the controller has one key more than the backup. Bring the backup in the main repo up to date – so that neither plain text nor a last-applied-configuration gets into it – and prove that both lists are the same.

Hint: The yq filter from Day 13 removes annotations.

Solution

# on argo
cd ~/sops/repo
kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o yaml | yq 'del(.items[].metadata.uid, .items[].metadata.resourceVersion,
  .items[].metadata.creationTimestamp, .items[].metadata.managedFields,
  .items[].metadata.annotations)' > backup/sealed-secrets-keys.yaml
sops encrypt -i backup/sealed-secrets-keys.yaml
git add backup && git commit -q -m "tag16: Sicherung nach Wettkampf" && git push -q
[ "$(sops decrypt backup/sealed-secrets-keys.yaml \
  | yq '[.items[].metadata.name] | sort | join(",")')" = "$(kubectl -n kube-system \
  get secret -l sealedsecrets.bitnami.com/sealed-secrets-key -o json \
  | jq -r '[.items[].metadata.name] | sort | join(",")')" ] && echo gleich

Expected (gleich means “same”):

gleich

Save your progress? Sign up for free

Cleanup

# on argo
kubectl delete namespace tag16 tag16b
rm -rf ~/sops/tag16
cd ~/sops/repo && git status -sb | head -1

Expected:

namespace "tag16" deleted
namespace "tag16b" deleted
## main...origin/main

Checkpoint

# on argo
kubectl get ns --no-headers | awk '{print $1}' | grep -cE 'tag16'
kubectl -n notes-stage get secret --no-headers | wc -l

Expected:

0
3

Interview Note

  • Which mishap with encrypted Secrets is the most common? Plain text in a commit – the hook helps against it, and against a bypassed hook the same check in CI.
  • How do you tell whether a recipient is missing or the wrong key is being used? From the list under Group 0: FAILED: it shows the recipients of the file.
  • What happens without a backup of the controller keys? All SealedSecrets are lost; the values have to be created anew.

Enterprise Note

  • Run a competition like this as a game day with the team: prepare the mishaps in secret, close every round with a runbook entry.
  • Round 7 is the most important exercise of the year – restore on an empty cluster, with the backup from the safe, not from the admin’s disk.
  • The messages from all rounds belong in a “Secrets” runbook next to the alerting.