Day 14: Error Patterns
Goal: You recognize eight typical mishaps with sops and Sealed Secrets by their message, find the cause with one command and fix them without losing values. · Duration: approx. 75 min · Prerequisite: Day 13
Briefing
- Almost every sops message names the place: MAC (the file as a whole), AES_GCM (a single value), Group 0: FAILED (keys) or metadata not found (not a sops file).
- For key errors, sops lists every recipient of the file with its result. Read the list – it tells you which key would be needed.
- Sealed Secrets essentially knows one message: no key could decrypt secret. The cause is either the scope (Day 12) or a foreign or missing key (Day 13).
- You practice in a clone of its own,
~/sops/tag14/klon, from which nothing is pushed. Your main clone and Forgejo stay clean. - Every error pattern follows the pattern break it → symptom → investigate → cause → fix → verify.
| Message | Layer | Day |
|---|---|---|
MAC mismatch |
file | 3, Drill 2 here |
Could not decrypt with AES_GCM |
value | Drill 1 here |
Group 0: FAILED |
keys | 4, 5, Drill 4/5 here |
no key could decrypt secret |
Sealed Secrets | 12, 13, Drill 7 here |
Drill 1: A hand on the encrypted value
Break it: A clone to practice in, then a “small
typo” in an ENC[…] value – say, while resolving a conflict
by hand.
# on argo
mkdir -p ~/sops/tag14 && cd ~/sops/tag14
git clone -q http://172.16.0.182:30300/lab/secrets.git klon
cd klon
git config core.hooksPath hooks
sed -i 's/password: ENC\[AES256_GCM,data:./password: ENC[AES256_GCM,data:A/' \
dev/notes-db.yamlSymptom:
# on argo
cd ~/sops/tag14/klon
sops decrypt dev/notes-db.yaml 2>&1 | tail -1Expected (line wrapped):
Error decrypting tree: Error walking tree: Could not decrypt value: Could not decrypt
with AES_GCM: cipher: message authentication failed
Investigation and cause: git diff shows
the changed line. AES-GCM protects every value with its own checksum;
change a single character, and the value can no longer be opened. It
cannot be rescued – only fetched back from Git.
Fix and verify:
# on argo
cd ~/sops/tag14/klon
git diff --stat
git checkout -q dev/notes-db.yaml
sops decrypt --extract '["metadata"]["name"]' dev/notes-db.yamlExpected:
dev/notes-db.yaml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
notes-db
Save your progress? Sign up for free
Drill 2: Plain text changed next to it
Break it: Someone moves the Secret to another
namespace – with sed, because metadata is
readable after all.
# on argo
cd ~/sops/tag14/klon
sed -i 's/namespace: notes-dev/namespace: notes-qa/' dev/notes-db.yaml
sops decrypt dev/notes-db.yaml 2>&1 | cut -c1-40Expected:
MAC mismatch. File has 66C57B21830010AFB
Cause: The MAC covers all values, including the unencrypted ones (Day 3). Any change to the plain text outside of sops breaks it.
Fix: The change is intended – so decrypt with
--ignore-mac and encrypt again. That computes the MAC over
the new state.
# on argo
cd ~/sops/tag14/klon
sops decrypt --ignore-mac dev/notes-db.yaml \
| sops encrypt --filename-override dev/notes-db.yaml /dev/stdin \
> ~/sops/tag14/fix.yaml
mv ~/sops/tag14/fix.yaml dev/notes-db.yaml
sops decrypt --extract '["metadata"]["namespace"]' dev/notes-db.yaml
git checkout -q dev/notes-db.yamlExpected:
notes-qa
Tip:
--ignore-macmeans: “I know who changed the file.” For a file from someone else, a MAC error is an alarm signal, not a nuisance.
Save your progress? Sign up for free
Drill 3: Merge conflict
Break it: Two branches change different values of the same file.
# on argo
cd ~/sops/tag14/klon
git switch -q -c pw
sops set dev/notes-db.yaml '["data"]["password"]' "\"$(printf lab-db-pw-1 | base64)\""
git commit -q -am "Passwort"
git switch -q main && git switch -q -c user
sops set dev/notes-db.yaml '["data"]["user"]' "\"$(printf notes | base64)\""
git commit -q -am "Benutzer"
git merge -q pw 2>&1 | grep CONFLICTExpected:
CONFLICT (content): Merge conflict in dev/notes-db.yaml
Symptom and investigation:
# on argo
cd ~/sops/tag14/klon
grep -c '^<<<<<<<' dev/notes-db.yaml
sops decrypt dev/notes-db.yaml 2>&1 | tail -1Expected:
2
yaml: line 3: could not find expected ':'
Cause: Both sides changed lastmodified
and mac, plus their values. Git cannot merge that, and
doing it by hand does not work (Drill 1).
Fix: Take one side completely and apply the other
side’s change with sops. git show :3:file is “their”
version; /dev/stdin with --filename-override
reads it without writing it to disk.
# on argo
cd ~/sops/tag14/klon
git checkout --ours dev/notes-db.yaml
diff <(git show :3:dev/notes-db.yaml \
| sops decrypt --filename-override dev/notes-db.yaml /dev/stdin) \
<(sops decrypt dev/notes-db.yaml)
P=$(git show :3:dev/notes-db.yaml | sops decrypt \
--filename-override dev/notes-db.yaml --extract '["data"]["password"]' /dev/stdin)
sops set dev/notes-db.yaml '["data"]["password"]' "\"$P\""
git add dev/notes-db.yaml && git commit -q -m "Merge pw"Expected (abridged):
Updated 1 path from the index
3c3,4
< password: bGFiLWRiLXB3LTE=
---
> password: bGFiLWRiLWRldi05OTk=
> user: bm90ZXM=
Verify:
# on argo
cd ~/sops/tag14/klon
sops decrypt dev/notes-db.yaml | yq '.data | map_values(@base64d)'
git switch -q mainExpected:
password: lab-db-pw-1
user: notes
Tip: Fewer conflicts arise when every Secret has its own file and branches are short-lived. The project itself has no merge driver for sops.
Save your progress? Sign up for free
Drill 4: Recipient only in
.sops.yaml
Break it: The colleague (Day 5) should read
dev/. Someone adds her to the rule – and commits.
# on argo
cd ~/sops/tag14/klon
K=$(age-keygen -y ~/sops/tag01/kollegin.key)
yq -i '(.creation_rules[] | select(.path_regex == "^dev/.*\\.yaml$")).age
+= ",'"$K"'"' .sops.yamlSymptom: The colleague still cannot read it.
# on argo
cd ~/sops/tag14/klon
XDG_CONFIG_HOME=/nirgends SOPS_AGE_KEY_FILE=~/sops/tag01/kollegin.key \
sops decrypt dev/notes-db.yaml 2>&1 | grep -E 'Group|FAILED'Expected (abridged):
Group 0: FAILED
age1rnz4tqupd0f3lm4kfs564cfgys5j80497rk7ajzn2ekxh3d2g3uqdtdnv2: FAILED
age1ymx7x29j3nctlanp5kwt76xve66dc0dm4lpklcq5jykxr7vq55fsmhu7ts: FAILED
Investigation and cause: The list shows the
recipients of the file – your key and the Argo CD key.
The colleague is missing, because .sops.yaml only takes
effect when encrypting and with updatekeys.
Fix and verify:
# on argo
cd ~/sops/tag14/klon
sops updatekeys -y dev/notes-db.yaml 2>&1 | tail -1 | cut -c21-
XDG_CONFIG_HOME=/nirgends SOPS_AGE_KEY_FILE=~/sops/tag01/kollegin.key \
sops decrypt --extract '["metadata"]["name"]' dev/notes-db.yaml
git checkout -q .Expected:
File /home/ubuntu/sops/tag14/klon/dev/notes-db.yaml synced with new keys
notes-db
Save your progress? Sign up for free
Drill 5: Which key is missing?
Break it: Ben (Day 6) wants to read
prod/notes-db.yaml.
# on argo
cd ~/sops/tag14/klon
env -u VAULT_TOKEN XDG_CONFIG_HOME=/nirgends SOPS_AGE_KEY_FILE=~/sops/tag06/ben.key \
sops decrypt prod/notes-db.yaml 2>&1 | grep -E 'FAILED|Code'Expected:
Group 0: FAILED
age1r025nv6kv083tjhqa7xpslvkc0m8qm2jljd0aqn55nzr859xqe7q4798ks: FAILED
http://172.16.0.182:30820/v1/transit/keys/sops: FAILED
| Code: 403. Errors:
Investigation and cause: Two ways open the file: the
prod key age1r025… and Transit. Ben has neither, and
without a token OpenBao says 403. age-keygen -y tells you
which key is which:
# on argo
for k in ~/sops/tag04/prod.key ~/sops/tag06/ben.key; do
echo "$(basename $k): $(age-keygen -y $k | cut -c1-12)…"
doneExpected:
prod.key: age1r025nv6k…
ben.key: age1z32l8zxd…
Fix: Not a technical one – this is a question of
permissions. If Ben should read prod, his key goes into the prod rule
and updatekeys runs (Drill 4). If he should not, the
message is exactly right.
Save your progress? Sign up for free
Drill 6: Not a sops file – or already one
Two messages that confuse because they seem to say the opposite of what you expect.
# on argo
cd ~/sops/tag14/klon
printf 'a: 1\n' > ~/sops/tag14/klar.yaml
sops decrypt ~/sops/tag14/klar.yaml 2>&1 | tail -1
sops encrypt dev/notes-db.yaml 2>&1 | head -1Expected:
sops metadata not found
The file you have provided contains a top-level entry called 'sops', or for
Cause: metadata not found – the file is
not encrypted at all (or the sops: block is missing, for
example after copying and pasting only the values). The second message:
the file is already encrypted; sops refuses to encrypt it twice and
suggests sops edit.
Fix: Check first with sops filestatus
instead of guessing:
# on argo
sops filestatus ~/sops/tag14/klar.yaml
sops filestatus ~/sops/tag14/klon/dev/notes-db.yamlExpected:
{"encrypted":false}
{"encrypted":true}
Save your progress? Sign up for free
Drill 7: SealedSecret from the wrong cluster
Break it: A colleague seals with the certificate of “his” cluster. Here, a self-generated certificate simulates the foreign cluster.
# on argo
cd ~/sops/tag14
openssl req -x509 -newkey rsa:4096 -nodes -keyout fremd.key -out fremd.pem \
-days 30 -subj "/CN=anderer-cluster" 2>/dev/null
kubectl create secret generic falsch -n notes-stage --from-literal=x=lab-x-123 \
--dry-run=client -o yaml | kubeseal --cert fremd.pem -o yaml > falsch.sealed.yaml
kubectl apply -f falsch.sealed.yaml
sleep 3Symptom:
# on argo
kubectl -n notes-stage get sealedsecret falsch \
-o jsonpath='{.status.conditions[0].message}'; echoExpected:
no key could decrypt secret (x)
Investigate: The same message as with the wrong
scope (Day 12). You can catch it before the merge –
kubeseal --validate asks the controller:
# on argo
cd ~/sops/tag14
kubeseal --validate < falsch.sealed.yaml; echo "rc=$?"
kubeseal --validate < ~/sops/repo/sealed/notes-stage/notes-db.sealed.yaml; echo "rc=$?"Expected:
error: unable to decrypt sealed secret: falsch
rc=1
rc=0
Cause and fix: Wrong certificate. Seal again with
kubeseal --fetch-cert of this cluster (or with the
cert.pem from the repo).
# on argo
cd ~/sops/tag14
kubectl delete -f falsch.sealed.yaml
shred -u fremd.key && rm fremd.pem falsch.sealed.yamlExpected:
sealedsecret.bitnami.com "falsch" deleted from notes-stage namespace
Save your progress? Sign up for free
Drill 8: My value disappears
Break it and symptom: A quick fix at night – change the password directly in the Secret and look right away.
# on argo
kubectl -n notes-stage patch secret notes-db -p '{"stringData":{"password":"von-hand"}}'
kubectl -n notes-stage get secret notes-db -o jsonpath='{.data.password}' \
| base64 -d; echoExpected:
secret/notes-db patched
lab-db-stage-456
The patch was accepted – and the old value is already back.
Cause: The Secret belongs to the SealedSecret
(ownerReferences, Day 12). The controller watches its
Secrets and restores the sealed state. Flux and Argo CD with self-heal
work the same way: the source of truth is Git.
Fix: The way through Git – seal again with
--merge-into (Day 12), commit, apply. For an emergency
without Git: delete the SealedSecret temporarily
(--cascade=orphan keeps the Secret) and then maintain the
Secret by hand – and record that in the ticket.
Save your progress? Sign up for free
Exercise: Speed Drills
Match each message to a cause in 10 seconds at most:
MAC mismatch. File has …, computed …Could not decrypt with AES_GCM: cipher: message authentication failedGroup 0: FAILEDwith exactly the keys from.sops.yaml– just not yoursno key could decrypt secret (password)right after a renamesops metadata not found
Solution
- Plain text changed outside of sops (Drill 2).
- An
ENC[…]value is damaged, usually by hand (Drill 1). - You are not a recipient of the file (Drill 4/5) –
.sops.yamlsays nothing about the file itself. - Scope
strict, name or namespace changed (Day 12). - The file is not encrypted (any more) (Drill 6).
Save your progress? Sign up for free
Checkpoint
# on argo
cd ~/sops/tag14/klon && git status -s | wc -l
git log --oneline origin/main..user | wc -l
kubectl get sealedsecrets -n notes-stage --no-headers | awk '{print $1, $2}'Expected:
0
3
notes-api True
notes-db True
regcred True
The clone only has local commits (branch user with the
resolved merge); none of them is in Forgejo.
Challenge
A file is broken, but you do not know how:
~/sops/tag14/raetsel.yaml. Build it yourself – copy
dev/neu.yaml from the clone and change one
random line in it without looking which one (for example with
sed and a random number). Use sops decrypt and
the messages from this day to find out what kind of damage it is.
Hint: shuf -i 1-20 -n 1 gives you a
random number.
Solution
# on argo
cd ~/sops/tag14
cp klon/dev/neu.yaml raetsel.yaml
N=$(shuf -i 1-20 -n 1)
sed -i "${N}s/[a-z]/X/" raetsel.yaml
sops decrypt raetsel.yaml >/dev/null 2>fehler.txt; echo "rc=$?"
grep -oE 'MAC mismatch|AES_GCM|metadata not found|yaml: .*|Failed to get the data key' \
fehler.txt | head -1Expected (one of them depending on the line, or
rc=0):
rc=1
MAC mismatch
rc=0 means: the line was in a part that sops does not
check (a comment in the metadata, or the spelling of a key in the
sops block that changes no value) – then try again.
AES_GCM = an ENC[…] value;
MAC mismatch = plain text;
Failed to get the data key = the sops.age
block is damaged.
Save your progress? Sign up for free
Cleanup
# on argo
rm -rf ~/sops/tag14
cd ~/sops/repo && git status -sb | head -1Expected:
## main...origin/main
The main clone is unchanged. The controller, notes-stage
and all keys stay for Day 15.
Interview Note
- sops says
MAC mismatch– what happened? Someone changed the file outside of sops, usually the plain text. If intended: decrypt with--ignore-macand encrypt again; if not: treat it as tampering. - How do you resolve a merge conflict in a sops file? Take
one side and apply the other side’s change with
sops set. Never mix theENC[…]lines by hand. - I added the recipient to
.sops.yaml, why doesn’t it work?updatekeysis missing.
Enterprise Note
- CI checks every sops file:
sops filestatus(encrypted?) and a decryption attempt with the CI key – that finds broken MACs before the merge. - For SealedSecrets, the same with
kubeseal --validateagainst the target cluster. - Short branches and one file per Secret keep merge conflicts rare.
- A runbook per message (like the table above) saves the most time at night.