Skip to content
Hands-on IT IT Drills

Day 14: Error Patterns

Goal: You recognize eight typical mishaps with sops and Sealed Secrets by their message, find the cause with one command and fix them without losing values. · Duration: approx. 75 min · Prerequisite: Day 13

Briefing

  • Almost every sops message names the place: MAC (the file as a whole), AES_GCM (a single value), Group 0: FAILED (keys) or metadata not found (not a sops file).
  • For key errors, sops lists every recipient of the file with its result. Read the list – it tells you which key would be needed.
  • Sealed Secrets essentially knows one message: no key could decrypt secret. The cause is either the scope (Day 12) or a foreign or missing key (Day 13).
  • You practice in a clone of its own, ~/sops/tag14/klon, from which nothing is pushed. Your main clone and Forgejo stay clean.
  • Every error pattern follows the pattern break it → symptom → investigate → cause → fix → verify.
Message Layer Day
MAC mismatch file 3, Drill 2 here
Could not decrypt with AES_GCM value Drill 1 here
Group 0: FAILED keys 4, 5, Drill 4/5 here
no key could decrypt secret Sealed Secrets 12, 13, Drill 7 here

Drill 1: A hand on the encrypted value

Break it: A clone to practice in, then a “small typo” in an ENC[…] value – say, while resolving a conflict by hand.

# on argo
mkdir -p ~/sops/tag14 && cd ~/sops/tag14
git clone -q http://172.16.0.182:30300/lab/secrets.git klon
cd klon
git config core.hooksPath hooks
sed -i 's/password: ENC\[AES256_GCM,data:./password: ENC[AES256_GCM,data:A/' \
  dev/notes-db.yaml

Symptom:

# on argo
cd ~/sops/tag14/klon
sops decrypt dev/notes-db.yaml 2>&1 | tail -1

Expected (line wrapped):

Error decrypting tree: Error walking tree: Could not decrypt value: Could not decrypt
with AES_GCM: cipher: message authentication failed

Investigation and cause: git diff shows the changed line. AES-GCM protects every value with its own checksum; change a single character, and the value can no longer be opened. It cannot be rescued – only fetched back from Git.

Fix and verify:

# on argo
cd ~/sops/tag14/klon
git diff --stat
git checkout -q dev/notes-db.yaml
sops decrypt --extract '["metadata"]["name"]' dev/notes-db.yaml

Expected:

 dev/notes-db.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
notes-db

Save your progress? Sign up for free

Drill 2: Plain text changed next to it

Break it: Someone moves the Secret to another namespace – with sed, because metadata is readable after all.

# on argo
cd ~/sops/tag14/klon
sed -i 's/namespace: notes-dev/namespace: notes-qa/' dev/notes-db.yaml
sops decrypt dev/notes-db.yaml 2>&1 | cut -c1-40

Expected:

MAC mismatch. File has 66C57B21830010AFB

Cause: The MAC covers all values, including the unencrypted ones (Day 3). Any change to the plain text outside of sops breaks it.

Fix: The change is intended – so decrypt with --ignore-mac and encrypt again. That computes the MAC over the new state.

# on argo
cd ~/sops/tag14/klon
sops decrypt --ignore-mac dev/notes-db.yaml \
  | sops encrypt --filename-override dev/notes-db.yaml /dev/stdin \
  > ~/sops/tag14/fix.yaml
mv ~/sops/tag14/fix.yaml dev/notes-db.yaml
sops decrypt --extract '["metadata"]["namespace"]' dev/notes-db.yaml
git checkout -q dev/notes-db.yaml

Expected:

notes-qa

Tip: --ignore-mac means: “I know who changed the file.” For a file from someone else, a MAC error is an alarm signal, not a nuisance.

Save your progress? Sign up for free

Drill 3: Merge conflict

Break it: Two branches change different values of the same file.

# on argo
cd ~/sops/tag14/klon
git switch -q -c pw
sops set dev/notes-db.yaml '["data"]["password"]' "\"$(printf lab-db-pw-1 | base64)\""
git commit -q -am "Passwort"
git switch -q main && git switch -q -c user
sops set dev/notes-db.yaml '["data"]["user"]' "\"$(printf notes | base64)\""
git commit -q -am "Benutzer"
git merge -q pw 2>&1 | grep CONFLICT

Expected:

CONFLICT (content): Merge conflict in dev/notes-db.yaml

Symptom and investigation:

# on argo
cd ~/sops/tag14/klon
grep -c '^<<<<<<<' dev/notes-db.yaml
sops decrypt dev/notes-db.yaml 2>&1 | tail -1

Expected:

2
yaml: line 3: could not find expected ':'

Cause: Both sides changed lastmodified and mac, plus their values. Git cannot merge that, and doing it by hand does not work (Drill 1).

Fix: Take one side completely and apply the other side’s change with sops. git show :3:file is “their” version; /dev/stdin with --filename-override reads it without writing it to disk.

# on argo
cd ~/sops/tag14/klon
git checkout --ours dev/notes-db.yaml
diff <(git show :3:dev/notes-db.yaml \
        | sops decrypt --filename-override dev/notes-db.yaml /dev/stdin) \
     <(sops decrypt dev/notes-db.yaml)
P=$(git show :3:dev/notes-db.yaml | sops decrypt \
  --filename-override dev/notes-db.yaml --extract '["data"]["password"]' /dev/stdin)
sops set dev/notes-db.yaml '["data"]["password"]' "\"$P\""
git add dev/notes-db.yaml && git commit -q -m "Merge pw"

Expected (abridged):

Updated 1 path from the index
3c3,4
<   password: bGFiLWRiLXB3LTE=
---
>   password: bGFiLWRiLWRldi05OTk=
>   user: bm90ZXM=

Verify:

# on argo
cd ~/sops/tag14/klon
sops decrypt dev/notes-db.yaml | yq '.data | map_values(@base64d)'
git switch -q main

Expected:

password: lab-db-pw-1
user: notes

Tip: Fewer conflicts arise when every Secret has its own file and branches are short-lived. The project itself has no merge driver for sops.

Save your progress? Sign up for free

Drill 4: Recipient only in .sops.yaml

Break it: The colleague (Day 5) should read dev/. Someone adds her to the rule – and commits.

# on argo
cd ~/sops/tag14/klon
K=$(age-keygen -y ~/sops/tag01/kollegin.key)
yq -i '(.creation_rules[] | select(.path_regex == "^dev/.*\\.yaml$")).age
  += ",'"$K"'"' .sops.yaml

Symptom: The colleague still cannot read it.

# on argo
cd ~/sops/tag14/klon
XDG_CONFIG_HOME=/nirgends SOPS_AGE_KEY_FILE=~/sops/tag01/kollegin.key \
  sops decrypt dev/notes-db.yaml 2>&1 | grep -E 'Group|FAILED'

Expected (abridged):

Group 0: FAILED
  age1rnz4tqupd0f3lm4kfs564cfgys5j80497rk7ajzn2ekxh3d2g3uqdtdnv2: FAILED
  age1ymx7x29j3nctlanp5kwt76xve66dc0dm4lpklcq5jykxr7vq55fsmhu7ts: FAILED

Investigation and cause: The list shows the recipients of the file – your key and the Argo CD key. The colleague is missing, because .sops.yaml only takes effect when encrypting and with updatekeys.

Fix and verify:

# on argo
cd ~/sops/tag14/klon
sops updatekeys -y dev/notes-db.yaml 2>&1 | tail -1 | cut -c21-
XDG_CONFIG_HOME=/nirgends SOPS_AGE_KEY_FILE=~/sops/tag01/kollegin.key \
  sops decrypt --extract '["metadata"]["name"]' dev/notes-db.yaml
git checkout -q .

Expected:

File /home/ubuntu/sops/tag14/klon/dev/notes-db.yaml synced with new keys
notes-db

Save your progress? Sign up for free

Drill 5: Which key is missing?

Break it: Ben (Day 6) wants to read prod/notes-db.yaml.

# on argo
cd ~/sops/tag14/klon
env -u VAULT_TOKEN XDG_CONFIG_HOME=/nirgends SOPS_AGE_KEY_FILE=~/sops/tag06/ben.key \
  sops decrypt prod/notes-db.yaml 2>&1 | grep -E 'FAILED|Code'

Expected:

Group 0: FAILED
  age1r025nv6kv083tjhqa7xpslvkc0m8qm2jljd0aqn55nzr859xqe7q4798ks: FAILED
  http://172.16.0.182:30820/v1/transit/keys/sops: FAILED
      | Code: 403. Errors:

Investigation and cause: Two ways open the file: the prod key age1r025… and Transit. Ben has neither, and without a token OpenBao says 403. age-keygen -y tells you which key is which:

# on argo
for k in ~/sops/tag04/prod.key ~/sops/tag06/ben.key; do
  echo "$(basename $k): $(age-keygen -y $k | cut -c1-12)…"
done

Expected:

prod.key: age1r025nv6k…
ben.key: age1z32l8zxd…

Fix: Not a technical one – this is a question of permissions. If Ben should read prod, his key goes into the prod rule and updatekeys runs (Drill 4). If he should not, the message is exactly right.

Save your progress? Sign up for free

Drill 6: Not a sops file – or already one

Two messages that confuse because they seem to say the opposite of what you expect.

# on argo
cd ~/sops/tag14/klon
printf 'a: 1\n' > ~/sops/tag14/klar.yaml
sops decrypt ~/sops/tag14/klar.yaml 2>&1 | tail -1
sops encrypt dev/notes-db.yaml 2>&1 | head -1

Expected:

sops metadata not found
The file you have provided contains a top-level entry called 'sops', or for

Cause: metadata not found – the file is not encrypted at all (or the sops: block is missing, for example after copying and pasting only the values). The second message: the file is already encrypted; sops refuses to encrypt it twice and suggests sops edit.

Fix: Check first with sops filestatus instead of guessing:

# on argo
sops filestatus ~/sops/tag14/klar.yaml
sops filestatus ~/sops/tag14/klon/dev/notes-db.yaml

Expected:

{"encrypted":false}
{"encrypted":true}

Save your progress? Sign up for free

Drill 7: SealedSecret from the wrong cluster

Break it: A colleague seals with the certificate of “his” cluster. Here, a self-generated certificate simulates the foreign cluster.

# on argo
cd ~/sops/tag14
openssl req -x509 -newkey rsa:4096 -nodes -keyout fremd.key -out fremd.pem \
  -days 30 -subj "/CN=anderer-cluster" 2>/dev/null
kubectl create secret generic falsch -n notes-stage --from-literal=x=lab-x-123 \
  --dry-run=client -o yaml | kubeseal --cert fremd.pem -o yaml > falsch.sealed.yaml
kubectl apply -f falsch.sealed.yaml
sleep 3

Symptom:

# on argo
kubectl -n notes-stage get sealedsecret falsch \
  -o jsonpath='{.status.conditions[0].message}'; echo

Expected:

no key could decrypt secret (x)

Investigate: The same message as with the wrong scope (Day 12). You can catch it before the merge – kubeseal --validate asks the controller:

# on argo
cd ~/sops/tag14
kubeseal --validate < falsch.sealed.yaml; echo "rc=$?"
kubeseal --validate < ~/sops/repo/sealed/notes-stage/notes-db.sealed.yaml; echo "rc=$?"

Expected:

error: unable to decrypt sealed secret: falsch
rc=1
rc=0

Cause and fix: Wrong certificate. Seal again with kubeseal --fetch-cert of this cluster (or with the cert.pem from the repo).

# on argo
cd ~/sops/tag14
kubectl delete -f falsch.sealed.yaml
shred -u fremd.key && rm fremd.pem falsch.sealed.yaml

Expected:

sealedsecret.bitnami.com "falsch" deleted from notes-stage namespace

Save your progress? Sign up for free

Drill 8: My value disappears

Break it and symptom: A quick fix at night – change the password directly in the Secret and look right away.

# on argo
kubectl -n notes-stage patch secret notes-db -p '{"stringData":{"password":"von-hand"}}'
kubectl -n notes-stage get secret notes-db -o jsonpath='{.data.password}' \
  | base64 -d; echo

Expected:

secret/notes-db patched
lab-db-stage-456

The patch was accepted – and the old value is already back.

Cause: The Secret belongs to the SealedSecret (ownerReferences, Day 12). The controller watches its Secrets and restores the sealed state. Flux and Argo CD with self-heal work the same way: the source of truth is Git.

Fix: The way through Git – seal again with --merge-into (Day 12), commit, apply. For an emergency without Git: delete the SealedSecret temporarily (--cascade=orphan keeps the Secret) and then maintain the Secret by hand – and record that in the ticket.

Save your progress? Sign up for free

Exercise: Speed Drills

Match each message to a cause in 10 seconds at most:

  1. MAC mismatch. File has …, computed …
  2. Could not decrypt with AES_GCM: cipher: message authentication failed
  3. Group 0: FAILED with exactly the keys from .sops.yaml – just not yours
  4. no key could decrypt secret (password) right after a rename
  5. sops metadata not found

Solution

  1. Plain text changed outside of sops (Drill 2).
  2. An ENC[…] value is damaged, usually by hand (Drill 1).
  3. You are not a recipient of the file (Drill 4/5) – .sops.yaml says nothing about the file itself.
  4. Scope strict, name or namespace changed (Day 12).
  5. The file is not encrypted (any more) (Drill 6).

Save your progress? Sign up for free

Checkpoint

# on argo
cd ~/sops/tag14/klon && git status -s | wc -l
git log --oneline origin/main..user | wc -l
kubectl get sealedsecrets -n notes-stage --no-headers | awk '{print $1, $2}'

Expected:

0
3
notes-api True
notes-db True
regcred True

The clone only has local commits (branch user with the resolved merge); none of them is in Forgejo.

Challenge

A file is broken, but you do not know how: ~/sops/tag14/raetsel.yaml. Build it yourself – copy dev/neu.yaml from the clone and change one random line in it without looking which one (for example with sed and a random number). Use sops decrypt and the messages from this day to find out what kind of damage it is.

Hint: shuf -i 1-20 -n 1 gives you a random number.

Solution

# on argo
cd ~/sops/tag14
cp klon/dev/neu.yaml raetsel.yaml
N=$(shuf -i 1-20 -n 1)
sed -i "${N}s/[a-z]/X/" raetsel.yaml
sops decrypt raetsel.yaml >/dev/null 2>fehler.txt; echo "rc=$?"
grep -oE 'MAC mismatch|AES_GCM|metadata not found|yaml: .*|Failed to get the data key' \
  fehler.txt | head -1

Expected (one of them depending on the line, or rc=0):

rc=1
MAC mismatch

rc=0 means: the line was in a part that sops does not check (a comment in the metadata, or the spelling of a key in the sops block that changes no value) – then try again. AES_GCM = an ENC[…] value; MAC mismatch = plain text; Failed to get the data key = the sops.age block is damaged.

Save your progress? Sign up for free

Cleanup

# on argo
rm -rf ~/sops/tag14
cd ~/sops/repo && git status -sb | head -1

Expected:

## main...origin/main

The main clone is unchanged. The controller, notes-stage and all keys stay for Day 15.

Interview Note

  • sops says MAC mismatch – what happened? Someone changed the file outside of sops, usually the plain text. If intended: decrypt with --ignore-mac and encrypt again; if not: treat it as tampering.
  • How do you resolve a merge conflict in a sops file? Take one side and apply the other side’s change with sops set. Never mix the ENC[…] lines by hand.
  • I added the recipient to .sops.yaml, why doesn’t it work? updatekeys is missing.

Enterprise Note

  • CI checks every sops file: sops filestatus (encrypted?) and a decryption attempt with the CI key – that finds broken MACs before the merge.
  • For SealedSecrets, the same with kubeseal --validate against the target cluster.
  • Short branches and one file per Secret keep merge conflicts rare.
  • A runbook per message (like the table above) saves the most time at night.