Skip to content
Hands-on IT IT Drills

Day 15: Competition – Disaster Drills

Goal: Find and fix eight failures of the kind that happen in everyday work with OpenTofu, each with a time limit. · Duration: approx. 120 min · Prerequisite: Day 14

Briefing

  • Each round has a game master: a block that introduces the fault. Read it only after the round, otherwise it’s not practice. Ideally, someone else types it.
  • Situation describes what is reported, time limit the time until the solution.
  • Tools: the error message (read all of it!), tofu plan, tofu state list, git diff, docker ps, s3ls.
 Error ──► Where does it break? init, plan, or apply?
       ──► What has changed? Code (git diff), state, real world
       ──► What does the plan say before you apply anything?

Start of the day – a dedicated environment wk on port 8097, with remote state, encryption, and module v1.2.0. The code goes into Git so every change becomes visible:

source ~/tf-lab/setup/env.sh
mkdir -p ~/tf-lab/tag15 && cd ~/tf-lab/tag15
sed 's|tag05/terraform.tfstate|tag15/terraform.tfstate|' ~/tf-lab/tag05/backend.tf \
  > backend.tf
cp ~/tf-lab/tag05/versions.tf ~/tf-lab/tag05/encryption.tf .
cat > main.tf <<'EOF'
provider "docker" {}

variable "instanzen" {
  type    = set(string)
  default = ["a", "b"]
}

module "notes" {
  source      = "git::file:///home/ubuntu/tf-lab/module//notes-docker?ref=v1.2.0"
  name        = "wk"
  port        = 8097
  instanzen   = var.instanzen
  db_passwort = "lab-db-pw-123"
}
EOF
printf '.terraform/\n' > .gitignore
tofu init -no-color >/dev/null
tofu apply -auto-approve -no-color | tail -1
git init -q -b main && git add -A
git -c user.name=Lab -c [email protected] commit -qm "Start"
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8097/healthz

Expected:

Apply complete! Resources: 10 added, 0 changed, 0 destroyed.
200

Round 1: Locked

Game master:

L='{"ID":"7d1c2a9e-0b6f-4c1e-9a55-3f0e8b2d4c11","Operation":"OperationTypeApply",'
L+='"Info":"","Who":"kollege@laptop","Version":"1.13.2",'
L+='"Created":"2026-11-02T07:12:00Z","Path":"tofu-state/tag15/terraform.tfstate"}'
curl -s -o /dev/null --aws-sigv4 "aws:amz:us-east-1:s3" \
  --user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" -X PUT --data "$L" \
  http://127.0.0.1:7070/tofu-state/tag15/terraform.tfstate.tflock

Situation: “My colleague started an apply early this morning, then his laptop crashed. Since then nothing works for me.” Time limit: 10 min

cd ~/tf-lab/tag15
tofu plan -no-color -lock-timeout=5s 2>&1 | grep -E '^Error|ID:|Who:'

Expected:

Error: Error acquiring the state lock
  ID:        7d1c2a9e-0b6f-4c1e-9a55-3f0e8b2d4c11
  Who:       kollege@laptop

Solution

The lock from Day 9 is still in the bucket. First make sure that really no run is active anymore (ask colleagues, check CI) – then release the lock with its ID:

s3ls 'tofu-state?list-type=2' | grep -o '<Key>tag15[^<]*</Key>'
tofu force-unlock -force 7d1c2a9e-0b6f-4c1e-9a55-3f0e8b2d4c11 -no-color | head -1
tofu plan -no-color | tail -1

Expected:

<Key>tag15/terraform.tfstate</Key>
<Key>tag15/terraform.tfstate.tflock</Key>
OpenTofu state has been successfully unlocked!
No changes. Your infrastructure matches the configuration.

-lock-timeout makes OpenTofu wait a while – good against brief overlaps, useless against an orphaned lock. force-unlock on a lock that still has a run behind it can destroy the state.

Save your progress? Sign up for free

Round 2: The Proxy Won’t Come Up

Game master:

docker rm -f wk-proxy
docker run -d --name fremd-web -p 127.0.0.1:8097:80 nginx:1.31.8-alpine

Situation: “notes on 8097 only shows the nginx welcome page. An apply should fix that, but it fails.” Time limit: 10 min

tofu apply -auto-approve -no-color 2>&1 | grep -oE 'Bind for [0-9.:]+ failed: .*'

Expected:

Bind for 127.0.0.1:8097 failed: port is already allocated

Solution

A foreign container is holding the port. Which one?

docker ps --filter publish=8097 --format '{{.Names}} {{.Image}}'
docker rm -f fremd-web
tofu apply -auto-approve -no-color | tail -1
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8097/healthz

Expected:

fremd-web nginx:1.31.8-alpine
fremd-web
Apply complete! Resources: 1 added, 0 changed, 1 destroyed.
200

The failed proxy is marked tainted in the state. That’s why the second apply replaces it instead of just starting it. OpenTofu only manages what’s in the state – it doesn’t see foreign resources, even when they’re in the way.

Save your progress? Sign up for free

Round 3: The Provider Update

Game master:

sed -i 's/version = "~> 4.6"/version = "~> 5.0"/' versions.tf

Situation: “I bumped the Docker provider to version 5, it’s supposed to be faster. Now not even init works.” Time limit: 10 min

tofu init -no-color 2>&1 | grep -E '^Error|constraint'

Expected:

Error: Failed to resolve provider packages
version constraint ~> 4.6, ~> 5.0; must use tofu init -upgrade to allow

Solution

The lock file says 4.6, the constraint 5.0. The obvious advice from the message gets you further:

tofu init -upgrade -no-color 2>&1 | grep -E '^Error|no available'

Expected:

Error: Failed to resolve provider packages
Could not resolve provider kreuzwerker/docker: no available releases match

There is no version 5. Back to the state in Git:

git diff --stat
git checkout versions.tf
tofu init -no-color >/dev/null && tofu plan -no-color | tail -1

Expected:

 versions.tf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Updated 1 path from the index
No changes. Your infrastructure matches the configuration.

Provider updates belong in a separate PR with init -upgrade, a new lock file, and a plan. Tools like Renovate propose them automatically – only versions that exist.

Save your progress? Sign up for free

Round 4: Just a New Name

Game master:

sed -i 's/^module "notes" {$/module "notes_app" {/' main.tf
tofu init -no-color >/dev/null

Situation: “Pure refactoring, the module name is just more descriptive. But the plan looks strange.” Time limit: 10 min

tofu plan -no-color | grep 'Plan:'

Expected:

Plan: 10 to add, 0 to change, 10 to destroy.

Solution

For OpenTofu, module.notes_app is a new module and module.notes a deleted one. An apply would rebuild the database including its volume. A moved block (Day 11) says that it’s the same thing:

cat > moved.tf <<'EOF'
moved {
  from = module.notes
  to   = module.notes_app
}
EOF
tofu plan -no-color | grep -cE 'has moved to'
tofu apply -auto-approve -no-color | tail -1
git add -A
git -c user.name=Lab -c [email protected] commit -qm "notes_app"

Expected:

10
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

A whole module moves with a single moved – all ten resources in it along with it. The rename and moved land in the same commit.

Save your progress? Sign up for free

Round 5: The Ghost Container

Game master:

tofu state rm module.notes_app.docker_container.db -no-color

Situation: “I just wanted to clean up the state. Now OpenTofu wants to create the database and fails.” Time limit: 15 min

tofu apply -auto-approve -no-color 2>&1 \
  | grep -oE 'Conflict\. The container name "[^"]*"'

Expected:

Conflict. The container name "/wk-db"

Solution

The container is still running; only the state has forgotten it. It has to go back into the state – with an import block:

ID=$(docker ps -aq --no-trunc --filter name=^wk-db$)
cat > import.tf <<EOF
import {
  to = module.notes_app.docker_container.db
  id = "$ID"
}
EOF
tofu plan -no-color | grep 'Plan:'
tofu apply -auto-approve -no-color | tail -1
rm import.tf
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8097/healthz

Expected:

Plan: 1 to import, 1 to add, 0 to change, 1 to destroy.
Apply complete! Resources: 1 imported, 1 added, 0 changed, 1 destroyed.
200

As on Day 11, the import replaces the container. That’s acceptable here: the data lives in the volume, and the state never forgot that. The faster alternative – docker rm -f wk-db and apply – leads to the same result, but it’s an intervention that bypasses OpenTofu.

Save your progress? Sign up for free

Round 6: More Instances

Game master:

printf 'instanzen = "a,b,c"\n' > wk.auto.tfvars

Situation: “I added a third instance, but the plan aborts.” Time limit: 5 min

tofu plan -no-color 2>&1 | grep -E '^Error|required'

Expected:

Error: Invalid value for input variable
set of string required, but have string.

Solution

instanzen is a set, not a string with commas:

printf 'instanzen = ["a", "b", "c"]\n' > wk.auto.tfvars
tofu apply -auto-approve -no-color | tail -1
docker ps --format '{{.Names}}' | grep -c '^wk-api-'

Expected:

Apply complete! Resources: 2 added, 0 changed, 1 destroyed.
3

A new instance plus a new proxy. The variable’s type declaration caught the error before the first access to Docker – that’s exactly what it’s there for.

Save your progress? Sign up for free

Round 7: Unreadable

Game master:

export TF_VAR_state_passphrase=lab-state-passphrase-124

Situation: “Since switching computers, I can’t read the state anymore. Is it broken?” Time limit: 10 min

tofu plan -no-color 2>&1 | grep -E '^Error'

Expected:

Error: error loading state: decryption failed for all provided methods

Solution

The state is intact, the key is wrong. The passphrase comes from the environment:

env | grep -c '^TF_VAR_state_passphrase='
grep 'TF_VAR_state_passphrase' ~/tf-lab/setup/env.sh | cut -d= -f2 | cut -c1-20
echo "$TF_VAR_state_passphrase"
source ~/tf-lab/setup/env.sh
tofu plan -no-color | tail -1

Expected:

1
lab-state-passphrase
lab-state-passphrase-124
No changes. Your infrastructure matches the configuration.

One digit of difference, no access – that’s how encryption should be. In operations, the key therefore comes from a vault or KMS, not from memory.

Save your progress? Sign up for free

Round 8: The New Module Version

Game master:

sed -i 's/ref=v1.2.0/ref=v1.3.0/' main.tf

Situation: “The platform team announced module version 1.3.0. I switched over, but init fails.” Time limit: 5 min

tofu init -no-color 2>&1 | grep -oE 'invalid ref: "[^"]*"'

Expected:

invalid ref: "v1.3.0"

Solution

Announced isn’t released. Which versions exist?

git -C ~/tf-lab/module tag
git checkout main.tf
tofu init -no-color >/dev/null && tofu plan -no-color | tail -1

Expected:

v1.0.0
v1.1.0
v1.2.0
Updated 1 path from the index
No changes. Your infrastructure matches the configuration.

git checkout restores the last commit – including the rename from Round 4. Whoever enters a module version first checks with git tag (or in the module repository’s release list) whether it exists.

Save your progress? Sign up for free

Scoring

Round Cause Tool
1 orphaned lock force-unlock with ID
2 foreign container on the port docker ps --filter publish=
3 provider version doesn’t exist init -upgrade, git checkout
4 rename without moved moved block
5 resource removed from the state import block
6 wrong type in tfvars the variable’s type
7 wrong passphrase env, env.sh
8 module version not released git tag

Under 75 minutes for all eight rounds: you’re ready for team operations.

Checkpoint

cd ~/tf-lab/tag15 && tofu plan -no-color | tail -1

Expected:

No changes. Your infrastructure matches the configuration.
s3ls 'tofu-state?list-type=2' | grep -c 'tag15/terraform.tfstate.tflock'

Expected:

0
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8097/healthz

Expected:

200

Enterprise Note

  • Runbooks for the classics: Orphaned lock, import after state rm, wrong key – each of these situations belongs in the team’s runbook, with commands.
  • state rm and force-unlock only with the four-eyes principle: Both commands bypass plan and review. Whoever needs them tells the team beforehand what they’re about to do.
  • Versioned state buckets: With object versions in the bucket, a botched state can be restored to how it was an hour ago. That’s the last resort when all the rounds of this day go wrong.