Day 9: Remote State, Locking, and Encryption
Goal: Move the state from the project folder into S3 storage, prevent concurrent runs with a lock, and protect the state with OpenTofu’s encryption. · Duration: approx. 90 min · Prerequisite: Day 8
Briefing
- A backend determines where the state lives. The
default is
local(the fileterraform.tfstate). In a team it lives centrally, for example in an S3 bucket. - If two
applyruns happen at the same time, the second overwrites the work of the first. A lock prevents that: OpenTofu creates a lock file next to the state – only if the storage can do this atomically (“conditional writes”). - The state contains passwords and keys. OpenTofu can encrypt it before it reaches the backend – the storage only sees ciphertext.
- Chicken and egg: the storage for the state isn’t managed with the state it stores. It is created once by hand or in a small configuration of its own.
tag05, tag06 … ─► backend "s3" ─► Versity S3 Gateway :7070 ─► bucket tofu-state
use_lockfile (volume s3-daten) tag05/terraform.tfstate
encryption: aes_gcm, key from passphrase …tfstate.tflock
Terraform: The S3 backend with
use_lockfilehas existed there since 1.11. State encryption in theencryptionblock exists only in OpenTofu.
Drill 1: S3 Storage
The Versity S3 Gateway serves as S3 storage: a lean service that offers a folder as an S3 API and supports conditional writes.
mkdir -p ~/tf-lab/tag09 && cd ~/tf-lab/tag09
docker volume create s3-daten >/dev/null
docker run -d --name s3 --restart unless-stopped -p 127.0.0.1:7070:7070 \
-e ROOT_ACCESS_KEY=tofu -e ROOT_SECRET_KEY=lab-s3-pw-123456 \
-v s3-daten:/data versity/versitygw:v1.10.0 posix /data | cut -c1-12
cat >> ~/tf-lab/setup/env.sh <<'EOF'
export AWS_ACCESS_KEY_ID=tofu # S3-Speicher (ab Tag 9)
export AWS_SECRET_ACCESS_KEY=lab-s3-pw-123456
EOF
source ~/tf-lab/setup/env.sh
sleep 2
curl -s -o /dev/null -w 'Bucket anlegen: %{http_code}\n' -X PUT \
--aws-sigv4 "aws:amz:us-east-1:s3" \
--user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" http://127.0.0.1:7070/tofu-state
cat >> ~/tf-lab/setup/env.sh <<'EOF'
s3ls() { curl -s --aws-sigv4 "aws:amz:us-east-1:s3" \
--user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" "http://127.0.0.1:7070/$1"; }
EOF
source ~/tf-lab/setup/env.sh
s3ls | grep -o '<Name>[^<]*</Name>'Expected:
3f9d2c1a7b4e
Bucket anlegen: 200
<Name>tofu-state</Name>
curl --aws-sigv4signs a request the way S3 requires – without a dedicated S3 tool.PUT /tofu-statecreates the bucket,GET /lists all of them.s3lsis the samecurlcall as a small function inenv.sh:s3lslists the buckets,s3ls 'tofu-state?list-type=2'the objects,s3ls tofu-state/<key>fetches one.- OpenTofu reads the credentials from
AWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEY. They are now inenv.sh– fine in the lab; in practice they come from a secret store.
Save your progress? Sign up for free
Drill 2: Migrating the State
The state of the two notes stacks from Day 5 moves into the bucket:
cd ~/tf-lab/tag05
cat > backend.tf <<'EOF'
terraform {
backend "s3" {
bucket = "tofu-state"
key = "tag05/terraform.tfstate"
region = "us-east-1"
endpoints = { s3 = "http://127.0.0.1:7070" }
use_path_style = true
use_lockfile = true
skip_credentials_validation = true
skip_requesting_account_id = true
skip_metadata_api_check = true
skip_region_validation = true
}
}
EOF
tofu init -migrate-state -force-copy -no-color | grep -E 'Successfully|copy|initialized'
s3ls 'tofu-state?list-type=2' | grep -o '<Key>[^<]*</Key>'
wc -c < terraform.tfstate
tofu plan -no-color | tail -1Expected:
Successfully configured the backend "s3"! OpenTofu will automatically
OpenTofu has been successfully initialized!
<Key>tag05/terraform.tfstate</Key>
0
No changes. Your infrastructure matches the configuration.
-migrate-statecopies the existing state into the new backend;-force-copyanswers the confirmation prompt with “yes”.- The local file is now empty; the state lives in the bucket. Same addresses, same objects – the plan sees no difference.
- The
skip_…settings turn off checks that only make sense with real AWS.
Save your progress? Sign up for free
Drill 3: The Lock
A small practice configuration with a deliberately slow step:
mkdir -p ~/tf-lab/tag09/sperre && cd ~/tf-lab/tag09/sperre
sed 's|tag05/terraform.tfstate|tag09/sperre.tfstate|' ~/tf-lab/tag05/backend.tf \
> backend.tf
cat > main.tf <<'EOF'
resource "terraform_data" "langsam" {
triggers_replace = timestamp()
provisioner "local-exec" {
command = "sleep 15"
}
}
EOF
tofu init -no-color >/dev/null
tofu apply -auto-approve -no-color > erster.log 2>&1 &
sleep 6
s3ls 'tofu-state?list-type=2' | grep -o '<Key>[^<]*sperre[^<]*</Key>'
tofu plan -no-color -lock-timeout=0s 2>&1 | grep -E 'Error acquiring|Operation:|Who:'
wait
grep 'Apply complete' erster.logExpected:
<Key>tag09/sperre.tfstate.tflock</Key>
Error: Error acquiring the state lock
Operation: OperationTypeApply
Who: ubuntu@tf
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
- While the first run is working,
….tfstate.tflocksits in the bucket. The second doesn’t get it and aborts – with details on who holds the lock and what they’re doing. -lock-timeout=60swould wait up to a minute instead.- If a lock is left behind after a crash,
tofu force-unlock <ID>releases it. Only when you’re sure no other run is still working (Day 15).
Save your progress? Sign up for free
Drill 4: Encrypting the State
cd ~/tf-lab/tag05
s3ls tofu-state/tag05/terraform.tfstate | grep -c 'lab-db-pw-123'
cat > encryption.tf <<'EOF'
variable "state_passphrase" {
type = string
sensitive = true
}
terraform {
encryption {
key_provider "pbkdf2" "lab" {
passphrase = var.state_passphrase
}
method "aes_gcm" "lab" {
keys = key_provider.pbkdf2.lab
}
method "unencrypted" "migration" {}
state {
method = method.aes_gcm.lab
fallback {
method = method.unencrypted.migration
}
}
}
}
EOF
echo 'export TF_VAR_state_passphrase=lab-state-passphrase-123 # nur Labor' \
>> ~/tf-lab/setup/env.sh
source ~/tf-lab/setup/env.sh
tofu apply -auto-approve -no-color | grep 'Apply complete'
s3ls tofu-state/tag05/terraform.tfstate | jq -c 'keys'
s3ls tofu-state/tag05/terraform.tfstate | grep -c 'lab-db-pw-123'Expected:
1
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
["encrypted_data","encryption_version","lineage","meta","serial"]
0
- Before, the database password was in the bucket in plain text
(
grep -ccounts the line – the state is one single long JSON line); now there is only ciphertext. The key is derived from the passphrase (PBKDF2) and never leaves the machine wheretofuruns. fallbackwithunencryptedallows OpenTofu to read the old unencrypted state once. Theapplywrites it back encrypted.
Now the way back is closed – the fallback method is removed:
python3 - <<'PY'
import re
s = open("encryption.tf").read()
s = re.sub(r'\n method "unencrypted" "migration" \{\}', "", s)
s = re.sub(r"\n fallback \{.*?\n \}", "", s, flags=re.S)
open("encryption.tf", "w").write(s)
PY
tofu plan -no-color | tail -1
TF_VAR_state_passphrase=falsche-passphrase-123 tofu plan -no-color 2>&1 \
| grep -m1 'Error'Expected:
No changes. Your infrastructure matches the configuration.
Error: error loading state: decryption failed for all provided methods
Without the right passphrase, nobody can read the state – not even
you. It belongs in a password manager or a vault, not just in
env.sh.
Save your progress? Sign up for free
Checkpoint
s3ls 'tofu-state?list-type=2' | grep -o '<Key>[^<]*</Key>'Expected:
<Key>tag05/terraform.tfstate</Key>
<Key>tag09/sperre.tfstate</Key>
cd ~/tf-lab/tag05 && tofu plan -detailed-exitcode -no-color >/dev/null; echo "Exit: $?"Expected:
Exit: 0
docker ps --filter name=^s3$ --format '{{.Names}} {{.Image}}'Expected:
s3 versity/versitygw:v1.10.0
Challenge
Move the state of Day 6 (notes on Kubernetes) into the bucket as
well, under the key tag06/terraform.tfstate, and encrypt it
right away.
Hint: Copy backend.tf and
encryption.tf from Day 5. The first run needs the fallback
method again – or you encrypt directly during the move, since the local
state is unencrypted anyway.
Solution
cd ~/tf-lab/tag06
export TF_VAR_vm_ip=$VM_IP
sed 's|tag05/terraform.tfstate|tag06/terraform.tfstate|' ~/tf-lab/tag05/backend.tf \
> backend.tf
cat > encryption.tf <<'EOF'
variable "state_passphrase" {
type = string
sensitive = true
}
terraform {
encryption {
key_provider "pbkdf2" "lab" {
passphrase = var.state_passphrase
}
method "aes_gcm" "lab" {
keys = key_provider.pbkdf2.lab
}
method "unencrypted" "migration" {}
state {
method = method.aes_gcm.lab
fallback {
method = method.unencrypted.migration
}
}
}
}
EOF
tofu init -migrate-state -force-copy -no-color >/dev/null
tofu apply -auto-approve -no-color | grep 'Apply complete'
s3ls tofu-state/tag06/terraform.tfstate | jq -r 'has("encrypted_data")'Expected:
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
true
During the move, OpenTofu reads the local state via the fallback and writes it encrypted to the bucket. Afterwards you can remove the fallback as in Drill 4.
Save your progress? Sign up for free
Enterprise Note
- Real backends: S3 (AWS or compatible), Azure
Storage, Google Cloud Storage,
pg(Postgres), HTTP. What matters are locking, bucket versioning, and restricted access – the state is as valuable as the credentials in it. - Keys instead of a passphrase: In production the key
comes from a KMS (AWS, GCP) or OpenBao/Vault
(
key_provider "openbao") and is rotated regularly. - One bucket, many states: One key per configuration
and environment (
team/dienst/umgebung.tfstate). Permissions per prefix prevent one team from reading another team’s state.