Skip to content
Hands-on IT IT Drills

Day 9: Remote State, Locking, and Encryption

Goal: Move the state from the project folder into S3 storage, prevent concurrent runs with a lock, and protect the state with OpenTofu’s encryption. · Duration: approx. 90 min · Prerequisite: Day 8

Briefing

  • A backend determines where the state lives. The default is local (the file terraform.tfstate). In a team it lives centrally, for example in an S3 bucket.
  • If two apply runs happen at the same time, the second overwrites the work of the first. A lock prevents that: OpenTofu creates a lock file next to the state – only if the storage can do this atomically (“conditional writes”).
  • The state contains passwords and keys. OpenTofu can encrypt it before it reaches the backend – the storage only sees ciphertext.
  • Chicken and egg: the storage for the state isn’t managed with the state it stores. It is created once by hand or in a small configuration of its own.
 tag05, tag06 … ─► backend "s3" ─► Versity S3 Gateway :7070 ─► bucket tofu-state
                     use_lockfile    (volume s3-daten)          tag05/terraform.tfstate
                     encryption: aes_gcm, key from passphrase        …tfstate.tflock

Terraform: The S3 backend with use_lockfile has existed there since 1.11. State encryption in the encryption block exists only in OpenTofu.

Drill 1: S3 Storage

The Versity S3 Gateway serves as S3 storage: a lean service that offers a folder as an S3 API and supports conditional writes.

mkdir -p ~/tf-lab/tag09 && cd ~/tf-lab/tag09
docker volume create s3-daten >/dev/null
docker run -d --name s3 --restart unless-stopped -p 127.0.0.1:7070:7070 \
  -e ROOT_ACCESS_KEY=tofu -e ROOT_SECRET_KEY=lab-s3-pw-123456 \
  -v s3-daten:/data versity/versitygw:v1.10.0 posix /data | cut -c1-12
cat >> ~/tf-lab/setup/env.sh <<'EOF'
export AWS_ACCESS_KEY_ID=tofu                         # S3-Speicher (ab Tag 9)
export AWS_SECRET_ACCESS_KEY=lab-s3-pw-123456
EOF
source ~/tf-lab/setup/env.sh
sleep 2
curl -s -o /dev/null -w 'Bucket anlegen: %{http_code}\n' -X PUT \
  --aws-sigv4 "aws:amz:us-east-1:s3" \
  --user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" http://127.0.0.1:7070/tofu-state
cat >> ~/tf-lab/setup/env.sh <<'EOF'
s3ls() { curl -s --aws-sigv4 "aws:amz:us-east-1:s3" \
  --user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" "http://127.0.0.1:7070/$1"; }
EOF
source ~/tf-lab/setup/env.sh
s3ls | grep -o '<Name>[^<]*</Name>'

Expected:

3f9d2c1a7b4e
Bucket anlegen: 200
<Name>tofu-state</Name>
  • curl --aws-sigv4 signs a request the way S3 requires – without a dedicated S3 tool. PUT /tofu-state creates the bucket, GET / lists all of them.
  • s3ls is the same curl call as a small function in env.sh: s3ls lists the buckets, s3ls 'tofu-state?list-type=2' the objects, s3ls tofu-state/<key> fetches one.
  • OpenTofu reads the credentials from AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. They are now in env.sh – fine in the lab; in practice they come from a secret store.

Save your progress? Sign up for free

Drill 2: Migrating the State

The state of the two notes stacks from Day 5 moves into the bucket:

cd ~/tf-lab/tag05
cat > backend.tf <<'EOF'
terraform {
  backend "s3" {
    bucket                      = "tofu-state"
    key                         = "tag05/terraform.tfstate"
    region                      = "us-east-1"
    endpoints                   = { s3 = "http://127.0.0.1:7070" }
    use_path_style              = true
    use_lockfile                = true
    skip_credentials_validation = true
    skip_requesting_account_id  = true
    skip_metadata_api_check     = true
    skip_region_validation      = true
  }
}
EOF
tofu init -migrate-state -force-copy -no-color | grep -E 'Successfully|copy|initialized'
s3ls 'tofu-state?list-type=2' | grep -o '<Key>[^<]*</Key>'
wc -c < terraform.tfstate
tofu plan -no-color | tail -1

Expected:

Successfully configured the backend "s3"! OpenTofu will automatically
OpenTofu has been successfully initialized!
<Key>tag05/terraform.tfstate</Key>
0
No changes. Your infrastructure matches the configuration.
  • -migrate-state copies the existing state into the new backend; -force-copy answers the confirmation prompt with “yes”.
  • The local file is now empty; the state lives in the bucket. Same addresses, same objects – the plan sees no difference.
  • The skip_… settings turn off checks that only make sense with real AWS.

Save your progress? Sign up for free

Drill 3: The Lock

A small practice configuration with a deliberately slow step:

mkdir -p ~/tf-lab/tag09/sperre && cd ~/tf-lab/tag09/sperre
sed 's|tag05/terraform.tfstate|tag09/sperre.tfstate|' ~/tf-lab/tag05/backend.tf \
  > backend.tf
cat > main.tf <<'EOF'
resource "terraform_data" "langsam" {
  triggers_replace = timestamp()
  provisioner "local-exec" {
    command = "sleep 15"
  }
}
EOF
tofu init -no-color >/dev/null
tofu apply -auto-approve -no-color > erster.log 2>&1 &
sleep 6
s3ls 'tofu-state?list-type=2' | grep -o '<Key>[^<]*sperre[^<]*</Key>'
tofu plan -no-color -lock-timeout=0s 2>&1 | grep -E 'Error acquiring|Operation:|Who:'
wait
grep 'Apply complete' erster.log

Expected:

<Key>tag09/sperre.tfstate.tflock</Key>
Error: Error acquiring the state lock
  Operation: OperationTypeApply
  Who:       ubuntu@tf
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
  • While the first run is working, ….tfstate.tflock sits in the bucket. The second doesn’t get it and aborts – with details on who holds the lock and what they’re doing.
  • -lock-timeout=60s would wait up to a minute instead.
  • If a lock is left behind after a crash, tofu force-unlock <ID> releases it. Only when you’re sure no other run is still working (Day 15).

Save your progress? Sign up for free

Drill 4: Encrypting the State

cd ~/tf-lab/tag05
s3ls tofu-state/tag05/terraform.tfstate | grep -c 'lab-db-pw-123'
cat > encryption.tf <<'EOF'
variable "state_passphrase" {
  type      = string
  sensitive = true
}

terraform {
  encryption {
    key_provider "pbkdf2" "lab" {
      passphrase = var.state_passphrase
    }
    method "aes_gcm" "lab" {
      keys = key_provider.pbkdf2.lab
    }
    method "unencrypted" "migration" {}
    state {
      method = method.aes_gcm.lab
      fallback {
        method = method.unencrypted.migration
      }
    }
  }
}
EOF
echo 'export TF_VAR_state_passphrase=lab-state-passphrase-123      # nur Labor' \
  >> ~/tf-lab/setup/env.sh
source ~/tf-lab/setup/env.sh
tofu apply -auto-approve -no-color | grep 'Apply complete'
s3ls tofu-state/tag05/terraform.tfstate | jq -c 'keys'
s3ls tofu-state/tag05/terraform.tfstate | grep -c 'lab-db-pw-123'

Expected:

1
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
["encrypted_data","encryption_version","lineage","meta","serial"]
0
  • Before, the database password was in the bucket in plain text (grep -c counts the line – the state is one single long JSON line); now there is only ciphertext. The key is derived from the passphrase (PBKDF2) and never leaves the machine where tofu runs.
  • fallback with unencrypted allows OpenTofu to read the old unencrypted state once. The apply writes it back encrypted.

Now the way back is closed – the fallback method is removed:

python3 - <<'PY'
import re
s = open("encryption.tf").read()
s = re.sub(r'\n    method "unencrypted" "migration" \{\}', "", s)
s = re.sub(r"\n      fallback \{.*?\n      \}", "", s, flags=re.S)
open("encryption.tf", "w").write(s)
PY
tofu plan -no-color | tail -1
TF_VAR_state_passphrase=falsche-passphrase-123 tofu plan -no-color 2>&1 \
  | grep -m1 'Error'

Expected:

No changes. Your infrastructure matches the configuration.
Error: error loading state: decryption failed for all provided methods

Without the right passphrase, nobody can read the state – not even you. It belongs in a password manager or a vault, not just in env.sh.

Save your progress? Sign up for free

Checkpoint

s3ls 'tofu-state?list-type=2' | grep -o '<Key>[^<]*</Key>'

Expected:

<Key>tag05/terraform.tfstate</Key>
<Key>tag09/sperre.tfstate</Key>
cd ~/tf-lab/tag05 && tofu plan -detailed-exitcode -no-color >/dev/null; echo "Exit: $?"

Expected:

Exit: 0
docker ps --filter name=^s3$ --format '{{.Names}} {{.Image}}'

Expected:

s3 versity/versitygw:v1.10.0

Challenge

Move the state of Day 6 (notes on Kubernetes) into the bucket as well, under the key tag06/terraform.tfstate, and encrypt it right away.

Hint: Copy backend.tf and encryption.tf from Day 5. The first run needs the fallback method again – or you encrypt directly during the move, since the local state is unencrypted anyway.

Solution

cd ~/tf-lab/tag06
export TF_VAR_vm_ip=$VM_IP
sed 's|tag05/terraform.tfstate|tag06/terraform.tfstate|' ~/tf-lab/tag05/backend.tf \
  > backend.tf
cat > encryption.tf <<'EOF'
variable "state_passphrase" {
  type      = string
  sensitive = true
}

terraform {
  encryption {
    key_provider "pbkdf2" "lab" {
      passphrase = var.state_passphrase
    }
    method "aes_gcm" "lab" {
      keys = key_provider.pbkdf2.lab
    }
    method "unencrypted" "migration" {}
    state {
      method = method.aes_gcm.lab
      fallback {
        method = method.unencrypted.migration
      }
    }
  }
}
EOF
tofu init -migrate-state -force-copy -no-color >/dev/null
tofu apply -auto-approve -no-color | grep 'Apply complete'
s3ls tofu-state/tag06/terraform.tfstate | jq -r 'has("encrypted_data")'

Expected:

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
true

During the move, OpenTofu reads the local state via the fallback and writes it encrypted to the bucket. Afterwards you can remove the fallback as in Drill 4.

Save your progress? Sign up for free

Enterprise Note

  • Real backends: S3 (AWS or compatible), Azure Storage, Google Cloud Storage, pg (Postgres), HTTP. What matters are locking, bucket versioning, and restricted access – the state is as valuable as the credentials in it.
  • Keys instead of a passphrase: In production the key comes from a KMS (AWS, GCP) or OpenBao/Vault (key_provider "openbao") and is rotated regularly.
  • One bucket, many states: One key per configuration and environment (team/dienst/umgebung.tfstate). Permissions per prefix prevent one team from reading another team’s state.