Skip to content
Hands-on IT IT Drills

Day 12: Secrets – sensitive, ephemeral, write-only

Goal: See everywhere a password ends up, and keep it out of output, code, and state step by step. · Duration: approx. 75 min · Prerequisite: Day 6, Day 9

Briefing

  • A secret can leak in four places: in the code (Git), in the output (plan, CI log), in the state, and in the target resource itself. The last place is intended, the other three are not.
  • sensitive = true hides values in the output – only there. In the state they remain in plain text.
  • Ephemeral values (ephemeral = true, ephemeral resources) exist only during a run. OpenTofu writes them neither to the plan nor to the state.
  • Write-only attributes (suffix _wo) accept ephemeral values and pass them to the target resource without storing them. That way a password reaches Kubernetes but never the state.
                 Code   Output    State   Target
 normal           ✗       ✗        ✗       ✓      ✗ = secret exposed here
 sensitive        ✗       ✓        ✗       ✓
 ephemeral + _wo  ✓       ✓        ✓       ✓

Drill 1: The Leak

A Kubernetes Secret with a password from a variable – the way you’d write it at first. Each block gets its own file – that way, in the next drills you only ever swap out one small file:

mkdir -p ~/tf-lab/tag12 && cd ~/tf-lab/tag12
sed 's|tag05/terraform.tfstate|tag12/terraform.tfstate|' ~/tf-lab/tag05/backend.tf \
  > backend.tf
cat > versions.tf <<'EOF'
terraform {
  required_version = ">= 1.13.0"
  required_providers {
    kubernetes = { source = "hashicorp/kubernetes", version = "~> 3.4" }
    random     = { source = "hashicorp/random", version = "~> 3.9" }
  }
}

provider "kubernetes" {
  config_path = "/etc/rancher/k3s/k3s.yaml"
}
EOF
cat > variables.tf <<'EOF'
variable "db_passwort" {
  type    = string
  default = "lab-geheim-123"
}
EOF
cat > namespace.tf <<'EOF'
resource "kubernetes_namespace_v1" "tag12" {
  metadata {
    name = "tag12"
  }
}
EOF
cat > secret.tf <<'EOF'
resource "kubernetes_secret_v1" "db" {
  metadata {
    name      = "db"
    namespace = kubernetes_namespace_v1.tag12.metadata[0].name
  }
  data = {
    password = var.db_passwort
  }
}
EOF
cat > outputs.tf <<'EOF'
output "passwort" {
  value = var.db_passwort
}
EOF
tofu init -no-color >/dev/null
tofu apply -auto-approve -no-color | grep -E ' data |passwort|Apply complete'
tofu state pull | grep -o 'lab-geheim-123' | wc -l

Expected:

      + data                           = (sensitive value)
  + passwort = "lab-geheim-123"
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
passwort = "lab-geheim-123"
2

Three leaks at once:

  1. Code: The password is the default in variables.tf – and therefore in Git.
  2. Output: The output shows it in the plan and at the end of the run. In a CI pipeline it therefore ends up in the job log.
  3. State: Twice in plain text – once in the Secret, once in the output. The provider did mark the value in the Secret as sensitive, but OpenTofu still writes it to the state.

Save your progress? Sign up for free

Drill 2: sensitive

First step: mark the variable as sensitive.

cat > variables.tf <<'EOF'
variable "db_passwort" {
  type      = string
  default   = "lab-geheim-123"
  sensitive = true
}
EOF
tofu plan -no-color 2>&1 | grep '^Error'

Expected:

Error: Output refers to sensitive values

OpenTofu refuses the output: a sensitive value may only leave through an output that is sensitive as well. That’s intentional – sensitivity is “inherited” and can’t be lost by accident.

cat > outputs.tf <<'EOF'
output "passwort" {
  value     = var.db_passwort
  sensitive = true
}
EOF
tofu apply -auto-approve -no-color | grep -E 'passwort|Apply complete'
tofu output passwort
tofu output -raw passwort; echo
tofu state pull | grep -o 'lab-geheim-123' | wc -l

Expected:

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
passwort = <sensitive>
"lab-geheim-123"
lab-geheim-123
2
  • The summary at the end of the run now only shows <sensitive> – the CI log is clean.
  • Anyone who asks for the output specifically (tofu output passwort, -raw) or reads the state still gets the plain text. sensitive protects against prying eyes, not against access.

Save your progress? Sign up for free

Drill 3: Out of the Code

The value doesn’t belong in the repository. Two common ways: a values file that Git ignores, or an environment variable.

cat > variables.tf <<'EOF'
variable "db_passwort" {
  type      = string
  sensitive = true
}
EOF
printf 'db_passwort = "lab-geheim-456"\n' > secret.auto.tfvars
printf '*.tfvars\n.terraform/\n' > .gitignore
tofu plan -no-color | grep -E 'will be|Plan:'
rm secret.auto.tfvars
export TF_VAR_db_passwort=lab-geheim-123
tofu plan -no-color | tail -1

Expected:

  # kubernetes_secret_v1.db will be updated in-place
Plan: 0 to add, 1 to change, 0 to destroy.
No changes. Your infrastructure matches the configuration.
  • OpenTofu reads *.auto.tfvars automatically. With the entry in .gitignore, the file doesn’t get into the repository – but it stays on disk in plain text.
  • TF_VAR_<name> sets a variable from the environment. In CI, the value thus comes from the pipeline’s secret store, not from a file.
  • Without a value and without a default, OpenTofu prompts interactively – in a pipeline the run aborts.

With that, code and output are clean. That leaves the state.

Save your progress? Sign up for free

Drill 4: ephemeral and write-only

Now the password shouldn’t end up in the state at all. For that, the variable becomes ephemeral:

cat > variables.tf <<'EOF'
variable "db_passwort" {
  type      = string
  sensitive = true
  ephemeral = true
}
EOF
tofu plan -no-color 2>&1 | grep '^Error'

Expected:

Error: Output does not allow ephemeral value
Error: Ephemeral value used in non-ephemeral context

An ephemeral value may not go into an output or into a normal attribute like data – both end up in the state. Only targets that store nothing are allowed: write-only attributes. For the Kubernetes Secret they are called data_wo and data_wo_revision.

rm outputs.tf
cat > secret.tf <<'EOF'
resource "kubernetes_secret_v1" "db" {
  metadata {
    name      = "db"
    namespace = kubernetes_namespace_v1.tag12.metadata[0].name
  }
  data_wo = {
    password = var.db_passwort
  }
  data_wo_revision = 1
}
EOF
tofu apply -auto-approve -no-color | grep -E 'data|Apply complete'
tofu state pull | grep -o 'lab-geheim-123' | wc -l
kubectl -n tag12 get secret db -o jsonpath='{.data.password}' | base64 -d; echo

The output goes away; the Secret gets data_wo instead of data and a data_wo_revision.

Expected:

      ~ binary_data_wo                 = (write-only attribute)
      ~ data_wo                        = (write-only attribute)
      + data_wo_revision               = 1
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
1
lab-geheim-123

Kubernetes has the value, the plan only shows “write-only attribute” – but the password is still in the state once. For the Kubernetes Secret, data is optional and computed: if you leave it out, OpenTofu keeps the old value in the state. What’s in the state once stays there until the resource is recreated:

tofu apply -auto-approve -no-color -replace=kubernetes_secret_v1.db \
  | grep -E 'replaced|Apply complete'
tofu state pull | grep -o 'lab-geheim-123' | wc -l
kubectl -n tag12 get secret db -o jsonpath='{.data.password}' | base64 -d; echo

Expected:

  # kubernetes_secret_v1.db will be replaced, as requested
Apply complete! Resources: 1 added, 0 changed, 1 destroyed.
0
lab-geheim-123

Now the password is nowhere in the state anymore. Lesson: create secrets as write-only from the start – doing it afterwards costs a replacement, and old versions of the state (versions in the bucket, backups) still contain the value.

Save your progress? Sign up for free

Drill 5: Rotating with data_wo_revision

Because OpenTofu doesn’t store the value, it can’t detect a change either:

export TF_VAR_db_passwort=lab-geheim-789
tofu plan -no-color | tail -1

Expected:

No changes. Your infrastructure matches the configuration.

A new password, but no plan. The number in data_wo_revision is the lever: if you increase it, OpenTofu writes the write-only value again.

sed -i 's/data_wo_revision = 1/data_wo_revision = 2/' secret.tf
tofu apply -auto-approve -no-color | grep -E 'revision|Apply complete'
kubectl -n tag12 get secret db -o jsonpath='{.data.password}' | base64 -d; echo

Expected:

      ~ data_wo_revision = 1 -> 2
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
lab-geheim-789

The revision is in the code and in the state – the password isn’t. A review sees “a rotation happened here”, but not to what.

Save your progress? Sign up for free

Checkpoint

cd ~/tf-lab/tag12 && tofu state pull | grep -o 'lab-geheim' | wc -l

Expected:

0
kubectl -n tag12 get secret db -o jsonpath='{.data.password}' | base64 -d; echo

Expected:

lab-geheim-789
grep -l 'lab-geheim' *.tf | wc -l

Expected:

0

Challenge

Nobody should have to make up the password anymore: a second Secret api in the namespace tag12 gets a random 24-character password – without it appearing in the state.

Hint: The hashicorp/random provider also offers random_password as an ephemeral resource: ephemeral "random_password" "…" { … }. Its result is in .result.

Solution

cd ~/tf-lab/tag12
cat > api.tf <<'EOF'
ephemeral "random_password" "api" {
  length  = 24
  special = false
}

resource "kubernetes_secret_v1" "api" {
  metadata {
    name      = "api"
    namespace = kubernetes_namespace_v1.tag12.metadata[0].name
  }
  data_wo = {
    password = ephemeral.random_password.api.result
  }
  data_wo_revision = 1
}
EOF
tofu init -no-color >/dev/null
tofu apply -auto-approve -no-color | grep 'Apply complete'
API_PW=$(kubectl -n tag12 get secret api -o jsonpath='{.data.password}' | base64 -d)
echo "${#API_PW}"
tofu state pull | grep -c "$API_PW"
tofu plan -no-color | tail -1

Expected:

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
24
0
No changes. Your infrastructure matches the configuration.

On every run, the ephemeral resource generates a new password – but it is only written when data_wo_revision changes. That’s why the plan stays empty.

Save your progress? Sign up for free

Enterprise Note

  • Encrypt the state and restrict access: Not every provider supports write-only attributes yet. State encryption from Day 9 and tight permissions on the bucket remain the second line of defense.
  • Secrets from a vault: Instead of TF_VAR_ from the pipeline, many teams fetch values as an ephemeral resource directly from Vault or OpenBao. Then not even the pipeline knows the password.
  • Plan rotation: data_wo_revision belongs in the rotation process. Whoever increments it needs the same approval as for any other change to prod.