Day 12: Secrets – sensitive, ephemeral, write-only
Goal: See everywhere a password ends up, and keep it out of output, code, and state step by step. · Duration: approx. 75 min · Prerequisite: Day 6, Day 9
Briefing
- A secret can leak in four places: in the code (Git), in the output (plan, CI log), in the state, and in the target resource itself. The last place is intended, the other three are not.
sensitive = truehides values in the output – only there. In the state they remain in plain text.- Ephemeral values (
ephemeral = true,ephemeralresources) exist only during a run. OpenTofu writes them neither to the plan nor to the state. - Write-only attributes (suffix
_wo) accept ephemeral values and pass them to the target resource without storing them. That way a password reaches Kubernetes but never the state.
Code Output State Target
normal ✗ ✗ ✗ ✓ ✗ = secret exposed here
sensitive ✗ ✓ ✗ ✓
ephemeral + _wo ✓ ✓ ✓ ✓
Drill 1: The Leak
A Kubernetes Secret with a password from a variable – the way you’d write it at first. Each block gets its own file – that way, in the next drills you only ever swap out one small file:
mkdir -p ~/tf-lab/tag12 && cd ~/tf-lab/tag12
sed 's|tag05/terraform.tfstate|tag12/terraform.tfstate|' ~/tf-lab/tag05/backend.tf \
> backend.tf
cat > versions.tf <<'EOF'
terraform {
required_version = ">= 1.13.0"
required_providers {
kubernetes = { source = "hashicorp/kubernetes", version = "~> 3.4" }
random = { source = "hashicorp/random", version = "~> 3.9" }
}
}
provider "kubernetes" {
config_path = "/etc/rancher/k3s/k3s.yaml"
}
EOF
cat > variables.tf <<'EOF'
variable "db_passwort" {
type = string
default = "lab-geheim-123"
}
EOF
cat > namespace.tf <<'EOF'
resource "kubernetes_namespace_v1" "tag12" {
metadata {
name = "tag12"
}
}
EOF
cat > secret.tf <<'EOF'
resource "kubernetes_secret_v1" "db" {
metadata {
name = "db"
namespace = kubernetes_namespace_v1.tag12.metadata[0].name
}
data = {
password = var.db_passwort
}
}
EOF
cat > outputs.tf <<'EOF'
output "passwort" {
value = var.db_passwort
}
EOF
tofu init -no-color >/dev/null
tofu apply -auto-approve -no-color | grep -E ' data |passwort|Apply complete'
tofu state pull | grep -o 'lab-geheim-123' | wc -lExpected:
+ data = (sensitive value)
+ passwort = "lab-geheim-123"
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
passwort = "lab-geheim-123"
2
Three leaks at once:
- Code: The password is the
defaultinvariables.tf– and therefore in Git. - Output: The output shows it in the plan and at the end of the run. In a CI pipeline it therefore ends up in the job log.
- State: Twice in plain text – once in the Secret, once in the output. The provider did mark the value in the Secret as sensitive, but OpenTofu still writes it to the state.
Save your progress? Sign up for free
Drill 2: sensitive
First step: mark the variable as sensitive.
cat > variables.tf <<'EOF'
variable "db_passwort" {
type = string
default = "lab-geheim-123"
sensitive = true
}
EOF
tofu plan -no-color 2>&1 | grep '^Error'Expected:
Error: Output refers to sensitive values
OpenTofu refuses the output: a sensitive value may only leave through an output that is sensitive as well. That’s intentional – sensitivity is “inherited” and can’t be lost by accident.
cat > outputs.tf <<'EOF'
output "passwort" {
value = var.db_passwort
sensitive = true
}
EOF
tofu apply -auto-approve -no-color | grep -E 'passwort|Apply complete'
tofu output passwort
tofu output -raw passwort; echo
tofu state pull | grep -o 'lab-geheim-123' | wc -lExpected:
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
passwort = <sensitive>
"lab-geheim-123"
lab-geheim-123
2
- The summary at the end of the run now only shows
<sensitive>– the CI log is clean. - Anyone who asks for the output specifically
(
tofu output passwort,-raw) or reads the state still gets the plain text.sensitiveprotects against prying eyes, not against access.
Save your progress? Sign up for free
Drill 3: Out of the Code
The value doesn’t belong in the repository. Two common ways: a values file that Git ignores, or an environment variable.
cat > variables.tf <<'EOF'
variable "db_passwort" {
type = string
sensitive = true
}
EOF
printf 'db_passwort = "lab-geheim-456"\n' > secret.auto.tfvars
printf '*.tfvars\n.terraform/\n' > .gitignore
tofu plan -no-color | grep -E 'will be|Plan:'
rm secret.auto.tfvars
export TF_VAR_db_passwort=lab-geheim-123
tofu plan -no-color | tail -1Expected:
# kubernetes_secret_v1.db will be updated in-place
Plan: 0 to add, 1 to change, 0 to destroy.
No changes. Your infrastructure matches the configuration.
- OpenTofu reads
*.auto.tfvarsautomatically. With the entry in.gitignore, the file doesn’t get into the repository – but it stays on disk in plain text. TF_VAR_<name>sets a variable from the environment. In CI, the value thus comes from the pipeline’s secret store, not from a file.- Without a value and without a
default, OpenTofu prompts interactively – in a pipeline the run aborts.
With that, code and output are clean. That leaves the state.
Save your progress? Sign up for free
Drill 4: ephemeral and write-only
Now the password shouldn’t end up in the state at all. For that, the variable becomes ephemeral:
cat > variables.tf <<'EOF'
variable "db_passwort" {
type = string
sensitive = true
ephemeral = true
}
EOF
tofu plan -no-color 2>&1 | grep '^Error'Expected:
Error: Output does not allow ephemeral value
Error: Ephemeral value used in non-ephemeral context
An ephemeral value may not go into an output or into a normal
attribute like data – both end up in the state. Only
targets that store nothing are allowed: write-only attributes. For the
Kubernetes Secret they are called data_wo and
data_wo_revision.
rm outputs.tf
cat > secret.tf <<'EOF'
resource "kubernetes_secret_v1" "db" {
metadata {
name = "db"
namespace = kubernetes_namespace_v1.tag12.metadata[0].name
}
data_wo = {
password = var.db_passwort
}
data_wo_revision = 1
}
EOF
tofu apply -auto-approve -no-color | grep -E 'data|Apply complete'
tofu state pull | grep -o 'lab-geheim-123' | wc -l
kubectl -n tag12 get secret db -o jsonpath='{.data.password}' | base64 -d; echoThe output goes away; the Secret gets data_wo instead of
data and a data_wo_revision.
Expected:
~ binary_data_wo = (write-only attribute)
~ data_wo = (write-only attribute)
+ data_wo_revision = 1
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
1
lab-geheim-123
Kubernetes has the value, the plan only shows “write-only attribute”
– but the password is still in the state once. For the Kubernetes
Secret, data is optional and computed: if you
leave it out, OpenTofu keeps the old value in the state. What’s in the
state once stays there until the resource is recreated:
tofu apply -auto-approve -no-color -replace=kubernetes_secret_v1.db \
| grep -E 'replaced|Apply complete'
tofu state pull | grep -o 'lab-geheim-123' | wc -l
kubectl -n tag12 get secret db -o jsonpath='{.data.password}' | base64 -d; echoExpected:
# kubernetes_secret_v1.db will be replaced, as requested
Apply complete! Resources: 1 added, 0 changed, 1 destroyed.
0
lab-geheim-123
Now the password is nowhere in the state anymore. Lesson: create secrets as write-only from the start – doing it afterwards costs a replacement, and old versions of the state (versions in the bucket, backups) still contain the value.
Save your progress? Sign up for free
Drill 5: Rotating with data_wo_revision
Because OpenTofu doesn’t store the value, it can’t detect a change either:
export TF_VAR_db_passwort=lab-geheim-789
tofu plan -no-color | tail -1Expected:
No changes. Your infrastructure matches the configuration.
A new password, but no plan. The number in
data_wo_revision is the lever: if you increase it, OpenTofu
writes the write-only value again.
sed -i 's/data_wo_revision = 1/data_wo_revision = 2/' secret.tf
tofu apply -auto-approve -no-color | grep -E 'revision|Apply complete'
kubectl -n tag12 get secret db -o jsonpath='{.data.password}' | base64 -d; echoExpected:
~ data_wo_revision = 1 -> 2
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
lab-geheim-789
The revision is in the code and in the state – the password isn’t. A review sees “a rotation happened here”, but not to what.
Save your progress? Sign up for free
Checkpoint
cd ~/tf-lab/tag12 && tofu state pull | grep -o 'lab-geheim' | wc -lExpected:
0
kubectl -n tag12 get secret db -o jsonpath='{.data.password}' | base64 -d; echoExpected:
lab-geheim-789
grep -l 'lab-geheim' *.tf | wc -lExpected:
0
Challenge
Nobody should have to make up the password anymore: a second Secret
api in the namespace tag12 gets a random
24-character password – without it appearing in the state.
Hint: The hashicorp/random provider
also offers random_password as an ephemeral resource:
ephemeral "random_password" "…" { … }. Its result is in
.result.
Solution
cd ~/tf-lab/tag12
cat > api.tf <<'EOF'
ephemeral "random_password" "api" {
length = 24
special = false
}
resource "kubernetes_secret_v1" "api" {
metadata {
name = "api"
namespace = kubernetes_namespace_v1.tag12.metadata[0].name
}
data_wo = {
password = ephemeral.random_password.api.result
}
data_wo_revision = 1
}
EOF
tofu init -no-color >/dev/null
tofu apply -auto-approve -no-color | grep 'Apply complete'
API_PW=$(kubectl -n tag12 get secret api -o jsonpath='{.data.password}' | base64 -d)
echo "${#API_PW}"
tofu state pull | grep -c "$API_PW"
tofu plan -no-color | tail -1Expected:
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
24
0
No changes. Your infrastructure matches the configuration.
On every run, the ephemeral resource generates a new password – but
it is only written when data_wo_revision changes. That’s
why the plan stays empty.
Save your progress? Sign up for free
Enterprise Note
- Encrypt the state and restrict access: Not every provider supports write-only attributes yet. State encryption from Day 9 and tight permissions on the bucket remain the second line of defense.
- Secrets from a vault: Instead of
TF_VAR_from the pipeline, many teams fetch values as an ephemeral resource directly from Vault or OpenBao. Then not even the pipeline knows the password. - Plan rotation:
data_wo_revisionbelongs in the rotation process. Whoever increments it needs the same approval as for any other change to prod.