Day 13: GitOps and Infrastructure Pipelines
Goal: The Git repo contains only references to secrets (ExternalSecret), and OpenTofu reads secrets from OpenBao with a short-lived pipeline token, without storing them in the state. · Duration: approx. 90 min · Prerequisite: Day 3, Day 10
Briefing
- GitOps means: Git is the source of truth. Secrets still do not belong in it, not even encrypted, when OpenBao is already the source.
- Argo CD synchronizes ExternalSecrets, ESO in the cluster fetches the values (Day 3). Argo CD never sees a secret.
argocd-vault-pluginreplaces placeholders during rendering. The secrets then end up in the rendered manifest and in Argo CD’s cache.- OpenTofu talks to OpenBao via the
hashicorp/vaultprovider. Data sources write secrets into the state, ephemeral resources do not.
Git: ExternalSecret ──▶ Argo CD ──▶ cluster ──▶ ESO ──▶ OpenBao ──▶ K8s Secret
Pipeline: JWT ──▶ tofu (auth_login_jwt) ──▶ OpenBao ──▶ ephemeral ──▶ write-only
Drill 1: A GitOps Repo Without Secrets
The repo contains the reference to the secret and the app. Sync waves
define the order: first the ExternalSecret (wave -1), then
the Deployment.
mkdir -p ~/secrets-lab/tag13/gitops/{apps,shop} && cd ~/secrets-lab/tag13/gitops
cat > shop/externalsecret.yaml <<'EOF'
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: shop-api-gitops
namespace: shop
annotations:
argocd.argoproj.io/sync-wave: "-1"
spec:
refreshInterval: 1m
secretStoreRef:
kind: SecretStore
name: openbao # SecretStore from Day 3
target:
name: shop-api-gitops
data:
- secretKey: DB_PASSWORD
remoteRef:
key: shop/api
property: db_password
EOF
cat > shop/deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: shop-api-gitops
namespace: shop
annotations:
argocd.argoproj.io/sync-wave: "0"
spec:
replicas: 1
selector:
matchLabels: {app: shop-api-gitops}
template:
metadata:
labels: {app: shop-api-gitops}
spec:
serviceAccountName: shop-api
containers:
- name: app
image: busybox:1.37
command: ["sh", "-c", "echo Secret da: ${DB_PASSWORD:+ja}; sleep 3600"]
envFrom:
- secretRef:
name: shop-api-gitops
EOF
git init -q && git add -A && git commit -qm "shop: ExternalSecret + Deployment"
git grep -n -E 'lab-[a-z-]*-[0-9]+' && echo "✘ Klartext im Repo" \
|| echo "✔ kein Secret-Wert im Repo"Expected:
✔ kein Secret-Wert im Repo
Without Argo CD, you check the ESO part directly in the cluster from Day 3:
kubectl apply -f shop/externalsecret.yaml
kubectl -n shop wait externalsecret/shop-api-gitops --for=condition=Ready --timeout=60s
kubectl -n shop get secret shop-api-gitops -o jsonpath='{.data.DB_PASSWORD}' \
| base64 -d; echoExpected (abridged, may differ depending on version):
externalsecret.external-secrets.io/shop-api-gitops created
externalsecret.external-secrets.io/shop-api-gitops condition met
lab-db-pw-123
Save your progress? Sign up for free
Drill 2: Argo CD – App of Apps, Sync Waves, Health
A root Application points to apps/; every file there is
a child Application (not executed, checked against the Argo CD
docs):
cat > apps/shop.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: shop
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL: https://git.example.com/acme/gitops.git
targetRevision: main
path: shop
destination:
server: https://kubernetes.default.svc
namespace: shop
syncPolicy:
automated:
prune: true
selfHeal: true
EOF
cat > root.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: root
namespace: argocd
spec:
project: default
source:
repoURL: https://git.example.com/acme/gitops.git
targetRevision: main
path: apps
destination:
server: https://kubernetes.default.svc
namespace: argocd
syncPolicy:
automated: {}
EOF
git add -A && git commit -qm "Argo CD: App-of-Apps" && git rev-list --count HEADExpected:
2
Argo CD ships a built-in health check for
ExternalSecret: Ready=True is
Healthy, Ready=False is Degraded.
Because wave -1 must be healthy first, the Deployment never
starts with a missing secret. In the UI or via the CLI it looks like
this (abridged, may differ depending on version):
$ argocd app get shop
GROUP KIND NAMESPACE NAME STATUS HEALTH
external-secrets.io ExternalSecret shop shop-api-gitops Synced Healthy
apps Deployment shop shop-api-gitops Synced Healthy
Warning: The K8s Secret created by ESO is not in Git, and Argo CD does not manage it. With
creationPolicy: Owner(default) it carries anownerReferenceto the ExternalSecret and disappears with it when Argo CD prunes the ExternalSecret.
Save your progress? Sign up for free
Drill 3: argocd-vault-plugin – Weighing the Alternative
With the plugin, the placeholder sits in the manifest, and Argo CD replaces it during rendering (pattern, do not run):
apiVersion: v1
kind: Secret
metadata:
name: shop-api-avp
namespace: shop
type: Opaque
stringData:
DB_PASSWORD: <path:secret/data/shop/api#db_password>The plugin runs in the repo server with AVP_TYPE=vault,
VAULT_ADDR and, for example, AVP_AUTH_TYPE=k8s
plus AVP_K8S_ROLE. The comparison:
| Criterion | ESO (Drill 1) | argocd-vault-plugin |
|---|---|---|
| Who reads OpenBao? | ESO in the target cluster | Argo CD repo server |
| Secret in the Argo CD cache | no | yes, plain text in the Redis cache |
| Rotation | refreshInterval, without a Git commit |
only on the next render |
| Project status | active | last version 1.18.1 (June 2024) |
The Argo CD docs explicitly advise against secret injection during rendering and recommend solutions in the target cluster such as ESO.
Save your progress? Sign up for free
Drill 4: OpenTofu Against OpenBao – and the State Problem
The hashicorp/vault provider speaks the Vault API and
works with OpenBao. It reads VAULT_ADDR and
VAULT_TOKEN, not the BAO_* variables. First
the pipeline role for the infra repo (admin step):
cd ~/secrets-lab/tag13 && cp ../tag10/{mint-jwt.sh,ci-issuer.key} .
export KEY=$PWD/ci-issuer.key # lab issuer key, also from subdirectories
export BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN=root
bao policy write tofu-plan - <<'HCL'
path "secret/data/ci/deploy" {
capabilities = ["read"]
}
path "secret/data/ci/tofu/*" {
capabilities = ["create", "update", "read"]
}
path "secret/metadata/ci/tofu/*" {
capabilities = ["read", "delete"]
}
HCL
bao write auth/jwt-ci/role/tofu-main - <<'EOF'
{
"role_type": "jwt", "user_claim": "sub", "bound_audiences": ["openbao"],
"bound_claims": {"repository": "acme/infra", "ref": "refs/heads/main"},
"token_policies": ["tofu-plan"], "token_ttl": "20m", "token_max_ttl": "30m"
}
EOF
INFRA="repository=acme/infra sub=repo:acme/infra:ref:refs/heads/main"Expected:
Success! Uploaded policy: tofu-plan
Success! Data written to: auth/jwt-ci/role/tofu-main
Now the naive approach with a data source:
mkdir -p tofu-data && cd tofu-data
cat > main.tf <<'EOF'
terraform {
required_providers {
vault = {
source = "hashicorp/vault"
version = "~> 5.12"
}
}
}
provider "vault" {}
data "vault_kv_secret_v2" "deploy" {
mount = "secret"
name = "ci/deploy"
}
output "deploy_keys" {
value = nonsensitive(keys(data.vault_kv_secret_v2.deploy.data))
}
EOF
tofu init -no-color >/dev/null && echo "✔ init"
export VAULT_ADDR=$BAO_ADDR
export VAULT_TOKEN=$(BAO_TOKEN= bao write -field=token auth/jwt-ci/login \
role=tofu-main jwt="$(../mint-jwt.sh $INFRA)")
tofu apply -auto-approve -no-color 2>&1 | grep -E '^(Error|Deprecated)' | sort -uExpected:
✔ init
Deprecated. Please use new Ephemeral KVV2 Secret resource
Error: failed to create limited child token: Error making API request.
Two lessons: the data source is deprecated, and by default the
provider creates a child token. The pipeline token is not allowed to do
that (rightly so). skip_child_token turns that off:
python3 - <<'PY'
s = open("main.tf").read()
s = s.replace('provider "vault" {}', 'provider "vault" {\n skip_child_token = true\n}')
open("main.tf", "w").write(s)
PY
tofu apply -auto-approve -no-color 2>&1 | sed -n '/^Outputs:/,$p'
grep -c lab-reg-pw terraform.tfstate
jq '.resources[0].instances[0].attributes.data' terraform.tfstateExpected:
Outputs:
deploy_keys = tolist([
"kubeconfig_token",
"registry_password",
])
2
{
"kubeconfig_token": "lab-kube-token",
"registry_password": "lab-reg-pw"
}
sensitive only hides the value in the output. In the
state it is in plain text, and the state usually lives in a bucket that
many people can read.
Save your progress? Sign up for free
Drill 5: Ephemeral Resource and Write-Only Argument
From 1.11 on, OpenTofu supports ephemeral blocks: the
value exists only during the run. You may use it only in ephemeral
places, such as provider configurations, locals, other
ephemeral values or write-only arguments (*_wo). Example:
the registry password is copied into an app path without ever landing in
the state.
mkdir -p ../tofu-eph && cd ../tofu-eph
cat > main.tf <<'EOF'
terraform {
required_providers {
vault = {
source = "hashicorp/vault"
version = "~> 5.12"
}
}
}
provider "vault" {
skip_child_token = true
}
ephemeral "vault_kv_secret_v2" "deploy" {
mount = "secret"
name = "ci/deploy"
}
# Consumer: write-only argument, never lands in the state
resource "vault_kv_secret_v2" "app_registry" {
mount = "secret"
name = "ci/tofu/app-registry"
data_json_wo = jsonencode({
password = ephemeral.vault_kv_secret_v2.deploy.data.registry_password
})
data_json_wo_version = 1
}
EOF
tofu init -no-color >/dev/null && echo "✔ init"
tofu apply -auto-approve -no-color 2>&1 | grep -E 'Opening|Apply complete'
grep -c lab-reg-pw terraform.tfstate
jq '.resources[0].instances[0].attributes | {data_json_wo, data_json_wo_version}' \
terraform.tfstate
BAO_TOKEN=root bao kv get -field=password secret/ci/tofu/app-registry; echoExpected:
✔ init
ephemeral.vault_kv_secret_v2.deploy: Opening...
ephemeral.vault_kv_secret_v2.deploy: Opening...
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
0
{
"data_json_wo": null,
"data_json_wo_version": 1
}
lab-reg-pw
The secret is opened twice, once for the plan and once for the apply,
and then closed again. You write new values by incrementing
data_json_wo_version. The provider cannot compare
write-only values.
Save your progress? Sign up for free
Drill 6: A Short-Lived Token Directly in the Provider
Instead of fetching the token beforehand with bao, the
provider logs in itself with the pipeline JWT. The JWT comes from
TERRAFORM_VAULT_AUTH_JWT; in the CI job, for example, from
$OPENBAO_ID_TOKEN (GitLab) or the OIDC call (GitHub, Day
11).
python3 - <<'PY'
s = open("main.tf").read()
s = s.replace('provider "vault" {\n skip_child_token = true\n}', '''provider "vault" {
skip_child_token = true
auth_login_jwt {
mount = "jwt-ci"
role = "tofu-main"
}
}''')
open("main.tf", "w").write(s)
PY
unset VAULT_TOKEN
export TERRAFORM_VAULT_AUTH_JWT=$(../mint-jwt.sh $INFRA)
tofu plan -no-color 2>&1 | grep -E 'No changes|Error'
export TERRAFORM_VAULT_AUTH_JWT=$(../mint-jwt.sh $INFRA ref=refs/heads/feature)
tofu plan -no-color 2>&1 | grep -A4 'Code: 400' | tr -s '\n'
unset TERRAFORM_VAULT_AUTH_JWTExpected:
No changes. Your infrastructure matches the configuration.
Code: 400. Errors:
* error validating claims: claim "ref" does not match any associated bound
claim values
Plans on feature branches thus get a separate role with
read permissions; apply only
main. The 20 minutes of token_ttl are
intentional: the provider warns in the debug log below 15 minutes,
because long applies would otherwise get a 403 in the middle of the
run.
Save your progress? Sign up for free
Checkpoint
cat > check.sh <<'EOF'
#!/usr/bin/env bash
cd ~/secrets-lab/tag13
git -C gitops grep -q -E 'lab-[a-z-]*-[0-9]+' && echo "✘ Secret-Wert in Git" \
|| echo "✔ GitOps-Repo ohne Secret-Werte"
grep -q 'argocd.argoproj.io/sync-wave: "-1"' gitops/shop/externalsecret.yaml \
&& echo "✔ ExternalSecret in Welle -1" || echo "✘ Sync-Wave fehlt"
grep -q lab-reg-pw tofu-eph/terraform.tfstate && echo "✘ Secret im ephemeral-State" \
|| echo "✔ ephemeral-State sauber"
grep -q lab-reg-pw tofu-data/terraform.tfstate \
&& echo "✔ Data-Source-State enthält Klartext (erwartet, Gegenbeispiel)"
grep -q 'skip_child_token = true' tofu-eph/main.tf && echo "✔ skip_child_token" \
|| echo "✘ skip_child_token fehlt"
EOF
bash check.shExpected:
✔ GitOps-Repo ohne Secret-Werte
✔ ExternalSecret in Welle -1
✔ ephemeral-State sauber
✔ Data-Source-State enthält Klartext (erwartet, Gegenbeispiel)
✔ skip_child_token
Challenge
The JWT roles themselves should be created as code. Create the role
deploy-staging (repo acme/shop, branch
staging, policy ci-deploy, 5 min) with
OpenTofu and verify the login with your lab issuer.
Hint: The resource is called
vault_jwt_auth_backend_role; bound_claims is a
map of strings, TTLs are in seconds. For this admin run in the lab, you
use root.
Solution
mkdir -p ../tofu-roles && cd ../tofu-roles
cat > main.tf <<'EOF'
terraform {
required_providers {
vault = {
source = "hashicorp/vault"
version = "~> 5.12"
}
}
}
provider "vault" {
skip_child_token = true
}
resource "vault_jwt_auth_backend_role" "deploy_staging" {
backend = "jwt-ci"
role_name = "deploy-staging"
role_type = "jwt"
user_claim = "sub"
bound_audiences = ["openbao"]
bound_claims = {
repository = "acme/shop"
ref = "refs/heads/staging"
}
token_policies = ["ci-deploy"]
token_ttl = 300
}
EOF
tofu init -no-color >/dev/null
VAULT_TOKEN=root tofu apply -auto-approve -no-color | grep 'Apply complete'
BAO_TOKEN= bao write -field=token_policies auth/jwt-ci/login role=deploy-staging \
jwt="$(../mint-jwt.sh ref=refs/heads/staging)"; echoExpected:
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
[ci-deploy default]
Save your progress? Sign up for free
Enterprise Note
- Encrypt the state backend (OpenTofu
encryptionblock) and still keep secrets out of it: ephemeral plus write-only, forbid data sources for secrets (review, policy as code). - Separate roles:
tofu-plan(read, all branches) andtofu-apply(write permissions, onlymain, protected environment). Never admin permissions onsys/in a normal run. - The provider checks the server version via
sys/seal-status; OpenBao reports 2.x. For feature checks against Vault versions, setskip_get_vault_versionandvault_version_overridedeliberately if needed, and test. - Argo CD: ESO per tenant with its own SecretStore (Day 4); Argo
projects allow only
ExternalSecret, never aSecretwith content (Kyverno, Day 14). - Test new provider versions against OpenBao in a test pipeline before the lockfile version goes up.