Skip to content
Hands-on IT IT Drills

Day 13: GitOps and Infrastructure Pipelines

Goal: The Git repo contains only references to secrets (ExternalSecret), and OpenTofu reads secrets from OpenBao with a short-lived pipeline token, without storing them in the state. · Duration: approx. 90 min · Prerequisite: Day 3, Day 10

Briefing

  • GitOps means: Git is the source of truth. Secrets still do not belong in it, not even encrypted, when OpenBao is already the source.
  • Argo CD synchronizes ExternalSecrets, ESO in the cluster fetches the values (Day 3). Argo CD never sees a secret.
  • argocd-vault-plugin replaces placeholders during rendering. The secrets then end up in the rendered manifest and in Argo CD’s cache.
  • OpenTofu talks to OpenBao via the hashicorp/vault provider. Data sources write secrets into the state, ephemeral resources do not.
Git: ExternalSecret ──▶ Argo CD ──▶ cluster ──▶ ESO ──▶ OpenBao ──▶ K8s Secret
Pipeline: JWT ──▶ tofu (auth_login_jwt) ──▶ OpenBao ──▶ ephemeral ──▶ write-only

Drill 1: A GitOps Repo Without Secrets

The repo contains the reference to the secret and the app. Sync waves define the order: first the ExternalSecret (wave -1), then the Deployment.

mkdir -p ~/secrets-lab/tag13/gitops/{apps,shop} && cd ~/secrets-lab/tag13/gitops
cat > shop/externalsecret.yaml <<'EOF'
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: shop-api-gitops
  namespace: shop
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
spec:
  refreshInterval: 1m
  secretStoreRef:
    kind: SecretStore
    name: openbao          # SecretStore from Day 3
  target:
    name: shop-api-gitops
  data:
    - secretKey: DB_PASSWORD
      remoteRef:
        key: shop/api
        property: db_password
EOF
cat > shop/deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: shop-api-gitops
  namespace: shop
  annotations:
    argocd.argoproj.io/sync-wave: "0"
spec:
  replicas: 1
  selector:
    matchLabels: {app: shop-api-gitops}
  template:
    metadata:
      labels: {app: shop-api-gitops}
    spec:
      serviceAccountName: shop-api
      containers:
        - name: app
          image: busybox:1.37
          command: ["sh", "-c", "echo Secret da: ${DB_PASSWORD:+ja}; sleep 3600"]
          envFrom:
            - secretRef:
                name: shop-api-gitops
EOF
git init -q && git add -A && git commit -qm "shop: ExternalSecret + Deployment"
git grep -n -E 'lab-[a-z-]*-[0-9]+' && echo "✘ Klartext im Repo" \
  || echo "✔ kein Secret-Wert im Repo"

Expected:

✔ kein Secret-Wert im Repo

Without Argo CD, you check the ESO part directly in the cluster from Day 3:

kubectl apply -f shop/externalsecret.yaml
kubectl -n shop wait externalsecret/shop-api-gitops --for=condition=Ready --timeout=60s
kubectl -n shop get secret shop-api-gitops -o jsonpath='{.data.DB_PASSWORD}' \
  | base64 -d; echo

Expected (abridged, may differ depending on version):

externalsecret.external-secrets.io/shop-api-gitops created
externalsecret.external-secrets.io/shop-api-gitops condition met
lab-db-pw-123

Save your progress? Sign up for free

Drill 2: Argo CD – App of Apps, Sync Waves, Health

A root Application points to apps/; every file there is a child Application (not executed, checked against the Argo CD docs):

cat > apps/shop.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: shop
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://git.example.com/acme/gitops.git
    targetRevision: main
    path: shop
  destination:
    server: https://kubernetes.default.svc
    namespace: shop
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
EOF
cat > root.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://git.example.com/acme/gitops.git
    targetRevision: main
    path: apps
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated: {}
EOF
git add -A && git commit -qm "Argo CD: App-of-Apps" && git rev-list --count HEAD

Expected:

2

Argo CD ships a built-in health check for ExternalSecret: Ready=True is Healthy, Ready=False is Degraded. Because wave -1 must be healthy first, the Deployment never starts with a missing secret. In the UI or via the CLI it looks like this (abridged, may differ depending on version):

$ argocd app get shop
GROUP               KIND            NAMESPACE  NAME             STATUS  HEALTH
external-secrets.io ExternalSecret  shop       shop-api-gitops  Synced  Healthy
apps                Deployment      shop       shop-api-gitops  Synced  Healthy

Warning: The K8s Secret created by ESO is not in Git, and Argo CD does not manage it. With creationPolicy: Owner (default) it carries an ownerReference to the ExternalSecret and disappears with it when Argo CD prunes the ExternalSecret.

Save your progress? Sign up for free

Drill 3: argocd-vault-plugin – Weighing the Alternative

With the plugin, the placeholder sits in the manifest, and Argo CD replaces it during rendering (pattern, do not run):

apiVersion: v1
kind: Secret
metadata:
  name: shop-api-avp
  namespace: shop
type: Opaque
stringData:
  DB_PASSWORD: <path:secret/data/shop/api#db_password>

The plugin runs in the repo server with AVP_TYPE=vault, VAULT_ADDR and, for example, AVP_AUTH_TYPE=k8s plus AVP_K8S_ROLE. The comparison:

Criterion ESO (Drill 1) argocd-vault-plugin
Who reads OpenBao? ESO in the target cluster Argo CD repo server
Secret in the Argo CD cache no yes, plain text in the Redis cache
Rotation refreshInterval, without a Git commit only on the next render
Project status active last version 1.18.1 (June 2024)

The Argo CD docs explicitly advise against secret injection during rendering and recommend solutions in the target cluster such as ESO.

Save your progress? Sign up for free

Drill 4: OpenTofu Against OpenBao – and the State Problem

The hashicorp/vault provider speaks the Vault API and works with OpenBao. It reads VAULT_ADDR and VAULT_TOKEN, not the BAO_* variables. First the pipeline role for the infra repo (admin step):

cd ~/secrets-lab/tag13 && cp ../tag10/{mint-jwt.sh,ci-issuer.key} .
export KEY=$PWD/ci-issuer.key   # lab issuer key, also from subdirectories
export BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN=root
bao policy write tofu-plan - <<'HCL'
path "secret/data/ci/deploy" {
  capabilities = ["read"]
}
path "secret/data/ci/tofu/*" {
  capabilities = ["create", "update", "read"]
}
path "secret/metadata/ci/tofu/*" {
  capabilities = ["read", "delete"]
}
HCL
bao write auth/jwt-ci/role/tofu-main - <<'EOF'
{
  "role_type": "jwt", "user_claim": "sub", "bound_audiences": ["openbao"],
  "bound_claims": {"repository": "acme/infra", "ref": "refs/heads/main"},
  "token_policies": ["tofu-plan"], "token_ttl": "20m", "token_max_ttl": "30m"
}
EOF
INFRA="repository=acme/infra sub=repo:acme/infra:ref:refs/heads/main"

Expected:

Success! Uploaded policy: tofu-plan
Success! Data written to: auth/jwt-ci/role/tofu-main

Now the naive approach with a data source:

mkdir -p tofu-data && cd tofu-data
cat > main.tf <<'EOF'
terraform {
  required_providers {
    vault = {
      source  = "hashicorp/vault"
      version = "~> 5.12"
    }
  }
}

provider "vault" {}

data "vault_kv_secret_v2" "deploy" {
  mount = "secret"
  name  = "ci/deploy"
}

output "deploy_keys" {
  value = nonsensitive(keys(data.vault_kv_secret_v2.deploy.data))
}
EOF
tofu init -no-color >/dev/null && echo "✔ init"
export VAULT_ADDR=$BAO_ADDR
export VAULT_TOKEN=$(BAO_TOKEN= bao write -field=token auth/jwt-ci/login \
  role=tofu-main jwt="$(../mint-jwt.sh $INFRA)")
tofu apply -auto-approve -no-color 2>&1 | grep -E '^(Error|Deprecated)' | sort -u

Expected:

✔ init
Deprecated. Please use new Ephemeral KVV2 Secret resource
Error: failed to create limited child token: Error making API request.

Two lessons: the data source is deprecated, and by default the provider creates a child token. The pipeline token is not allowed to do that (rightly so). skip_child_token turns that off:

python3 - <<'PY'
s = open("main.tf").read()
s = s.replace('provider "vault" {}', 'provider "vault" {\n  skip_child_token = true\n}')
open("main.tf", "w").write(s)
PY
tofu apply -auto-approve -no-color 2>&1 | sed -n '/^Outputs:/,$p'
grep -c lab-reg-pw terraform.tfstate
jq '.resources[0].instances[0].attributes.data' terraform.tfstate

Expected:

Outputs:

deploy_keys = tolist([
  "kubeconfig_token",
  "registry_password",
])
2
{
  "kubeconfig_token": "lab-kube-token",
  "registry_password": "lab-reg-pw"
}

sensitive only hides the value in the output. In the state it is in plain text, and the state usually lives in a bucket that many people can read.

Save your progress? Sign up for free

Drill 5: Ephemeral Resource and Write-Only Argument

From 1.11 on, OpenTofu supports ephemeral blocks: the value exists only during the run. You may use it only in ephemeral places, such as provider configurations, locals, other ephemeral values or write-only arguments (*_wo). Example: the registry password is copied into an app path without ever landing in the state.

mkdir -p ../tofu-eph && cd ../tofu-eph
cat > main.tf <<'EOF'
terraform {
  required_providers {
    vault = {
      source  = "hashicorp/vault"
      version = "~> 5.12"
    }
  }
}

provider "vault" {
  skip_child_token = true
}

ephemeral "vault_kv_secret_v2" "deploy" {
  mount = "secret"
  name  = "ci/deploy"
}

# Consumer: write-only argument, never lands in the state
resource "vault_kv_secret_v2" "app_registry" {
  mount = "secret"
  name  = "ci/tofu/app-registry"
  data_json_wo = jsonencode({
    password = ephemeral.vault_kv_secret_v2.deploy.data.registry_password
  })
  data_json_wo_version = 1
}
EOF
tofu init -no-color >/dev/null && echo "✔ init"
tofu apply -auto-approve -no-color 2>&1 | grep -E 'Opening|Apply complete'
grep -c lab-reg-pw terraform.tfstate
jq '.resources[0].instances[0].attributes | {data_json_wo, data_json_wo_version}' \
  terraform.tfstate
BAO_TOKEN=root bao kv get -field=password secret/ci/tofu/app-registry; echo

Expected:

✔ init
ephemeral.vault_kv_secret_v2.deploy: Opening...
ephemeral.vault_kv_secret_v2.deploy: Opening...
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
0
{
  "data_json_wo": null,
  "data_json_wo_version": 1
}
lab-reg-pw

The secret is opened twice, once for the plan and once for the apply, and then closed again. You write new values by incrementing data_json_wo_version. The provider cannot compare write-only values.

Save your progress? Sign up for free

Drill 6: A Short-Lived Token Directly in the Provider

Instead of fetching the token beforehand with bao, the provider logs in itself with the pipeline JWT. The JWT comes from TERRAFORM_VAULT_AUTH_JWT; in the CI job, for example, from $OPENBAO_ID_TOKEN (GitLab) or the OIDC call (GitHub, Day 11).

python3 - <<'PY'
s = open("main.tf").read()
s = s.replace('provider "vault" {\n  skip_child_token = true\n}', '''provider "vault" {
  skip_child_token = true
  auth_login_jwt {
    mount = "jwt-ci"
    role  = "tofu-main"
  }
}''')
open("main.tf", "w").write(s)
PY
unset VAULT_TOKEN
export TERRAFORM_VAULT_AUTH_JWT=$(../mint-jwt.sh $INFRA)
tofu plan -no-color 2>&1 | grep -E 'No changes|Error'
export TERRAFORM_VAULT_AUTH_JWT=$(../mint-jwt.sh $INFRA ref=refs/heads/feature)
tofu plan -no-color 2>&1 | grep -A4 'Code: 400' | tr -s '\n'
unset TERRAFORM_VAULT_AUTH_JWT

Expected:

No changes. Your infrastructure matches the configuration.
Code: 400. Errors:
* error validating claims: claim "ref" does not match any associated bound
claim values

Plans on feature branches thus get a separate role with read permissions; apply only main. The 20 minutes of token_ttl are intentional: the provider warns in the debug log below 15 minutes, because long applies would otherwise get a 403 in the middle of the run.

Save your progress? Sign up for free

Checkpoint

cat > check.sh <<'EOF'
#!/usr/bin/env bash
cd ~/secrets-lab/tag13
git -C gitops grep -q -E 'lab-[a-z-]*-[0-9]+' && echo "✘ Secret-Wert in Git" \
  || echo "✔ GitOps-Repo ohne Secret-Werte"
grep -q 'argocd.argoproj.io/sync-wave: "-1"' gitops/shop/externalsecret.yaml \
  && echo "✔ ExternalSecret in Welle -1" || echo "✘ Sync-Wave fehlt"
grep -q lab-reg-pw tofu-eph/terraform.tfstate && echo "✘ Secret im ephemeral-State" \
  || echo "✔ ephemeral-State sauber"
grep -q lab-reg-pw tofu-data/terraform.tfstate \
  && echo "✔ Data-Source-State enthält Klartext (erwartet, Gegenbeispiel)"
grep -q 'skip_child_token = true' tofu-eph/main.tf && echo "✔ skip_child_token" \
  || echo "✘ skip_child_token fehlt"
EOF
bash check.sh

Expected:

✔ GitOps-Repo ohne Secret-Werte
✔ ExternalSecret in Welle -1
✔ ephemeral-State sauber
✔ Data-Source-State enthält Klartext (erwartet, Gegenbeispiel)
✔ skip_child_token

Challenge

The JWT roles themselves should be created as code. Create the role deploy-staging (repo acme/shop, branch staging, policy ci-deploy, 5 min) with OpenTofu and verify the login with your lab issuer.

Hint: The resource is called vault_jwt_auth_backend_role; bound_claims is a map of strings, TTLs are in seconds. For this admin run in the lab, you use root.

Solution

mkdir -p ../tofu-roles && cd ../tofu-roles
cat > main.tf <<'EOF'
terraform {
  required_providers {
    vault = {
      source  = "hashicorp/vault"
      version = "~> 5.12"
    }
  }
}

provider "vault" {
  skip_child_token = true
}

resource "vault_jwt_auth_backend_role" "deploy_staging" {
  backend         = "jwt-ci"
  role_name       = "deploy-staging"
  role_type       = "jwt"
  user_claim      = "sub"
  bound_audiences = ["openbao"]
  bound_claims = {
    repository = "acme/shop"
    ref        = "refs/heads/staging"
  }
  token_policies = ["ci-deploy"]
  token_ttl      = 300
}
EOF
tofu init -no-color >/dev/null
VAULT_TOKEN=root tofu apply -auto-approve -no-color | grep 'Apply complete'
BAO_TOKEN= bao write -field=token_policies auth/jwt-ci/login role=deploy-staging \
  jwt="$(../mint-jwt.sh ref=refs/heads/staging)"; echo

Expected:

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
[ci-deploy default]

Save your progress? Sign up for free

Enterprise Note

  • Encrypt the state backend (OpenTofu encryption block) and still keep secrets out of it: ephemeral plus write-only, forbid data sources for secrets (review, policy as code).
  • Separate roles: tofu-plan (read, all branches) and tofu-apply (write permissions, only main, protected environment). Never admin permissions on sys/ in a normal run.
  • The provider checks the server version via sys/seal-status; OpenBao reports 2.x. For feature checks against Vault versions, set skip_get_vault_version and vault_version_override deliberately if needed, and test.
  • Argo CD: ESO per tenant with its own SecretStore (Day 4); Argo projects allow only ExternalSecret, never a Secret with content (Kyverno, Day 14).
  • Test new provider versions against OpenBao in a test pipeline before the lockfile version goes up.