Skip to content
Hands-on IT IT Drills

Chapter 6: Secrets management, rotation and replacing credentials in the repository (Vault/OpenBao, External Secrets Operator)

In this chapter you will learn:

  • how to find checked-in credentials in a repository, including its history
  • how Vault (or OpenBao) stores, distributes, logs and revokes secrets
  • the difference between static and dynamic secrets with lease and TTL
  • how the External Secrets Operator syncs secrets from Vault into Kubernetes and follows rotations
  • a robust replacement plan for a leaked credential: rotate first, then clean up

What is this about?

Picture an office building. In many companies, the master key hangs on a hook next to the door, visible to everyone, the same one for years. Anyone who has been there once can have a copy made. Nobody knows how many copies exist.

A better building has a gatekeeper with a key cabinet. You show your badge, get a key for exactly the rooms you need, and have to return it after a fixed time. The gatekeeper writes every handout into a logbook. If someone loses a key, the gatekeeper changes the lock.

A password in a Git repository is the key on the hook. Vault or OpenBao is the gatekeeper with the key cabinet and the logbook. The External Secrets Operator (ESO) is the messenger who fetches keys from the cabinet and puts them in Kubernetes where the application expects them.

  Developer/CI                 Vault / OpenBao                 Kubernetes (kind)
 +-------------+   Login     +-------------------+   Login    +--------------------+
 | GitLab job  |-- JWT ----->| Auth methods      |<-- SA JWT--| External Secrets   |
 | (id_tokens) |             |  jwt, kubernetes, |            | Operator (ESO)     |
 +-------------+             |  approle          |            +---------+----------+
                             | Policies          |                      |
  Git repo                   | Secrets engines   |   reads kv/...       v
 +-------------+  gitleaks   |  kv v2, database  |----------->  K8s Secret
 | config.py   |--> finding! | Audit log         |              (refreshInterval)
 +-------------+             +---------+---------+                      |
                                       | creates/revokes                v
                                       v                          demo-app pod
                               +---------------+
                               | PostgreSQL    |  dynamic users with TTL
                               +---------------+

The concepts in plain terms

What is a secret?

A secret is any piece of information that lets someone impersonate somebody or decrypt something: passwords, API tokens, private keys, certificates, connection strings with a password, cloud access keys. Rule of thumb: if an attacker finds it in a public gist and you sleep badly afterwards, it is a secret.

The lifecycle of a secret

Secrets are not configuration you set once. They have a lifecycle (secret lifecycle):

Phase Question Example in the lab
Create Who creates it, how strong? Vault creates DB users
Store Where is it stored encrypted? KV v2 in Vault
Distribute How does it reach the application? ESO, Vault Agent, CI job
Use Who may read it? Policy demo-app-read
Rotate How often is it renewed? new KV value, new lease
Revoke How do I withdraw it immediately? vault lease revoke
Audit Who read it, and when? audit log

A password in the repository skips almost every phase: it is created by hand, stored in plain text, distributed to everyone via git clone, never rotated, and cannot be revoked, because nobody knows where all the copies are.

Static and dynamic secrets

A static secret exists independently of the vault. For example, you put the database password into Vault. Vault only keeps it. You have to rotate it yourself (or a script does).

A dynamic secret is only created on request. When the application asks for database credentials, Vault creates a new database user at that moment, hands it out, and deletes it again when it expires. Every consumer gets its own credentials. A leak only affects a short time span and can be traced to one consumer.

Lease and TTL

Every dynamic secret comes with a lease. It has an ID and a TTL (time to live). When the TTL runs out, Vault revokes the secret automatically. The application can renew the lease up to max_ttl. A human can revoke it early at any time, including all leases of a role at once (-prefix).

Vault tokens have a TTL too. A token that expires after 15 minutes is a much smaller problem than one that is valid forever.

Auth methods: how does a consumer prove its identity?

Vault hands out nothing unless the consumer logs in. The chicken-and-egg problem is: what do you log in with, without storing yet another password? Answer: with an identity that the platform issues anyway.

Method Who uses it? Proof of identity
kubernetes pods, ESO ServiceAccount token (JWT)
jwt / oidc GitLab CI jobs, humans signed JWT from the IdP
approle machines without a platform ID role ID + secret ID
token admin, emergency Vault token

Kubernetes auth: The pod sends its ServiceAccount token to Vault. Vault asks the Kubernetes API via TokenReview: “Is this genuine, and which ServiceAccount does it belong to?” If the answer matches a Vault role (namespace and ServiceAccount name), the pod gets a Vault token with that role’s policies.

JWT/OIDC for GitLab CI: On request, GitLab issues every job a signed JWT. You request it in .gitlab-ci.yml via id_tokens. The token contains claims such as project_path, ref, ref_protected and environment. Vault verifies the signature via GitLab’s JWKS endpoint and compares the claims with the role’s bound_claims. That way, only the job on the protected main branch of the right project gets the production secret. No stored Vault token in CI variables is needed.

vault auth enable jwt
vault write auth/jwt/config \
  oidc_discovery_url="https://gitlab.example.internal" \
  bound_issuer="https://gitlab.example.internal"
vault write auth/jwt/role/demo-app-ci - <<'EOF'
{
  "role_type": "jwt",
  "user_claim": "user_login",
  "bound_audiences": ["https://vault.example.internal"],
  "bound_claims": {
    "project_path": "root/demo-app",
    "ref": "main",
    "ref_protected": "true"
  },
  "token_policies": ["demo-app-read"],
  "token_explicit_max_ttl": 300
}
EOF

The matching job in GitLab:

read-secret:
  image: hashicorp/vault:1.18
  id_tokens:
    VAULT_ID_TOKEN:
      aud: https://vault.example.internal
  variables:
    VAULT_ADDR: https://vault.example.internal
  script:
    - LOGIN="auth/jwt/login role=demo-app-ci jwt=$VAULT_ID_TOKEN"
    - export VAULT_TOKEN="$(vault write -field=token $LOGIN)"
    - vault kv get -field=username kv/demo-app/db

Tip: With the secrets: keyword, GitLab Premium offers a built-in Vault integration that performs exactly this login for you. The mechanism underneath is the same.

AppRole is meant for machines without their own platform identity, for example a VM or an old Jenkins. The role ID is like a user name, the secret ID like a short-lived password. This only shifts the problem, though: the secret ID has to reach the machine securely (keyword “secret zero”). So use AppRole only when Kubernetes or JWT auth will not work.

vault auth enable approle
vault write auth/approle/role/legacy-job \
  token_policies=demo-app-read secret_id_ttl=10m secret_id_num_uses=1
vault read -field=role_id auth/approle/role/legacy-job/role-id
vault write -f -field=secret_id auth/approle/role/legacy-job/secret-id

Policies

A policy is a list of paths with permitted capabilities (read, create, update, delete, list, deny). Everything that is not allowed is forbidden (default deny). Policies are written in HCL.

Remember: With KV version 2, the data lives under <mount>/data/... and the metadata under <mount>/metadata/.... The CLI hides this (vault kv get kv/demo-app/db), but the policy does not. A policy on kv/demo-app/* therefore matches nothing. The correct path is kv/data/demo-app/*.

Audit log

Vault can write every request and every response to an audit device (file, syslog, socket). Secret values do not appear there in plain text, but as an HMAC hash. That way you can later prove who read which path when, without the log itself becoming a secret store.

Warning: If an audit device is enabled but not writable (disk full), Vault refuses requests. That is intentional: better no access than unlogged access. For production this means: at least two audit devices, and monitor disk space.

Vault or OpenBao?

In August 2023, starting with version 1.15, HashiCorp switched Vault from the open source license MPL 2.0 to the Business Source License (BSL 1.1). The BSL allows your own use, but forbids building a competing product on it. For many public authorities and companies with strict licensing rules, this is a checkpoint. In 2025, IBM acquired HashiCorp.

In response, OpenBao was created, a fork of the last MPL version under the umbrella of the Linux Foundation. OpenBao is API-compatible; the CLI is called bao, the container image openbao/openbao. Most commands in this chapter work if you replace vault with bao. Environment variables there are called BAO_ADDR and BAO_TOKEN. Features of the Vault Enterprise edition (such as performance replication or namespaces in their old form) are missing or implemented differently. ESO talks to OpenBao via the same vault provider.

Feature Vault OpenBao
License BSL 1.1 (from 1.15) MPL 2.0
Steward HashiCorp / IBM Linux Foundation
CLI / image vault, hashicorp/vault bao, openbao/openbao
Support commercially available community, service providers

External Secrets Operator (ESO)

ESO is a Kubernetes operator. It reads secrets from an external vault and creates them as normal Kubernetes Secrets. The application does not notice Vault at all. Three objects matter:

Object Scope Job
SecretStore one namespace How do I reach the vault? (URL, auth)
ClusterSecretStore whole cluster the same, usable from all namespaces
ExternalSecret one namespace Which values, into which K8s Secret?

The refreshInterval field in the ExternalSecret defines how often ESO checks the vault. If a value changes there, ESO updates the Kubernetes Secret. That way a rotation in Vault automatically becomes visible in the cluster.

Remember: A ClusterSecretStore is convenient but powerful: any namespace can reference it. In multi-tenant clusters, a SecretStore per namespace with its own Vault role is the safer choice.

Hands-on Lab: From a leak in the repo to a rotating secret in the cluster

You work in ~/seclab/kap06. Prerequisite: the kind cluster seclab from Chapter 0 is running, and the tools gitleaks, vault, helm, kubectl and jq are installed.

Step 1: Preparation

mkdir -p ~/seclab/kap06 && cd ~/seclab/kap06
kubectl config use-context kind-seclab
docker network ls --filter name=^kind$
gitleaks version
vault version

Expected: the kind network is listed, gitleaks reports a version 8.x, vault a version 1.x. Note the gitleaks version. From 8.19 on there are the git and dir subcommands; older versions only know detect and protect.

Step 2: Create a demo repo with a leak

You simulate a typical mistake: someone checks in credentials and “fixes” it later with a second commit.

mkdir leak-demo && cd leak-demo
git init -b main
cat > config.py <<'EOF'
# Lab values, deliberately fake and invalid!
DB_HOST = "postgres"
DB_PASSWORD = "Lab-9xQ2mZ7vK4pR8tW1"
GITLAB_TOKEN = "glpat-LabFake0000Abcd1234X"
EOF
git add config.py && git commit -m "feat: DB-Konfiguration"
cat > config.py <<'EOF'
import os
DB_HOST = os.environ.get("DB_HOST", "postgres")
DB_PASSWORD = os.environ["DB_PASSWORD"]
EOF
git commit -am "fix: Secrets aus Code entfernt"
git log --oneline

Expected: two commits. The current config.py no longer contains a secret.

Security: All values here are made-up lab values. Never use real tokens in exercises, not even “just briefly”.

Step 3: gitleaks on the working copy and history

First only the files as they are on disk right now:

# gitleaks >= 8.19:
gitleaks dir . -v
# older versions:
gitleaks detect --source . --no-git -v

Expected: no leaks found. The working copy is clean. Now the entire Git history:

# gitleaks >= 8.19:
gitleaks git . -v
# older versions (still present in new versions, but deprecated):
gitleaks detect --source . -v
echo "Exit-Code: $?"

Expected output (shortened):

Finding:     GITLAB_TOKEN = "glpat-LabFake0000Abcd1234X"
Secret:      glpat-LabFake0000Abcd1234X
RuleID:      gitlab-pat
File:        config.py
Line:        4
Commit:      3f1c2a9...
Fingerprint: 3f1c2a9...:config.py:gitlab-pat:4

WRN leaks found: 1
Exit-Code: 1

Depending on the gitleaks version, the generic-api-key rule additionally reports the DB_PASSWORD. What matters: the finding is in the first commit. The fix commit only removed the secret from the current version, not from the history. Anyone who clones the repo has it.

For reports and automation, use JSON and --redact, so the value does not end up in the CI log:

gitleaks git . --redact --report-format json --report-path gitleaks.json
jq -r '.[] | "\(.RuleID)  \(.File):\(.StartLine)  \(.Commit[0:7])"' gitleaks.json

Exit code 1 on findings is important: it makes a CI job fail.

Step 4: Start the Vault dev server

cd ~/seclab/kap06
docker run -d --name vault --network kind -p 8200:8200 \
  --cap-add=IPC_LOCK \
  -e VAULT_DEV_ROOT_TOKEN_ID=root \
  -e VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200 \
  hashicorp/vault:1.18
export VAULT_ADDR=http://127.0.0.1:8200
export VAULT_TOKEN=root
vault status

Expected: Sealed false, Storage Type inmem. The container is attached to the kind network. That way, the Postgres container and the kind cluster can later reach it under the name vault.

Warning: Dev mode keeps everything in memory, is unsealed automatically, speaks unencrypted HTTP and has the root token root. A restart deletes all data. It is meant for the lab only.

OpenBao equivalent:

docker run -d --name vault --network kind -p 8200:8200 openbao/openbao \
  server -dev -dev-root-token-id=root -dev-listen-address=0.0.0.0:8200
export BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN=root
bao status

Step 5: Create KV v2, write and read a secret

The dev server already comes with a KV engine under secret/. You create your own, just as you would in production:

vault secrets enable -path=kv -version=2 kv
vault kv put kv/demo-app/db username=demo password='Lab-Start-1'
vault kv get kv/demo-app/db

Expected output (shortened):

==== Secret Path ====
kv/data/demo-app/db

======= Metadata =======
Key                Value
---                -----
created_time       2026-10-08T09:12:44.1Z
version            1

====== Data ======
Key         Value
---         -----
password    Lab-Start-1
username    demo

Note kv/data/demo-app/db in the secret path: that is the real API path. KV v2 stores versions:

vault kv put kv/demo-app/db username=demo password='Lab-Start-2'
vault kv get -version=1 -field=password kv/demo-app/db
vault kv metadata get kv/demo-app/db | grep -E 'current_version|max_versions'

Expected: version 1 returns Lab-Start-1, current_version is 2. Versions are useful for rolling back a failed rotation (vault kv rollback -version=1 kv/demo-app/db).

Step 6: Policy, restricted token and audit log

First enable the audit log, so that the following accesses are logged:

vault audit enable file file_path=/vault/logs/audit.log

Expected: Success! Enabled the file audit device at: file/.

Now a policy that may only read:

cat > demo-app-read.hcl <<'EOF'
path "kv/data/demo-app/*" {
  capabilities = ["read"]
}
EOF
vault policy write demo-app-read demo-app-read.hcl
APP_TOKEN=$(vault token create -policy=demo-app-read -ttl=15m -field=token)
VAULT_TOKEN=$APP_TOKEN vault kv get -field=username kv/demo-app/db
VAULT_TOKEN=$APP_TOKEN vault kv put kv/demo-app/db password=hack

Expected: reading returns demo. Writing fails with Code: 403 and permission denied. Look at the last entry in the audit log:

docker exec vault tail -n 1 /vault/logs/audit.log | jq '{
  type, path: .request.path, op: .request.operation,
  policies: .auth.policies, error }'

Expected: type is response, path is kv/data/demo-app/db, policies contains demo-app-read and error the text permission denied. The token itself appears there only as an HMAC hash.

Step 7: Postgres and the database secrets engine

Start a Postgres database in the same Docker network:

docker run -d --name postgres --network kind -p 5432:5432 \
  -e POSTGRES_PASSWORD=lab-postgres-pw -e POSTGRES_DB=demo postgres:16
sleep 5
docker exec postgres pg_isready -U postgres

Expected: accepting connections. Now Vault gets the admin credentials and a role that describes how a short-lived user is created:

vault secrets enable database
vault write database/config/demo-postgres \
  plugin_name=postgresql-database-plugin \
  connection_url="postgresql://{{username}}:{{password}}@\
postgres:5432/demo?sslmode=disable" \
  allowed_roles="demo-app-ro" \
  username="postgres" password="lab-postgres-pw"

vault write database/roles/demo-app-ro \
  db_name=demo-postgres \
  default_ttl=5m max_ttl=1h \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' \
VALID UNTIL '{{expiration}}'; \
GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";"

Expected: Success! Data written to: ... each time. Vault reaches the database at postgres:5432 because both containers are attached to the kind network and Docker resolves container names there.

Tip: At this point, everyone who reads this book still knows the password lab-postgres-pw. With vault write -f database/rotate-root/demo-postgres, Vault sets a new admin password that only Vault knows. In production, this is part of the setup. In the lab you leave it out, so you can still log in yourself.

Step 8: Use and revoke dynamic credentials

vault read -format=json database/creds/demo-app-ro > creds.json
jq '{lease_id, lease_duration, user: .data.username}' creds.json
LEASE=$(jq -r .lease_id creds.json)
DBUSER=$(jq -r .data.username creds.json)
DBPASS=$(jq -r .data.password creds.json)

Expected output:

{
  "lease_id": "database/creds/demo-app-ro/Xy7kP...",
  "lease_duration": 300,
  "user": "v-root-demo-app-ro-AbC1d2EfG3h4IjK5lM6n-1791451234"
}

300 seconds are the 5 minutes from default_ttl. Test the login from a second container, so that a password check really happens over the network:

docker run --rm --network kind -e PGPASSWORD="$DBPASS" postgres:16 \
  psql -h postgres -U "$DBUSER" -d demo -c 'select current_user;'
docker exec postgres psql -U postgres -c '\du' | grep v-root

Expected: current_user shows the v-root-... name, and \du lists this role. Now you revoke the lease as if the credential had leaked:

vault lease lookup "$LEASE" | grep ttl
vault lease revoke "$LEASE"
docker run --rm --network kind -e PGPASSWORD="$DBPASS" postgres:16 \
  psql -h postgres -U "$DBUSER" -d demo -c 'select 1;'

Expected: after the revoke, the login fails (password authentication failed or role ... does not exist). Vault has deleted the user in Postgres. Had you done nothing, the same would have happened automatically after 5 minutes. You revoke all leases of a role with vault lease revoke -prefix database/creds/demo-app-ro.

Step 9: Install the External Secrets Operator via Helm

helm repo add external-secrets https://charts.external-secrets.io
helm repo update
helm install external-secrets external-secrets/external-secrets \
  -n external-secrets --create-namespace --wait
kubectl get pods -n external-secrets
kubectl api-resources --api-group=external-secrets.io | grep -i secretstore

Expected: three pods Running (controller, webhook, cert controller). The last line shows the API version, for example external-secrets.io/v1. Older ESO versions (before 0.16) only know v1beta1. In that case, adjust apiVersion in the YAML files below.

Step 10: Configure Kubernetes auth in Vault

Here lies the networking peculiarity of this lab. Vault does not run in the cluster, but as a container next to it. Both directions have to work:

 Docker network "kind"
 +-------------------------------------------------------------------+
 |  Container "vault"  --- TokenReview ---> seclab-control-plane:6443 |
 |        ^                                  (K8s API in kind node)   |
 |        |  http://vault:8200                                        |
 |  ESO pod in kind node (pod DNS -> node -> Docker DNS)              |
 +-------------------------------------------------------------------+
  1. ESO in the cluster must reach Vault: via the name vault, which Docker resolves in the kind network.
  2. Vault must reach the Kubernetes API to verify ServiceAccount tokens: via the name of the control plane container seclab-control-plane on port 6443. The 127.0.0.1:... entry from your kubeconfig does not work from inside the Vault container.

First check the cluster → Vault direction:

kubectl create namespace demo --dry-run=client -o yaml | kubectl apply -f -
kubectl run nettest -n demo --rm -i --restart=Never \
  --image=curlimages/curl -- curl -s http://vault:8200/v1/sys/health

Expected: JSON with "initialized":true,"sealed":false. kind itself shows you the internal API address:

kind get kubeconfig --name seclab --internal | grep server

Expected: server: https://seclab-control-plane:6443. Now you create the ServiceAccount that ESO uses to log in to Vault. It may additionally perform TokenReviews (system:auth-delegator), so that Vault can use its token for verification:

kubectl create serviceaccount eso-vault -n demo
kubectl create clusterrolebinding eso-vault-tokenreview \
  --clusterrole=system:auth-delegator --serviceaccount=demo:eso-vault

kubectl config view --raw --minify --flatten \
  -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' \
  | base64 --decode > k8s-ca.crt

vault auth enable kubernetes
vault write auth/kubernetes/config \
  kubernetes_host="https://seclab-control-plane:6443" \
  [email protected]
vault write auth/kubernetes/role/demo-app \
  bound_service_account_names=eso-vault \
  bound_service_account_namespaces=demo \
  token_policies=demo-app-read token_ttl=15m

Because you did not set a token_reviewer_jwt and Vault runs outside the cluster, Vault uses the token of the ServiceAccount that is logging in for the TokenReview itself. That is why the ClusterRoleBinding is needed. Test the login by hand before ESO comes into play:

SA_JWT=$(kubectl create token eso-vault -n demo)
vault write auth/kubernetes/login role=demo-app jwt="$SA_JWT" | grep policies

Expected: token_policies ["default" "demo-app-read"]. If this does not work, the error lies in Vault or the network, not in ESO. That saves a lot of searching.

Step 11: SecretStore and ExternalSecret

cat > secretstore.yaml <<'EOF'
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: demo
spec:
  provider:
    vault:
      server: "http://vault:8200"
      path: "kv"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "demo-app"
          serviceAccountRef:
            name: "eso-vault"
EOF
cat > externalsecret.yaml <<'EOF'
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: demo-app-db
  namespace: demo
spec:
  refreshInterval: 15s
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: demo-app-db
    creationPolicy: Owner
  data:
    - secretKey: DB_USER
      remoteRef:
        key: demo-app/db
        property: username
    - secretKey: DB_PASSWORD
      remoteRef:
        key: demo-app/db
        property: password
EOF
kubectl apply -f secretstore.yaml -f externalsecret.yaml
kubectl get secretstore,externalsecret -n demo

Expected output:

NAME                                          AGE  STATUS  CAPABILITIES  READY
secretstore.external-secrets.io/vault-backend 10s  Valid   ReadWrite     True

NAME                                           STORE          STATUS        READY
externalsecret.external-secrets.io/demo-app-db vault-backend  SecretSynced  True

(The columns are shortened here; depending on the ESO version you also see REFRESH.)

Note: the SecretStore says path: "kv" and the ExternalSecret says key: demo-app/db, without data/. ESO adds that itself, because version: "v2" is set. Check the content:

kubectl get secret demo-app-db -n demo \
  -o jsonpath='{.data.DB_PASSWORD}' | base64 --decode; echo

Expected: Lab-Start-2.

Remember: refreshInterval: 15s is this short only for the demonstration. Every sync is a Vault login plus a read. In production, 5 to 60 minutes is common.

Step 12: Demonstrate rotation

Change the value in Vault, the way a rotation process would:

vault kv put kv/demo-app/db username=demo password='Lab-Rotiert-3'
sleep 20
kubectl get secret demo-app-db -n demo \
  -o jsonpath='{.data.DB_PASSWORD}' | base64 --decode; echo
kubectl get externalsecret demo-app-db -n demo \
  -o jsonpath='{.status.refreshTime}'; echo

Expected: Lab-Rotiert-3 and a fresh timestamp. If you do not want to wait, force a sync:

kubectl annotate externalsecret demo-app-db -n demo \
  force-sync="$(date +%s)" --overwrite

Warning: The Kubernetes Secret is now current, but the running pod is not necessarily. Environment variables are read only at pod start. Secrets mounted as files are updated by the kubelet after a while (not with subPath). The application then has to reread the file. Common approaches are a rollout restart (kubectl rollout restart) or a tool like Stakater Reloader, which restarts automatically when a Secret changes.

Check in the audit log that ESO read with its own identity:

docker exec vault cat /vault/logs/audit.log \
  | jq -r 'select(.type=="response" and .auth.metadata.role=="demo-app")
           | .request.path' | sort | uniq -c

Expected: lines for auth/kubernetes/login and kv/data/demo-app/db. The counts grow with every refresh. The role appears in the metadata in plain text, the secret values do not.

Step 13: Replacement plan for the leaked credential

Back to the leak from Step 2. The order is decisive:

 1. Contain     -> revoke/rotate the credential     (minutes)
 2. Assess      -> Was it used? Check the logs       (hours)
 3. Switch over -> code reads from secret source     (days)
 4. Clean up    -> rewrite history (optional)        (planned)
 5. Prevent     -> pre-commit + CI scan              (permanent)

1. Contain. You revoke the GitLab token in GitLab under User Settings → Access Tokens → Revoke. You replace the DB password with a new value in Vault. You already practiced that in Step 12. Even better: dynamic credentials as in Step 8, then there is no long-lived password at all.

2. Assess. Check the logs of the affected system (GitLab audit events, DB logins) to see whether the credential was used between the commit and the revocation. That determines whether it becomes a security incident with a reporting obligation.

3. Switch over. The code now reads os.environ["DB_PASSWORD"]. The Kubernetes Secret demo-app-db from ESO provides exactly this variable. In the demo-app Deployment, it looks like this:

          envFrom:
            - secretRef:
                name: demo-app-db

4. Clean up. Only now, once the old credential is worthless, do you rewrite the history.

Warning: Cleaning up history is no substitute for rotation. The secret already sits in clones on laptops, in CI caches, in forks, in merge request diffs and possibly in backups. You cannot get it back. A rotated secret is harmless, even if it still appears somewhere. A cleaned-up but unrotated secret is still valid.

Step 14: Clean up history with git filter-repo

git filter-repo is a standalone tool (install it, e.g., via pip install git-filter-repo or your package manager). For safety reasons, it only works on a fresh clone:

cd ~/seclab/kap06
git clone --no-local leak-demo leak-demo-clean
cd leak-demo-clean
cat > ../replacements.txt <<'EOF'
glpat-LabFake0000Abcd1234X==>***ENTFERNT***
Lab-9xQ2mZ7vK4pR8tW1==>***ENTFERNT***
EOF
git log --oneline
git filter-repo --replace-text ../replacements.txt
git log --oneline
gitleaks git . -v

Expected: the commit IDs before and after filter-repo differ, because every affected commit was rewritten. gitleaks reports no leaks found. With git show HEAD~1:config.py you see ***ENTFERNT*** (“removed”) in place of the values.

In a real project, further steps follow:

  • filter-repo removes the origin remote. You add it again and push with git push --force --all and git push --force --tags. For this, branch protection has to be relaxed briefly.
  • Everyone involved has to clone again. A git pull on an old clone brings the old history back.
  • GitLab still keeps old objects in merge request refs and caches. Housekeeping in the project (Settings → General → Advanced → Housekeeping) and, if necessary, removing affected MRs are part of it.

Step 15: Pre-commit hook with gitleaks

So that it does not happen again, a hook checks every commit before it is saved:

cd ~/seclab/kap06/leak-demo-clean
cat > .git/hooks/pre-commit <<'EOF'
#!/bin/sh
# gitleaks >= 8.19; older versions: gitleaks protect --staged --redact -v
exec gitleaks git --pre-commit --staged --redact -v
EOF
chmod +x .git/hooks/pre-commit
echo 'TOKEN = "glpat-LabFake9999Wxyz5678Q"' > neu.py
git add neu.py
git commit -m "test: sollte blockiert werden"
echo "Exit-Code: $?"

Expected: leaks found: 1, exit code 1, no new commit (git log --oneline does not show it). Remove the test file: git reset neu.py && rm neu.py.

In a team, you distribute the hook via the pre-commit framework with a .pre-commit-config.yaml in the repo:

repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.21.2
    hooks:
      - id: gitleaks

Local hooks can be bypassed with git commit --no-verify. That is why the same scan also belongs in the pipeline. Nobody can skip it there:

secret-scan:
  stage: test
  image:
    name: zricethezav/gitleaks:v8.21.2
    entrypoint: [""]
  variables:
    GIT_DEPTH: "0"
  script:
    - gitleaks git . --redact -v --report-format json --report-path gitleaks.json
  artifacts:
    when: always
    paths: [gitleaks.json]

GIT_DEPTH: "0" fetches the full history. With the default shallow clone, gitleaks would only see the last commits. In an isolated environment, you pull the image from your mirror registry (see Chapter 3).

Step 16: Clean up

kubectl delete -f ~/seclab/kap06/externalsecret.yaml -f ~/seclab/kap06/secretstore.yaml
kubectl delete clusterrolebinding eso-vault-tokenreview
kubectl delete serviceaccount eso-vault -n demo
helm uninstall external-secrets -n external-secrets
kubectl delete namespace external-secrets
docker rm -f vault postgres

Expected: all commands run without errors. docker ps shows neither vault nor postgres. Leave the demo namespace in place; later chapters need it.

Save your progress? Sign up for free

Common pitfalls

Symptom Cause Fix
permission denied despite a policy Policy on kv/demo-app/* instead of kv/data/... with KV v2, data/ in the policy path
gitleaks finds nothing in CI Shallow clone Set GIT_DEPTH: "0"
SecretStore InvalidProviderConfig Vault name cannot be resolved Check the kind network; as a last resort, IP via docker inspect
Login: permission denied with K8s auth SA or namespace does not match the role Check bound_service_account_*
Login: x509 or connection refused kubernetes_host points to 127.0.0.1 https://seclab-control-plane:6443
TokenReview 403 SA lacks system:auth-delegator Create the ClusterRoleBinding
Secret updated, app uses the old value Env variables only at start Rollout restart or Reloader
Everything gone after a Vault restart Dev mode stores in RAM Repeat the lab steps from 4 on
no matches for kind ExternalSecret Wrong apiVersion Check kubectl api-resources

As a stopgap for name resolution, you determine the IP like this: docker inspect -f '{{.NetworkSettings.Networks.kind.IPAddress}}' vault. However, this IP changes when the container restarts.

Exercise: A second team, separate vault compartments

A second team runs an application in the team-b namespace. It should get its own secret under kv/team-b/api with the key apikey (lab value Lab-TeamB-1). Set up Vault and ESO so that:

  1. a Kubernetes Secret team-b-api with the value is created in the team-b namespace,
  2. the SecretStore in the demo namespace can not read this secret (prove it!),
  3. the team B secret is synced at most every 5 minutes.

Restart Vault and ESO first if you have already cleaned up (Steps 4, 5, 6, 9, 10, 11).

Hint 1

You need three things per team: a policy and a role in the Kubernetes auth method in Vault, and a ServiceAccount in the cluster. The Vault role connects ServiceAccount and namespace with the policy. Remember the ClusterRoleBinding for TokenReview.

Hint 2

The easiest way to prove point 2 is a second ExternalSecret in the demo namespace that tries to read key: team-b/api from the vault-backend store. Its status should not be SecretSynced. kubectl describe externalsecret shows you Vault’s error message.

Solution

vault kv put kv/team-b/api apikey='Lab-TeamB-1'
cat > team-b-read.hcl <<'EOF'
path "kv/data/team-b/*" {
  capabilities = ["read"]
}
EOF
vault policy write team-b-read team-b-read.hcl

kubectl create namespace team-b
kubectl create serviceaccount eso-vault -n team-b
kubectl create clusterrolebinding eso-vault-tokenreview-team-b \
  --clusterrole=system:auth-delegator --serviceaccount=team-b:eso-vault
vault write auth/kubernetes/role/team-b \
  bound_service_account_names=eso-vault \
  bound_service_account_namespaces=team-b \
  token_policies=team-b-read token_ttl=15m

For the Kubernetes objects, copy secretstore.yaml and externalsecret.yaml and change namespace: team-b, role: "team-b", refreshInterval: 5m, target.name: team-b-api and the data block:

  data:
    - secretKey: APIKEY
      remoteRef:
        key: team-b/api
        property: apikey

After kubectl apply, kubectl get externalsecret -n team-b shows the status SecretSynced. For the negative test, create an ExternalSecret klau-versuch (“theft attempt”) with key: team-b/api in the demo namespace. Expected: READY False, and kubectl describe externalsecret klau-versuch -n demo shows an error with permission denied or 403. The reason: the store in demo logs in with the demo-app role, and its policy only allows kv/data/demo-app/*.

So the separation does not come from ESO, but from the combination of Vault role (who am I?) and policy (what may I do?). A shared ClusterSecretStore with a single Vault role would have removed this separation.

Save your progress? Sign up for free

In practice

Running Vault/OpenBao. In production, Vault does not run in dev mode, but as a cluster of three or five nodes with integrated Raft storage, TLS on all connections and auto-unseal via an HSM or a KMS. In isolated (air-gapped) environments, the HSM is often the only option, because no cloud KMS is reachable. The root token is revoked after setup (vault token revoke) and only regenerated in an emergency via the holders of the unseal or recovery keys (four-eyes principle with Shamir key shares).

Who manages Vault itself? Policies, roles and auth methods belong in Git and are rolled out via Terraform/OpenTofu or a dedicated pipeline job. Then permission changes can be reviewed via merge requests. No human clicks policies together in the UI.

Audit and monitoring. The audit log goes to the central SIEM. Sensible alerts: root token used, policy changed, many permission denied from one identity, audit device failed. Vault exports metrics in Prometheus format (/v1/sys/metrics). Worth watching there: number of leases, seal status and request latency. Many leases without revokes point to applications that pull new credentials on every start and never clean up.

Public authorities and KRITIS. Auditors at public authorities and operators of critical infrastructure (KRITIS) usually ask: Where are the secrets? Who has access? How often are they rotated? How quickly can they be revoked? Is there proof? With Vault you answer this from the policies, the TTLs and the audit log. In the IT-Grundschutz of the BSI (the German Federal Office for Information Security’s baseline protection catalog), you will find matching requirements, for example in the modules on crypto concept and key management and on identity and access management. Clarify licensing questions (BSL vs. MPL) early with procurement. Some clients explicitly require open source licenses; then OpenBao is the given choice.

Replacement in an existing estate. In organizations that have grown over time, a first gitleaks run over all repos often finds hundreds of findings. Do not tackle them all at once. What has proven itself:

  1. Classify findings by type (cloud keys, DB passwords, test dummies).
  2. Rotate real, still valid credentials immediately.
  3. Create a baseline for legacy findings (--baseline-path in gitleaks), so that new findings stand out immediately.
  4. Agree on transition periods with the teams for switching to Vault (see Chapter 10).

No secret in Git, not even encrypted? Tools like SOPS or Sealed Secrets put encrypted secrets into the repo. That is better than plain text, but does not solve rotation, revocation and audit. ESO with Vault cleanly separates storage and distribution and is therefore often the preferred solution in regulated environments. Argo CD or Flux (see Chapter 1) then only roll out the ExternalSecret, never the value itself.

Interview questions

Q: A colleague pushed a production database password to an internal repo and deleted it again ten minutes later. What do you do?

A: First rotate the password or revoke access, because it is still in the history and in every clone. Then check the DB logs to see whether it was misused, and depending on the result, start the incident process. Then switch the code to a secret source, clean up the history with git filter-repo, and introduce gitleaks as a pre-commit hook and CI job.

Q: What is the difference between static and dynamic secrets?

A: A static secret exists independently of the vault and is only stored there. You have to rotate it yourself. A dynamic secret is created by Vault only on request, individually per consumer and with a lease. When the TTL expires or on revoke, Vault deletes it in the target system again.

Q: How does a GitLab CI job authenticate to Vault without a token stored in CI variables?

A: Via id_tokens, the job requests a JWT signed by GitLab with the matching audience. Vault verifies it with the JWT auth method against GitLab’s JWKS endpoint and compares claims such as project_path, ref and ref_protected with the role’s bound_claims. The job gets a short-lived Vault token with exactly that role’s policy.

Q: Why does Vault need access to the Kubernetes API when it uses Kubernetes auth?

A: Vault verifies the pod’s ServiceAccount token via the TokenReview API. That is the only way it knows whether the token is valid and which ServiceAccount and namespace it belongs to. If Vault runs outside the cluster, it needs network access to the API, the CA certificate and a JWT with permission for TokenReview.

Q: What is the difference between SecretStore and ClusterSecretStore in the External Secrets Operator?

A: A SecretStore applies to one namespace and can only be used by ExternalSecrets there. A ClusterSecretStore applies cluster-wide. For tenant separation, you use SecretStores per namespace with their own Vault role and policy; otherwise every namespace could read everything the shared role is allowed to.

Q: How does a rotation in Vault reach the running application?

A: ESO checks at every refreshInterval and updates the Kubernetes Secret. The kubelet then updates mounted secret files, and the application has to reread them. Environment variables stay stale until a restart. That is why you need a rollout restart or a reloader. For databases, you let old and new credentials be valid in parallel for a short time.

Q: Vault or OpenBao, how do you decide?

A: Technically, both are largely the same for the core functions and API-compatible. What decides is the license (BSL for Vault from 1.15, MPL 2.0 for OpenBao), the need for enterprise features, support requirements and procurement rules. In environments with an open source mandate, OpenBao is the obvious choice.

Q: Why is a policy on kv/myapp/* ineffective with KV v2?

A: KV v2 stores the data under kv/data/myapp/... and the metadata under kv/metadata/myapp/.... The CLI hides this, but policies work with the real API paths.

Summary

Here is what you learned:

  • A deleted secret is not gone: gitleaks finds it in the history, and only rotation makes it worthless.
  • A secret has a lifecycle. Vault covers storing, distributing, rotating, revoking and auditing.
  • Dynamic secrets with lease and TTL are unique per consumer, expire on their own and can be revoked selectively.
  • Consumers log in with platform identities: Kubernetes ServiceAccounts, GitLab id_tokens via JWT, AppRole as a last resort.
  • Policies work with real API paths and default deny. The audit log proves every access.
  • ESO syncs secrets from Vault into Kubernetes. With refreshInterval, rotations arrive automatically, but running pods need a restart or reload.
  • If Vault runs outside the kind cluster, the Docker network kind connects both sides via the names vault and seclab-control-plane.
  • OpenBao is the MPL-licensed fork of Vault with the bao CLI and a compatible API.