Chapter 6: Secrets management, rotation and replacing credentials in the repository (Vault/OpenBao, External Secrets Operator)
In this chapter you will learn:
- how to find checked-in credentials in a repository, including its history
- how Vault (or OpenBao) stores, distributes, logs and revokes secrets
- the difference between static and dynamic secrets with lease and TTL
- how the External Secrets Operator syncs secrets from Vault into Kubernetes and follows rotations
- a robust replacement plan for a leaked credential: rotate first, then clean up
What is this about?
Picture an office building. In many companies, the master key hangs on a hook next to the door, visible to everyone, the same one for years. Anyone who has been there once can have a copy made. Nobody knows how many copies exist.
A better building has a gatekeeper with a key cabinet. You show your badge, get a key for exactly the rooms you need, and have to return it after a fixed time. The gatekeeper writes every handout into a logbook. If someone loses a key, the gatekeeper changes the lock.
A password in a Git repository is the key on the hook. Vault or OpenBao is the gatekeeper with the key cabinet and the logbook. The External Secrets Operator (ESO) is the messenger who fetches keys from the cabinet and puts them in Kubernetes where the application expects them.
Developer/CI Vault / OpenBao Kubernetes (kind)
+-------------+ Login +-------------------+ Login +--------------------+
| GitLab job |-- JWT ----->| Auth methods |<-- SA JWT--| External Secrets |
| (id_tokens) | | jwt, kubernetes, | | Operator (ESO) |
+-------------+ | approle | +---------+----------+
| Policies | |
Git repo | Secrets engines | reads kv/... v
+-------------+ gitleaks | kv v2, database |-----------> K8s Secret
| config.py |--> finding! | Audit log | (refreshInterval)
+-------------+ +---------+---------+ |
| creates/revokes v
v demo-app pod
+---------------+
| PostgreSQL | dynamic users with TTL
+---------------+
The concepts in plain terms
What is a secret?
A secret is any piece of information that lets someone impersonate somebody or decrypt something: passwords, API tokens, private keys, certificates, connection strings with a password, cloud access keys. Rule of thumb: if an attacker finds it in a public gist and you sleep badly afterwards, it is a secret.
The lifecycle of a secret
Secrets are not configuration you set once. They have a lifecycle (secret lifecycle):
| Phase | Question | Example in the lab |
|---|---|---|
| Create | Who creates it, how strong? | Vault creates DB users |
| Store | Where is it stored encrypted? | KV v2 in Vault |
| Distribute | How does it reach the application? | ESO, Vault Agent, CI job |
| Use | Who may read it? | Policy demo-app-read |
| Rotate | How often is it renewed? | new KV value, new lease |
| Revoke | How do I withdraw it immediately? | vault lease revoke |
| Audit | Who read it, and when? | audit log |
A password in the repository skips almost every phase: it is created
by hand, stored in plain text, distributed to everyone via
git clone, never rotated, and cannot be revoked, because
nobody knows where all the copies are.
Static and dynamic secrets
A static secret exists independently of the vault. For example, you put the database password into Vault. Vault only keeps it. You have to rotate it yourself (or a script does).
A dynamic secret is only created on request. When the application asks for database credentials, Vault creates a new database user at that moment, hands it out, and deletes it again when it expires. Every consumer gets its own credentials. A leak only affects a short time span and can be traced to one consumer.
Lease and TTL
Every dynamic secret comes with a lease. It has an
ID and a TTL (time to live). When the TTL runs out,
Vault revokes the secret automatically. The application can renew the
lease up to max_ttl. A human can revoke it early at any
time, including all leases of a role at once (-prefix).
Vault tokens have a TTL too. A token that expires after 15 minutes is a much smaller problem than one that is valid forever.
Auth methods: how does a consumer prove its identity?
Vault hands out nothing unless the consumer logs in. The chicken-and-egg problem is: what do you log in with, without storing yet another password? Answer: with an identity that the platform issues anyway.
| Method | Who uses it? | Proof of identity |
|---|---|---|
kubernetes |
pods, ESO | ServiceAccount token (JWT) |
jwt / oidc |
GitLab CI jobs, humans | signed JWT from the IdP |
approle |
machines without a platform ID | role ID + secret ID |
token |
admin, emergency | Vault token |
Kubernetes auth: The pod sends its ServiceAccount token to Vault. Vault asks the Kubernetes API via TokenReview: “Is this genuine, and which ServiceAccount does it belong to?” If the answer matches a Vault role (namespace and ServiceAccount name), the pod gets a Vault token with that role’s policies.
JWT/OIDC for GitLab CI: On request, GitLab issues
every job a signed JWT. You request it in .gitlab-ci.yml
via id_tokens. The token contains claims such as
project_path, ref, ref_protected
and environment. Vault verifies the signature via GitLab’s
JWKS endpoint and compares the claims with the role’s
bound_claims. That way, only the job on the protected
main branch of the right project gets the production
secret. No stored Vault token in CI variables is needed.
vault auth enable jwt
vault write auth/jwt/config \
oidc_discovery_url="https://gitlab.example.internal" \
bound_issuer="https://gitlab.example.internal"
vault write auth/jwt/role/demo-app-ci - <<'EOF'
{
"role_type": "jwt",
"user_claim": "user_login",
"bound_audiences": ["https://vault.example.internal"],
"bound_claims": {
"project_path": "root/demo-app",
"ref": "main",
"ref_protected": "true"
},
"token_policies": ["demo-app-read"],
"token_explicit_max_ttl": 300
}
EOFThe matching job in GitLab:
read-secret:
image: hashicorp/vault:1.18
id_tokens:
VAULT_ID_TOKEN:
aud: https://vault.example.internal
variables:
VAULT_ADDR: https://vault.example.internal
script:
- LOGIN="auth/jwt/login role=demo-app-ci jwt=$VAULT_ID_TOKEN"
- export VAULT_TOKEN="$(vault write -field=token $LOGIN)"
- vault kv get -field=username kv/demo-app/dbTip: With the
secrets:keyword, GitLab Premium offers a built-in Vault integration that performs exactly this login for you. The mechanism underneath is the same.
AppRole is meant for machines without their own platform identity, for example a VM or an old Jenkins. The role ID is like a user name, the secret ID like a short-lived password. This only shifts the problem, though: the secret ID has to reach the machine securely (keyword “secret zero”). So use AppRole only when Kubernetes or JWT auth will not work.
vault auth enable approle
vault write auth/approle/role/legacy-job \
token_policies=demo-app-read secret_id_ttl=10m secret_id_num_uses=1
vault read -field=role_id auth/approle/role/legacy-job/role-id
vault write -f -field=secret_id auth/approle/role/legacy-job/secret-idPolicies
A policy is a list of paths with permitted capabilities
(read, create, update,
delete, list, deny). Everything
that is not allowed is forbidden (default deny). Policies are written in
HCL.
Remember: With KV version 2, the data lives under
<mount>/data/...and the metadata under<mount>/metadata/.... The CLI hides this (vault kv get kv/demo-app/db), but the policy does not. A policy onkv/demo-app/*therefore matches nothing. The correct path iskv/data/demo-app/*.
Audit log
Vault can write every request and every response to an audit device (file, syslog, socket). Secret values do not appear there in plain text, but as an HMAC hash. That way you can later prove who read which path when, without the log itself becoming a secret store.
Warning: If an audit device is enabled but not writable (disk full), Vault refuses requests. That is intentional: better no access than unlogged access. For production this means: at least two audit devices, and monitor disk space.
Vault or OpenBao?
In August 2023, starting with version 1.15, HashiCorp switched Vault from the open source license MPL 2.0 to the Business Source License (BSL 1.1). The BSL allows your own use, but forbids building a competing product on it. For many public authorities and companies with strict licensing rules, this is a checkpoint. In 2025, IBM acquired HashiCorp.
In response, OpenBao was created, a fork of the last
MPL version under the umbrella of the Linux Foundation. OpenBao is
API-compatible; the CLI is called bao, the container image
openbao/openbao. Most commands in this chapter work if you
replace vault with bao. Environment variables
there are called BAO_ADDR and BAO_TOKEN.
Features of the Vault Enterprise edition (such as performance
replication or namespaces in their old form) are missing or implemented
differently. ESO talks to OpenBao via the same vault
provider.
| Feature | Vault | OpenBao |
|---|---|---|
| License | BSL 1.1 (from 1.15) | MPL 2.0 |
| Steward | HashiCorp / IBM | Linux Foundation |
| CLI / image | vault, hashicorp/vault |
bao, openbao/openbao |
| Support | commercially available | community, service providers |
External Secrets Operator (ESO)
ESO is a Kubernetes operator. It reads secrets from an external vault and creates them as normal Kubernetes Secrets. The application does not notice Vault at all. Three objects matter:
| Object | Scope | Job |
|---|---|---|
SecretStore |
one namespace | How do I reach the vault? (URL, auth) |
ClusterSecretStore |
whole cluster | the same, usable from all namespaces |
ExternalSecret |
one namespace | Which values, into which K8s Secret? |
The refreshInterval field in the ExternalSecret defines
how often ESO checks the vault. If a value changes there, ESO updates
the Kubernetes Secret. That way a rotation in Vault automatically
becomes visible in the cluster.
Remember: A
ClusterSecretStoreis convenient but powerful: any namespace can reference it. In multi-tenant clusters, aSecretStoreper namespace with its own Vault role is the safer choice.
Hands-on Lab: From a leak in the repo to a rotating secret in the cluster
You work in ~/seclab/kap06. Prerequisite: the kind
cluster seclab from Chapter 0 is running, and the tools
gitleaks, vault, helm,
kubectl and jq are installed.
Step 1: Preparation
mkdir -p ~/seclab/kap06 && cd ~/seclab/kap06
kubectl config use-context kind-seclab
docker network ls --filter name=^kind$
gitleaks version
vault versionExpected: the kind network is listed,
gitleaks reports a version 8.x, vault a
version 1.x. Note the gitleaks version. From 8.19 on there are the
git and dir subcommands; older versions only
know detect and protect.
Step 2: Create a demo repo with a leak
You simulate a typical mistake: someone checks in credentials and “fixes” it later with a second commit.
mkdir leak-demo && cd leak-demo
git init -b main
cat > config.py <<'EOF'
# Lab values, deliberately fake and invalid!
DB_HOST = "postgres"
DB_PASSWORD = "Lab-9xQ2mZ7vK4pR8tW1"
GITLAB_TOKEN = "glpat-LabFake0000Abcd1234X"
EOF
git add config.py && git commit -m "feat: DB-Konfiguration"
cat > config.py <<'EOF'
import os
DB_HOST = os.environ.get("DB_HOST", "postgres")
DB_PASSWORD = os.environ["DB_PASSWORD"]
EOF
git commit -am "fix: Secrets aus Code entfernt"
git log --onelineExpected: two commits. The current config.py no longer
contains a secret.
Security: All values here are made-up lab values. Never use real tokens in exercises, not even “just briefly”.
Step 3: gitleaks on the working copy and history
First only the files as they are on disk right now:
# gitleaks >= 8.19:
gitleaks dir . -v
# older versions:
gitleaks detect --source . --no-git -vExpected: no leaks found. The working copy is clean. Now
the entire Git history:
# gitleaks >= 8.19:
gitleaks git . -v
# older versions (still present in new versions, but deprecated):
gitleaks detect --source . -v
echo "Exit-Code: $?"Expected output (shortened):
Finding: GITLAB_TOKEN = "glpat-LabFake0000Abcd1234X"
Secret: glpat-LabFake0000Abcd1234X
RuleID: gitlab-pat
File: config.py
Line: 4
Commit: 3f1c2a9...
Fingerprint: 3f1c2a9...:config.py:gitlab-pat:4
WRN leaks found: 1
Exit-Code: 1
Depending on the gitleaks version, the generic-api-key
rule additionally reports the DB_PASSWORD. What matters:
the finding is in the first commit. The fix commit only
removed the secret from the current version, not from the history.
Anyone who clones the repo has it.
For reports and automation, use JSON and --redact, so
the value does not end up in the CI log:
gitleaks git . --redact --report-format json --report-path gitleaks.json
jq -r '.[] | "\(.RuleID) \(.File):\(.StartLine) \(.Commit[0:7])"' gitleaks.jsonExit code 1 on findings is important: it makes a CI job fail.
Step 4: Start the Vault dev server
cd ~/seclab/kap06
docker run -d --name vault --network kind -p 8200:8200 \
--cap-add=IPC_LOCK \
-e VAULT_DEV_ROOT_TOKEN_ID=root \
-e VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200 \
hashicorp/vault:1.18
export VAULT_ADDR=http://127.0.0.1:8200
export VAULT_TOKEN=root
vault statusExpected: Sealed false, Storage Type inmem.
The container is attached to the kind network. That way,
the Postgres container and the kind cluster can later reach it under the
name vault.
Warning: Dev mode keeps everything in memory, is unsealed automatically, speaks unencrypted HTTP and has the root token
root. A restart deletes all data. It is meant for the lab only.
OpenBao equivalent:
docker run -d --name vault --network kind -p 8200:8200 openbao/openbao \
server -dev -dev-root-token-id=root -dev-listen-address=0.0.0.0:8200
export BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN=root
bao statusStep 5: Create KV v2, write and read a secret
The dev server already comes with a KV engine under
secret/. You create your own, just as you would in
production:
vault secrets enable -path=kv -version=2 kv
vault kv put kv/demo-app/db username=demo password='Lab-Start-1'
vault kv get kv/demo-app/dbExpected output (shortened):
==== Secret Path ====
kv/data/demo-app/db
======= Metadata =======
Key Value
--- -----
created_time 2026-10-08T09:12:44.1Z
version 1
====== Data ======
Key Value
--- -----
password Lab-Start-1
username demo
Note kv/data/demo-app/db in the secret path: that is the
real API path. KV v2 stores versions:
vault kv put kv/demo-app/db username=demo password='Lab-Start-2'
vault kv get -version=1 -field=password kv/demo-app/db
vault kv metadata get kv/demo-app/db | grep -E 'current_version|max_versions'Expected: version 1 returns Lab-Start-1,
current_version is 2. Versions are useful for rolling back
a failed rotation
(vault kv rollback -version=1 kv/demo-app/db).
Step 6: Policy, restricted token and audit log
First enable the audit log, so that the following accesses are logged:
vault audit enable file file_path=/vault/logs/audit.logExpected:
Success! Enabled the file audit device at: file/.
Now a policy that may only read:
cat > demo-app-read.hcl <<'EOF'
path "kv/data/demo-app/*" {
capabilities = ["read"]
}
EOF
vault policy write demo-app-read demo-app-read.hcl
APP_TOKEN=$(vault token create -policy=demo-app-read -ttl=15m -field=token)
VAULT_TOKEN=$APP_TOKEN vault kv get -field=username kv/demo-app/db
VAULT_TOKEN=$APP_TOKEN vault kv put kv/demo-app/db password=hackExpected: reading returns demo. Writing fails with
Code: 403 and permission denied. Look at the
last entry in the audit log:
docker exec vault tail -n 1 /vault/logs/audit.log | jq '{
type, path: .request.path, op: .request.operation,
policies: .auth.policies, error }'Expected: type is response,
path is kv/data/demo-app/db,
policies contains demo-app-read and
error the text permission denied. The token
itself appears there only as an HMAC hash.
Step 7: Postgres and the database secrets engine
Start a Postgres database in the same Docker network:
docker run -d --name postgres --network kind -p 5432:5432 \
-e POSTGRES_PASSWORD=lab-postgres-pw -e POSTGRES_DB=demo postgres:16
sleep 5
docker exec postgres pg_isready -U postgresExpected: accepting connections. Now Vault gets the
admin credentials and a role that describes how a short-lived user is
created:
vault secrets enable database
vault write database/config/demo-postgres \
plugin_name=postgresql-database-plugin \
connection_url="postgresql://{{username}}:{{password}}@\
postgres:5432/demo?sslmode=disable" \
allowed_roles="demo-app-ro" \
username="postgres" password="lab-postgres-pw"
vault write database/roles/demo-app-ro \
db_name=demo-postgres \
default_ttl=5m max_ttl=1h \
creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' \
VALID UNTIL '{{expiration}}'; \
GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";"Expected: Success! Data written to: ... each time. Vault
reaches the database at postgres:5432 because both
containers are attached to the kind network and Docker
resolves container names there.
Tip: At this point, everyone who reads this book still knows the password
lab-postgres-pw. Withvault write -f database/rotate-root/demo-postgres, Vault sets a new admin password that only Vault knows. In production, this is part of the setup. In the lab you leave it out, so you can still log in yourself.
Step 8: Use and revoke dynamic credentials
vault read -format=json database/creds/demo-app-ro > creds.json
jq '{lease_id, lease_duration, user: .data.username}' creds.json
LEASE=$(jq -r .lease_id creds.json)
DBUSER=$(jq -r .data.username creds.json)
DBPASS=$(jq -r .data.password creds.json)Expected output:
{
"lease_id": "database/creds/demo-app-ro/Xy7kP...",
"lease_duration": 300,
"user": "v-root-demo-app-ro-AbC1d2EfG3h4IjK5lM6n-1791451234"
}
300 seconds are the 5 minutes from default_ttl. Test the
login from a second container, so that a password check really happens
over the network:
docker run --rm --network kind -e PGPASSWORD="$DBPASS" postgres:16 \
psql -h postgres -U "$DBUSER" -d demo -c 'select current_user;'
docker exec postgres psql -U postgres -c '\du' | grep v-rootExpected: current_user shows the v-root-...
name, and \du lists this role. Now you revoke the lease as
if the credential had leaked:
vault lease lookup "$LEASE" | grep ttl
vault lease revoke "$LEASE"
docker run --rm --network kind -e PGPASSWORD="$DBPASS" postgres:16 \
psql -h postgres -U "$DBUSER" -d demo -c 'select 1;'Expected: after the revoke, the login fails
(password authentication failed or
role ... does not exist). Vault has deleted the user in
Postgres. Had you done nothing, the same would have happened
automatically after 5 minutes. You revoke all leases of a role with
vault lease revoke -prefix database/creds/demo-app-ro.
Step 9: Install the External Secrets Operator via Helm
helm repo add external-secrets https://charts.external-secrets.io
helm repo update
helm install external-secrets external-secrets/external-secrets \
-n external-secrets --create-namespace --wait
kubectl get pods -n external-secrets
kubectl api-resources --api-group=external-secrets.io | grep -i secretstoreExpected: three pods Running (controller, webhook, cert
controller). The last line shows the API version, for example
external-secrets.io/v1. Older ESO versions (before 0.16)
only know v1beta1. In that case, adjust
apiVersion in the YAML files below.
Step 10: Configure Kubernetes auth in Vault
Here lies the networking peculiarity of this lab. Vault does not run in the cluster, but as a container next to it. Both directions have to work:
Docker network "kind"
+-------------------------------------------------------------------+
| Container "vault" --- TokenReview ---> seclab-control-plane:6443 |
| ^ (K8s API in kind node) |
| | http://vault:8200 |
| ESO pod in kind node (pod DNS -> node -> Docker DNS) |
+-------------------------------------------------------------------+
- ESO in the cluster must reach Vault: via the name
vault, which Docker resolves in thekindnetwork. - Vault must reach the Kubernetes API to verify ServiceAccount tokens:
via the name of the control plane container
seclab-control-planeon port 6443. The127.0.0.1:...entry from your kubeconfig does not work from inside the Vault container.
First check the cluster → Vault direction:
kubectl create namespace demo --dry-run=client -o yaml | kubectl apply -f -
kubectl run nettest -n demo --rm -i --restart=Never \
--image=curlimages/curl -- curl -s http://vault:8200/v1/sys/healthExpected: JSON with "initialized":true,"sealed":false.
kind itself shows you the internal API address:
kind get kubeconfig --name seclab --internal | grep serverExpected: server: https://seclab-control-plane:6443. Now
you create the ServiceAccount that ESO uses to log in to Vault. It may
additionally perform TokenReviews (system:auth-delegator),
so that Vault can use its token for verification:
kubectl create serviceaccount eso-vault -n demo
kubectl create clusterrolebinding eso-vault-tokenreview \
--clusterrole=system:auth-delegator --serviceaccount=demo:eso-vault
kubectl config view --raw --minify --flatten \
-o jsonpath='{.clusters[0].cluster.certificate-authority-data}' \
| base64 --decode > k8s-ca.crt
vault auth enable kubernetes
vault write auth/kubernetes/config \
kubernetes_host="https://seclab-control-plane:6443" \
[email protected]
vault write auth/kubernetes/role/demo-app \
bound_service_account_names=eso-vault \
bound_service_account_namespaces=demo \
token_policies=demo-app-read token_ttl=15mBecause you did not set a token_reviewer_jwt and Vault
runs outside the cluster, Vault uses the token of the ServiceAccount
that is logging in for the TokenReview itself. That is why the
ClusterRoleBinding is needed. Test the login by hand before ESO comes
into play:
SA_JWT=$(kubectl create token eso-vault -n demo)
vault write auth/kubernetes/login role=demo-app jwt="$SA_JWT" | grep policiesExpected: token_policies ["default" "demo-app-read"].
If this does not work, the error lies in Vault or the network, not in
ESO. That saves a lot of searching.
Step 11: SecretStore and ExternalSecret
cat > secretstore.yaml <<'EOF'
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
name: vault-backend
namespace: demo
spec:
provider:
vault:
server: "http://vault:8200"
path: "kv"
version: "v2"
auth:
kubernetes:
mountPath: "kubernetes"
role: "demo-app"
serviceAccountRef:
name: "eso-vault"
EOF
cat > externalsecret.yaml <<'EOF'
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: demo-app-db
namespace: demo
spec:
refreshInterval: 15s
secretStoreRef:
name: vault-backend
kind: SecretStore
target:
name: demo-app-db
creationPolicy: Owner
data:
- secretKey: DB_USER
remoteRef:
key: demo-app/db
property: username
- secretKey: DB_PASSWORD
remoteRef:
key: demo-app/db
property: password
EOF
kubectl apply -f secretstore.yaml -f externalsecret.yaml
kubectl get secretstore,externalsecret -n demoExpected output:
NAME AGE STATUS CAPABILITIES READY
secretstore.external-secrets.io/vault-backend 10s Valid ReadWrite True
NAME STORE STATUS READY
externalsecret.external-secrets.io/demo-app-db vault-backend SecretSynced True
(The columns are shortened here; depending on the ESO version you
also see REFRESH.)
Note: the SecretStore says path: "kv" and the
ExternalSecret says key: demo-app/db, without
data/. ESO adds that itself, because
version: "v2" is set. Check the content:
kubectl get secret demo-app-db -n demo \
-o jsonpath='{.data.DB_PASSWORD}' | base64 --decode; echoExpected: Lab-Start-2.
Remember:
refreshInterval: 15sis this short only for the demonstration. Every sync is a Vault login plus a read. In production, 5 to 60 minutes is common.
Step 12: Demonstrate rotation
Change the value in Vault, the way a rotation process would:
vault kv put kv/demo-app/db username=demo password='Lab-Rotiert-3'
sleep 20
kubectl get secret demo-app-db -n demo \
-o jsonpath='{.data.DB_PASSWORD}' | base64 --decode; echo
kubectl get externalsecret demo-app-db -n demo \
-o jsonpath='{.status.refreshTime}'; echoExpected: Lab-Rotiert-3 and a fresh timestamp. If you do
not want to wait, force a sync:
kubectl annotate externalsecret demo-app-db -n demo \
force-sync="$(date +%s)" --overwriteWarning: The Kubernetes Secret is now current, but the running pod is not necessarily. Environment variables are read only at pod start. Secrets mounted as files are updated by the kubelet after a while (not with
subPath). The application then has to reread the file. Common approaches are a rollout restart (kubectl rollout restart) or a tool like Stakater Reloader, which restarts automatically when a Secret changes.
Check in the audit log that ESO read with its own identity:
docker exec vault cat /vault/logs/audit.log \
| jq -r 'select(.type=="response" and .auth.metadata.role=="demo-app")
| .request.path' | sort | uniq -cExpected: lines for auth/kubernetes/login and
kv/data/demo-app/db. The counts grow with every refresh.
The role appears in the metadata in plain text, the secret values do
not.
Step 13: Replacement plan for the leaked credential
Back to the leak from Step 2. The order is decisive:
1. Contain -> revoke/rotate the credential (minutes)
2. Assess -> Was it used? Check the logs (hours)
3. Switch over -> code reads from secret source (days)
4. Clean up -> rewrite history (optional) (planned)
5. Prevent -> pre-commit + CI scan (permanent)
1. Contain. You revoke the GitLab token in GitLab under User Settings → Access Tokens → Revoke. You replace the DB password with a new value in Vault. You already practiced that in Step 12. Even better: dynamic credentials as in Step 8, then there is no long-lived password at all.
2. Assess. Check the logs of the affected system (GitLab audit events, DB logins) to see whether the credential was used between the commit and the revocation. That determines whether it becomes a security incident with a reporting obligation.
3. Switch over. The code now reads
os.environ["DB_PASSWORD"]. The Kubernetes Secret
demo-app-db from ESO provides exactly this variable. In the
demo-app Deployment, it looks like this:
envFrom:
- secretRef:
name: demo-app-db4. Clean up. Only now, once the old credential is worthless, do you rewrite the history.
Warning: Cleaning up history is no substitute for rotation. The secret already sits in clones on laptops, in CI caches, in forks, in merge request diffs and possibly in backups. You cannot get it back. A rotated secret is harmless, even if it still appears somewhere. A cleaned-up but unrotated secret is still valid.
Step 14: Clean up history with git filter-repo
git filter-repo is a standalone tool (install it, e.g.,
via pip install git-filter-repo or your package manager).
For safety reasons, it only works on a fresh clone:
cd ~/seclab/kap06
git clone --no-local leak-demo leak-demo-clean
cd leak-demo-clean
cat > ../replacements.txt <<'EOF'
glpat-LabFake0000Abcd1234X==>***ENTFERNT***
Lab-9xQ2mZ7vK4pR8tW1==>***ENTFERNT***
EOF
git log --oneline
git filter-repo --replace-text ../replacements.txt
git log --oneline
gitleaks git . -vExpected: the commit IDs before and after filter-repo
differ, because every affected commit was rewritten.
gitleaks reports no leaks found. With
git show HEAD~1:config.py you see
***ENTFERNT*** (“removed”) in place of the values.
In a real project, further steps follow:
filter-reporemoves theoriginremote. You add it again and push withgit push --force --allandgit push --force --tags. For this, branch protection has to be relaxed briefly.- Everyone involved has to clone again. A
git pullon an old clone brings the old history back. - GitLab still keeps old objects in merge request refs and caches. Housekeeping in the project (Settings → General → Advanced → Housekeeping) and, if necessary, removing affected MRs are part of it.
Step 15: Pre-commit hook with gitleaks
So that it does not happen again, a hook checks every commit before it is saved:
cd ~/seclab/kap06/leak-demo-clean
cat > .git/hooks/pre-commit <<'EOF'
#!/bin/sh
# gitleaks >= 8.19; older versions: gitleaks protect --staged --redact -v
exec gitleaks git --pre-commit --staged --redact -v
EOF
chmod +x .git/hooks/pre-commit
echo 'TOKEN = "glpat-LabFake9999Wxyz5678Q"' > neu.py
git add neu.py
git commit -m "test: sollte blockiert werden"
echo "Exit-Code: $?"Expected: leaks found: 1, exit code 1, no new commit
(git log --oneline does not show it). Remove the test file:
git reset neu.py && rm neu.py.
In a team, you distribute the hook via the pre-commit
framework with a .pre-commit-config.yaml in the repo:
repos:
- repo: https://github.com/gitleaks/gitleaks
rev: v8.21.2
hooks:
- id: gitleaksLocal hooks can be bypassed with git commit --no-verify.
That is why the same scan also belongs in the pipeline. Nobody can skip
it there:
secret-scan:
stage: test
image:
name: zricethezav/gitleaks:v8.21.2
entrypoint: [""]
variables:
GIT_DEPTH: "0"
script:
- gitleaks git . --redact -v --report-format json --report-path gitleaks.json
artifacts:
when: always
paths: [gitleaks.json]GIT_DEPTH: "0" fetches the full history. With the
default shallow clone, gitleaks would only see the last commits. In an
isolated environment, you pull the image from your mirror registry (see
Chapter 3).
Step 16: Clean up
kubectl delete -f ~/seclab/kap06/externalsecret.yaml -f ~/seclab/kap06/secretstore.yaml
kubectl delete clusterrolebinding eso-vault-tokenreview
kubectl delete serviceaccount eso-vault -n demo
helm uninstall external-secrets -n external-secrets
kubectl delete namespace external-secrets
docker rm -f vault postgresExpected: all commands run without errors. docker ps
shows neither vault nor postgres. Leave the
demo namespace in place; later chapters need it.
Save your progress? Sign up for free
Common pitfalls
| Symptom | Cause | Fix |
|---|---|---|
permission denied despite a policy |
Policy on kv/demo-app/* instead of
kv/data/... |
with KV v2, data/ in the policy path |
| gitleaks finds nothing in CI | Shallow clone | Set GIT_DEPTH: "0" |
SecretStore InvalidProviderConfig |
Vault name cannot be resolved | Check the kind network; as a last resort, IP via
docker inspect |
Login: permission denied with K8s auth |
SA or namespace does not match the role | Check bound_service_account_* |
Login: x509 or connection refused |
kubernetes_host points to 127.0.0.1 |
https://seclab-control-plane:6443 |
TokenReview 403 |
SA lacks system:auth-delegator |
Create the ClusterRoleBinding |
| Secret updated, app uses the old value | Env variables only at start | Rollout restart or Reloader |
| Everything gone after a Vault restart | Dev mode stores in RAM | Repeat the lab steps from 4 on |
no matches for kind ExternalSecret |
Wrong apiVersion |
Check kubectl api-resources |
As a stopgap for name resolution, you determine the IP like this:
docker inspect -f '{{.NetworkSettings.Networks.kind.IPAddress}}' vault.
However, this IP changes when the container restarts.
Exercise: A second team, separate vault compartments
A second team runs an application in the team-b
namespace. It should get its own secret under kv/team-b/api
with the key apikey (lab value Lab-TeamB-1).
Set up Vault and ESO so that:
- a Kubernetes Secret
team-b-apiwith the value is created in theteam-bnamespace, - the SecretStore in the
demonamespace can not read this secret (prove it!), - the team B secret is synced at most every 5 minutes.
Restart Vault and ESO first if you have already cleaned up (Steps 4, 5, 6, 9, 10, 11).
Hint 1
You need three things per team: a policy and a role in the Kubernetes auth method in Vault, and a ServiceAccount in the cluster. The Vault role connects ServiceAccount and namespace with the policy. Remember the ClusterRoleBinding for TokenReview.
Hint 2
The easiest way to prove point 2 is a second ExternalSecret in the
demo namespace that tries to read
key: team-b/api from the vault-backend store.
Its status should not be SecretSynced.
kubectl describe externalsecret shows you Vault’s error
message.
Solution
vault kv put kv/team-b/api apikey='Lab-TeamB-1'
cat > team-b-read.hcl <<'EOF'
path "kv/data/team-b/*" {
capabilities = ["read"]
}
EOF
vault policy write team-b-read team-b-read.hcl
kubectl create namespace team-b
kubectl create serviceaccount eso-vault -n team-b
kubectl create clusterrolebinding eso-vault-tokenreview-team-b \
--clusterrole=system:auth-delegator --serviceaccount=team-b:eso-vault
vault write auth/kubernetes/role/team-b \
bound_service_account_names=eso-vault \
bound_service_account_namespaces=team-b \
token_policies=team-b-read token_ttl=15mFor the Kubernetes objects, copy secretstore.yaml and
externalsecret.yaml and change
namespace: team-b, role: "team-b",
refreshInterval: 5m, target.name: team-b-api
and the data block:
data:
- secretKey: APIKEY
remoteRef:
key: team-b/api
property: apikeyAfter kubectl apply,
kubectl get externalsecret -n team-b shows the status
SecretSynced. For the negative test, create an
ExternalSecret klau-versuch (“theft attempt”) with
key: team-b/api in the demo namespace.
Expected: READY False, and
kubectl describe externalsecret klau-versuch -n demo shows
an error with permission denied or 403. The
reason: the store in demo logs in with the
demo-app role, and its policy only allows
kv/data/demo-app/*.
So the separation does not come from ESO, but from the combination of
Vault role (who am I?) and policy (what may I do?). A shared
ClusterSecretStore with a single Vault role would have
removed this separation.
Save your progress? Sign up for free
In practice
Running Vault/OpenBao. In production, Vault does not
run in dev mode, but as a cluster of three or five nodes with integrated
Raft storage, TLS on all connections and auto-unseal via an HSM or a
KMS. In isolated (air-gapped) environments, the HSM is often the only
option, because no cloud KMS is reachable. The root token is revoked
after setup (vault token revoke) and only regenerated in an
emergency via the holders of the unseal or recovery keys (four-eyes
principle with Shamir key shares).
Who manages Vault itself? Policies, roles and auth methods belong in Git and are rolled out via Terraform/OpenTofu or a dedicated pipeline job. Then permission changes can be reviewed via merge requests. No human clicks policies together in the UI.
Audit and monitoring. The audit log goes to the
central SIEM. Sensible alerts: root token used, policy changed, many
permission denied from one identity, audit device failed.
Vault exports metrics in Prometheus format
(/v1/sys/metrics). Worth watching there: number of leases,
seal status and request latency. Many leases without revokes point to
applications that pull new credentials on every start and never clean
up.
Public authorities and KRITIS. Auditors at public authorities and operators of critical infrastructure (KRITIS) usually ask: Where are the secrets? Who has access? How often are they rotated? How quickly can they be revoked? Is there proof? With Vault you answer this from the policies, the TTLs and the audit log. In the IT-Grundschutz of the BSI (the German Federal Office for Information Security’s baseline protection catalog), you will find matching requirements, for example in the modules on crypto concept and key management and on identity and access management. Clarify licensing questions (BSL vs. MPL) early with procurement. Some clients explicitly require open source licenses; then OpenBao is the given choice.
Replacement in an existing estate. In organizations that have grown over time, a first gitleaks run over all repos often finds hundreds of findings. Do not tackle them all at once. What has proven itself:
- Classify findings by type (cloud keys, DB passwords, test dummies).
- Rotate real, still valid credentials immediately.
- Create a baseline for legacy findings (
--baseline-pathin gitleaks), so that new findings stand out immediately. - Agree on transition periods with the teams for switching to Vault (see Chapter 10).
No secret in Git, not even encrypted? Tools like
SOPS or Sealed Secrets put encrypted secrets into the repo. That is
better than plain text, but does not solve rotation, revocation and
audit. ESO with Vault cleanly separates storage and distribution and is
therefore often the preferred solution in regulated environments. Argo
CD or Flux (see Chapter 1) then only roll out the
ExternalSecret, never the value itself.
Interview questions
Q: A colleague pushed a production database password to an internal repo and deleted it again ten minutes later. What do you do?
A: First rotate the password or revoke access,
because it is still in the history and in every clone. Then check the DB
logs to see whether it was misused, and depending on the result, start
the incident process. Then switch the code to a secret source, clean up
the history with git filter-repo, and introduce gitleaks as
a pre-commit hook and CI job.
Q: What is the difference between static and dynamic secrets?
A: A static secret exists independently of the vault and is only stored there. You have to rotate it yourself. A dynamic secret is created by Vault only on request, individually per consumer and with a lease. When the TTL expires or on revoke, Vault deletes it in the target system again.
Q: How does a GitLab CI job authenticate to Vault without a token stored in CI variables?
A: Via id_tokens, the job requests a
JWT signed by GitLab with the matching audience. Vault verifies it with
the JWT auth method against GitLab’s JWKS endpoint and compares claims
such as project_path, ref and
ref_protected with the role’s bound_claims.
The job gets a short-lived Vault token with exactly that role’s
policy.
Q: Why does Vault need access to the Kubernetes API when it uses Kubernetes auth?
A: Vault verifies the pod’s ServiceAccount token via the TokenReview API. That is the only way it knows whether the token is valid and which ServiceAccount and namespace it belongs to. If Vault runs outside the cluster, it needs network access to the API, the CA certificate and a JWT with permission for TokenReview.
Q: What is the difference between
SecretStore and ClusterSecretStore in the
External Secrets Operator?
A: A SecretStore applies to one
namespace and can only be used by ExternalSecrets there. A
ClusterSecretStore applies cluster-wide. For tenant
separation, you use SecretStores per namespace with their own Vault role
and policy; otherwise every namespace could read everything the shared
role is allowed to.
Q: How does a rotation in Vault reach the running application?
A: ESO checks at every refreshInterval
and updates the Kubernetes Secret. The kubelet then updates mounted
secret files, and the application has to reread them. Environment
variables stay stale until a restart. That is why you need a rollout
restart or a reloader. For databases, you let old and new credentials be
valid in parallel for a short time.
Q: Vault or OpenBao, how do you decide?
A: Technically, both are largely the same for the core functions and API-compatible. What decides is the license (BSL for Vault from 1.15, MPL 2.0 for OpenBao), the need for enterprise features, support requirements and procurement rules. In environments with an open source mandate, OpenBao is the obvious choice.
Q: Why is a policy on kv/myapp/*
ineffective with KV v2?
A: KV v2 stores the data under
kv/data/myapp/... and the metadata under
kv/metadata/myapp/.... The CLI hides this, but policies
work with the real API paths.
Summary
Here is what you learned:
- A deleted secret is not gone: gitleaks finds it in the history, and only rotation makes it worthless.
- A secret has a lifecycle. Vault covers storing, distributing, rotating, revoking and auditing.
- Dynamic secrets with lease and TTL are unique per consumer, expire on their own and can be revoked selectively.
- Consumers log in with platform identities: Kubernetes
ServiceAccounts, GitLab
id_tokensvia JWT, AppRole as a last resort. - Policies work with real API paths and default deny. The audit log proves every access.
- ESO syncs secrets from Vault into Kubernetes. With
refreshInterval, rotations arrive automatically, but running pods need a restart or reload. - If Vault runs outside the kind cluster, the Docker network
kindconnects both sides via the namesvaultandseclab-control-plane. - OpenBao is the MPL-licensed fork of Vault with the
baoCLI and a compatible API.