Day 5: Certificates
Goal: The mesh trusts your own trust anchor with a ten-year lifetime, the switch ran without TLS errors, and cert-manager renews the issuer on its own. · Duration: approx. 90 min · Prerequisite: Day 4
Briefing
- Three levels: the trust anchor signs the
issuer, the issuer (in
linkerd-identity) signs the proxies’ 24-hour certificates (Day 3). - On Day 1,
linkerd installgenerated a single self-signed certificate that is both: anchor and issuer, valid for one year, without renewal. - You replace the anchor in three rounds: a bundle of old and new, swap the issuer, remove the old anchor. After each round, all proxies restart, because they only read the anchors at startup.
- Afterwards, cert-manager takes over the issuer (48 hours, renewed automatically) and trust-manager the ConfigMap with the anchor.
ca.crtandca.keystay around as files: the second cluster on Day 12 needs the same anchor.
ca.crt/ca.key (10 years, file + Secret in cert-manager)
└─ signs ─> linkerd-identity-issuer (48 h, cert-manager)
└─ signs ─> proxy certificates (24 h, linkerd-identity)
trust-manager: ca.crt ─> ConfigMap linkerd/linkerd-identity-trust-roots
Drill 1: Inventory and tooling
mkdir -p ~/linkerd/tag05 && cd ~/linkerd/tag05
kubectl -n linkerd get configmap linkerd-identity-trust-roots \
-o jsonpath='{.data.ca-bundle\.crt}' > original-trust.crt
openssl x509 -in original-trust.crt -noout -subject -issuer -enddateExpected:
subject=CN = identity.linkerd.
issuer=CN = identity.linkerd.
notAfter=Nov 19 11:23:11 2027 GMT
Subject equals issuer: self-signed. The same certificate is also in
the Secret linkerd-identity-issuer. In one year, the entire
mesh would come to a halt.
For your own certificates, you use the step CLI from
Smallstep:
ARCH=$(dpkg --print-architecture)
V=0.31.0
curl -fsSLO \
https://github.com/smallstep/cli/releases/download/v$V/step_linux_${V}_$ARCH.tar.gz
curl -fsSL https://api.github.com/repos/smallstep/cli/releases/tags/v$V \
| grep -c "$(sha256sum step_linux_${V}_$ARCH.tar.gz | cut -d' ' -f1)"
tar xzf step_linux_${V}_$ARCH.tar.gz
sudo install -m 0755 step_$V/bin/step /usr/local/bin/step
step versionExpected:
1
Smallstep CLI/0.31.0 (linux/amd64)
Release Date: 2026-09-25T19:53:51Z
The 1 means: the checksum of your file is listed in the
release – as with the Linkerd CLI on Day 1.
Save your progress? Sign up for free
Drill 2: Your own trust anchor and issuer
step certificate create root.linkerd.cluster.local ca.crt ca.key \
--profile root-ca --no-password --insecure --not-after 87600h
step certificate create identity.linkerd.cluster.local issuer.crt issuer.key \
--profile intermediate-ca --not-after 8760h --no-password --insecure \
--ca ca.crt --ca-key ca.key
chmod 600 ca.key issuer.key
step certificate inspect ca.crt --short
step certificate inspect issuer.crt --short
step certificate verify issuer.crt --roots ca.crt && echo verify-okExpected:
Your certificate has been saved in ca.crt.
Your private key has been saved in ca.key.
Your certificate has been saved in issuer.crt.
Your private key has been saved in issuer.key.
X.509v3 Root CA Certificate (ECDSA P-256) [Serial: 6266...5288]
Subject: root.linkerd.cluster.local
Issuer: root.linkerd.cluster.local
Valid from: 2026-11-19T15:51:50Z
to: 2036-11-16T15:51:50Z
X.509v3 Intermediate CA Certificate (ECDSA P-256) [Serial: 1187...8861]
Subject: identity.linkerd.cluster.local
Issuer: root.linkerd.cluster.local
Valid from: 2026-11-19T15:51:50Z
to: 2027-11-19T15:51:50Z
verify-ok
87,600 hours are ten years. The names follow the Linkerd docs: the
issuer is called identity.linkerd.<trust-domain>.
--no-password --insecure leaves the keys unencrypted –
required for the issuer, acceptable for ca.key only in the
lab.
Warning: From now on,
ca.keyis the most valuable key in the mesh. Whoever has it can forge any identity. In production, it is kept offline or in a vault.
Save your progress? Sign up for free
Drill 3: Round 1 – both anchors
Note the start time; in Drill 6 you use it to measure the entire switch.
date +%s > start.txt
step certificate bundle ca.crt original-trust.crt bundle.crt
grep -c 'BEGIN CERTIFICATE' bundle.crt
linkerd upgrade --identity-trust-anchors-file=./bundle.crt > upgrade-1.yaml
kubectl apply -f upgrade-1.yaml | grep -E 'trust-roots|identity '
kubectl -n linkerd rollout status deploy --timeout=300s
linkerd check --proxy 2>&1 | grep -A4 'match CA'Expected (abridged):
Your certificate has been saved in bundle.crt.
2
configmap/linkerd-identity-trust-roots configured
...
deployment "linkerd-proxy-injector" successfully rolled out
‼ data plane proxies certificate match CA
Some pods do not have the current trust bundle and must be restarted:
* linkerd-viz/metrics-api-6cfdd5799-kzwzj
* linkerd-viz/tap-854d684c76-s554k
...
The Control Plane has already restarted –
linkerd upgrade generates new webhook certificates on every
run. All other proxies only know both anchors after a restart:
kubectl -n linkerd-viz rollout restart deploy
kubectl -n notes rollout restart deploy
kubectl -n notes rollout restart statefulset
kubectl -n linkerd-viz rollout status deploy --timeout=300s
kubectl -n notes rollout status deploy/notes-api --timeout=300s
kubectl -n notes rollout status statefulset/postgres --timeout=300s
sleep 20
linkerd check --proxy 2>&1 | grep -E '‼|×|Status'Expected:
Status check results are √
If a ‼ still appears, an old Pod is stuck terminating.
Wait and check again. If you have more namespaces with proxies, restart
them the same way.
Save your progress? Sign up for free
Drill 4: Round 2 – new issuer
linkerd upgrade --identity-issuer-certificate-file=./issuer.crt \
--identity-issuer-key-file=./issuer.key > upgrade-2.yaml
kubectl apply -f upgrade-2.yaml | grep -E 'identity-issuer|identity '
kubectl -n linkerd rollout status deploy --timeout=300s
kubectl get events -n linkerd --field-selector reason=IssuerUpdatedExpected (abridged):
secret/linkerd-identity-issuer configured
deployment.apps/linkerd-identity configured
...
LAST SEEN TYPE REASON OBJECT MESSAGE
26s Normal IssuerUpdated deployment/linkerd-identity Updated identity iss…
From now on, identity issues certificates under the new
anchor. This works because every proxy has trusted both anchors since
Round 1. Restart all proxies again so that they fetch new
certificates:
kubectl -n linkerd-viz rollout restart deploy
kubectl -n notes rollout restart deploy
kubectl -n notes rollout restart statefulset
kubectl -n linkerd-viz rollout status deploy --timeout=300s
kubectl -n notes rollout status deploy/notes-api --timeout=300s
kubectl -n notes rollout status statefulset/postgres --timeout=300s
sleep 15
linkerd identity -n notes -l app=web | grep 'Issuer:' | sort | uniq -cExpected:
2 Issuer: CN=identity.linkerd.cluster.local
Before, it said CN=identity.linkerd. – the old,
self-signed issuer.
Save your progress? Sign up for free
Drill 5: Round 3 – remove the old anchor
linkerd upgrade --identity-trust-anchors-file=./ca.crt > upgrade-3.yaml
kubectl apply -f upgrade-3.yaml | grep trust-roots
kubectl -n linkerd rollout status deploy --timeout=300s
kubectl -n linkerd-viz rollout restart deploy
kubectl -n notes rollout restart deploy
kubectl -n notes rollout restart statefulset
kubectl -n linkerd-viz rollout status deploy --timeout=300s
kubectl -n notes rollout status deploy/notes-api --timeout=300s
kubectl -n notes rollout status statefulset/postgres --timeout=300s
sleep 20
kubectl -n linkerd get configmap linkerd-identity-trust-roots \
-o jsonpath='{.data.ca-bundle\.crt}' | step certificate inspect --short -
linkerd check --proxy 2>&1 | grep -E '‼|×|Status'Expected (abridged):
‼ Rotating the trust anchors will affect existing proxies
See https://linkerd.io/2/tasks/rotating_identity_certificates/ for more information
configmap/linkerd-identity-trust-roots configured
...
X.509v3 Root CA Certificate (ECDSA P-256) [Serial: 6266...5288]
Subject: root.linkerd.cluster.local
Issuer: root.linkerd.cluster.local
Valid from: 2026-11-19T15:51:50Z
to: 2036-11-16T15:51:50Z
Status check results are √
The warning from linkerd upgrade refers to exactly the
restart you just did. The old anchor is gone.
Warning: If you skip Round 2 and remove the old anchor directly, you take down the mesh: the proxies then hold certificates that nobody trusts anymore.
Save your progress? Sign up for free
Drill 6: The proof
Three rounds of restarts – what did the load generator get hit with? Its log is gone with every restart, but Prometheus has everything:
kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090 \
>/dev/null 2>&1 &
sleep 4
R=$(( $(date +%s) - $(cat start.txt) ))s
Q="sum by (dst_deployment, status_code) (increase(response_total{namespace=\"notes\",
deployment=\"web\", direction=\"outbound\"}[$R]))"
curl -s localhost:9090/api/v1/query --data-urlencode "query=$Q" | jq -r \
'.data.result[] | .metric.dst_deployment + " " + .metric.status_code + " "
+ (.value[1] | tonumber | round | tostring)'Expected (numbers vary):
helper 200 425
helper 500 156
notes-api 200 1224
notes-api 500 8
helper delivers its usual 30 percent errors. Not a
single 502 or 503 – those would be the proxy’s responses to failed TLS.
That leaves eight 500s from notes-api. When did they
occur?
Q='sum(increase(response_total{namespace="notes",deployment="web",
direction="outbound",dst_deployment="notes-api",classification="failure"}[1m]))'
curl -s localhost:9090/api/v1/query_range --data-urlencode "query=$Q" \
--data-urlencode start=$(cat start.txt) --data-urlencode end=$(date +%s) \
--data-urlencode step=20 \
| jq -r '.data.result[0].values[] | (.[0]|todate)+" "+.[1]' | awk '$2>0'
Q='max(kube_pod_start_time{namespace="notes",pod="postgres-0"})'
curl -s localhost:9090/api/v1/query_range --data-urlencode "query=$Q" \
--data-urlencode start=$(cat start.txt) --data-urlencode end=$(date +%s) \
--data-urlencode step=30 \
| jq -r '[.data.result[0].values[][1]] | unique | .[] | tonumber | todate'
kill %1Expected (abridged):
2026-11-19T15:53:17Z 4
2026-11-19T15:53:37Z 4
2026-11-19T15:55:37Z 4.295
2026-11-19T15:55:57Z 4
2026-11-19T15:56:17Z 3.705
2026-11-19T15:41:16Z
2026-11-19T15:52:47Z
2026-11-19T15:55:19Z
2026-11-19T15:57:29Z
The first start time is the restart from Day 4, followed by the three
rounds. The errors coincide with the restarts of postgres-0
in Rounds 1 and 2: a single replica that is gone for a few seconds. This
has nothing to do with the certificates – you saw the same thing on Day
4 without a switch. The anchor switch itself ran without an outage.
Save your progress? Sign up for free
Drill 7: trust-manager takes over the anchor
cert-manager v1.21.3 is already running in the lab (namespace
cert-manager). If you do not have it:
helm install cert-manager oci://quay.io/jetstack/charts/cert-manager --version v1.21.3 -n cert-manager --create-namespace --set crds.enabled=true.
helm install trust-manager oci://quay.io/jetstack/charts/trust-manager \
--version v0.26.0 -n cert-manager --set app.trust.namespace=cert-manager --wait
kubectl -n cert-manager create secret tls linkerd-trust-anchor \
--cert=ca.crt --key=ca.key
cat > cert-manager.yaml <<'EOF'
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata: {name: linkerd-identity-issuer}
spec:
ca: {secretName: linkerd-trust-anchor}
---
apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata: {name: linkerd-identity-trust-roots}
spec:
sources:
- secret: {name: linkerd-trust-anchor, key: tls.crt}
target:
configMap: {key: ca-bundle.crt}
namespaceSelector:
matchLabels: {linkerd.io/is-control-plane: "true"}
EOF
kubectl apply -f cert-manager.yaml
sleep 10
kubectl get clusterissuer linkerd-identity-issuer
kubectl get bundle linkerd-identity-trust-roots
kubectl -n linkerd get configmap linkerd-identity-trust-roots \
-o jsonpath='{.metadata.ownerReferences[0].kind}{"\n"}'Expected (abridged):
STATUS: deployed
secret/linkerd-trust-anchor created
clusterissuer.cert-manager.io/linkerd-identity-issuer created
bundle.trust.cert-manager.io/linkerd-identity-trust-roots created
NAME READY AGE
linkerd-identity-issuer True 11s
NAME CONFIGMAP TARGET SECRET TARGET SYNCED REASON AGE
linkerd-identity-trust-roots ca-bundle.crt True Synced 11s
Bundle
- The anchor goes as a Secret into
cert-manager. trust-manager may only read from its trust namespace, and the ClusterIssuer also looks for its Secrets there. - The Bundle name is the name of the target ConfigMap. trust-manager has taken over the existing ConfigMap; the content remains the same anchor, no restart needed.
linkerd installhas already set the labellinkerd.io/is-control-plane=true.
Save your progress? Sign up for free
Drill 8: cert-manager takes over the issuer
cat > identity-issuer.yaml <<'EOF'
apiVersion: cert-manager.io/v1
kind: Certificate
metadata: {name: linkerd-identity-issuer, namespace: linkerd}
spec:
issuerRef: {name: linkerd-identity-issuer, kind: ClusterIssuer}
secretName: linkerd-identity-issuer
isCA: true
commonName: identity.linkerd.cluster.local
duration: 48h0m0s
renewBefore: 25h0m0s
privateKey: {rotationPolicy: Always, algorithm: ECDSA}
EOF
kubectl apply -f identity-issuer.yaml
sleep 15
kubectl -n linkerd get certificate
kubectl -n linkerd get secret linkerd-identity-issuer -o jsonpath='{.data}' \
| jq -r 'keys[]'Expected:
certificate.cert-manager.io/linkerd-identity-issuer created
NAME READY SECRET AGE
linkerd-identity-issuer True linkerd-identity-issuer 15s
ca.crt
crt.pem
key.pem
tls.crt
tls.key
cert-manager writes into the existing Secret: tls.* are
added, the old crt.pem/key.pem from Round 2
remain. Now you switch Linkerd to the external issuer – only then does
identity read the tls.* keys:
linkerd upgrade --set identity.externalCA=true \
--set identity.issuer.scheme=kubernetes.io/tls > upgrade-4.yaml
grep -c 'name: linkerd-identity-issuer$' upgrade-4.yaml
kubectl apply -f upgrade-4.yaml | grep 'identity '
kubectl -n linkerd rollout status deploy --timeout=300s
kubectl -n linkerd logs deploy/linkerd-identity -c identity | grep 'Issuer cert loaded'
linkerd check 2>&1 | grep -A1 '‼'Expected (abridged):
0
deployment.apps/linkerd-identity configured
...
level=info msg="Issuer cert loaded" invalid_after=1795276812 ... ttl_seconds=172762
‼ issuer cert is valid for at least 60 days
issuer certificate will expire on 2026-11-21T16:00:12Z
The manifest no longer contains the issuer Secret or the trust
ConfigMap – both now belong to cert-manager and trust-manager. The
warning about the 60-day period is intended for a 48-hour issuer;
linkerd check still ends with √.
A pitfall awaits at the next linkerd upgrade – so right
away:
linkerd upgrade > /dev/nullExpected (wrapped):
--identity-issuer-certificate-file must not be specified if
--identity-external-issuer=true
The CLI remembers all values from earlier upgrades in the Secret
linkerd-config-overrides – including the certificate and
private key of the issuer from Round 2. Both have to
go:
(umask 077; kubectl -n linkerd get secret linkerd-config-overrides \
-o jsonpath='{.data.linkerd-config-overrides}' | base64 -d > overrides.yaml)
sed -i '/^ tls:/,/^identityTrustAnchorsPEM/{/^identityTrustAnchorsPEM/!d}' \
overrides.yaml
grep -c 'PRIVATE KEY' overrides.yaml
head -5 overrides.yaml
kubectl -n linkerd patch secret linkerd-config-overrides \
-p "{\"data\":{\"linkerd-config-overrides\":\"$(base64 -w0 overrides.yaml)\"}}"
rm overrides.yaml
linkerd upgrade > upgrade-5.yaml && echo upgrade-ok
kubectl apply -f upgrade-5.yaml | grep 'configmap/linkerd-config '
kubectl -n linkerd rollout status deploy --timeout=300sExpected (abridged):
0
identity:
externalCA: true
issuer:
scheme: kubernetes.io/tls
identityTrustAnchorsPEM: |
secret/linkerd-config-overrides patched
upgrade-ok
configmap/linkerd-config configured
The tls: section under issuer: is gone, the
anchor remains. Now linkerd upgrade works again, and the
result no longer contains a key.
Now for the actual payoff – a renewal during live operation.
cmctl triggers it immediately instead of waiting 23
hours:
ARCH=$(dpkg --print-architecture)
curl -fsSLO https://github.com/cert-manager/cmctl/releases/download/v2.6.2/\
cmctl_linux_$ARCH.tar.gz
curl -fsSL https://api.github.com/repos/cert-manager/cmctl/releases/tags/v2.6.2 \
| grep -c "$(sha256sum cmctl_linux_$ARCH.tar.gz | cut -d' ' -f1)"
tar xzf cmctl_linux_$ARCH.tar.gz cmctl
sudo install -m 0755 cmctl /usr/local/bin/cmctl
cmctl renew -n linkerd linkerd-identity-issuer
sleep 15
kubectl -n linkerd get certificate linkerd-identity-issuer \
-o jsonpath='{.status.revision} {.status.notAfter}{"\n"}'
kubectl get events -n linkerd --field-selector reason=IssuerUpdated | tail -1
kubectl -n linkerd get secret linkerd-identity-issuer -o jsonpath='{.data.tls\.crt}' \
| base64 -d | step certificate inspect --format json - \
| jq -r '.extensions.subject_key_id'
kubectl -n notes rollout restart deploy/helper
kubectl -n notes rollout status deploy/helper --timeout=180s
sleep 5
linkerd identity -n notes -l app=helper | grep -A1 'Authority Key' | grep keyidExpected (abridged):
1
Manually triggered issuance of Certificate linkerd/linkerd-identity-issuer
2 2026-11-21T16:02:00Z
15s Normal IssuerUpdated deployment/linkerd-identity Updated identity iss…
ae3e4dc7e07f4657ad799822da17c000b387c4e1
...
keyid:AE:3E:4D:C7:E0:7F:46:57:AD:79:98:22:DA:17:C0:00:B3:87:C4:E1
keyid:AE:3E:4D:C7:E0:7F:46:57:AD:79:98:22:DA:17:C0:00:B3:87:C4:E1
A new key (rotationPolicy: Always),
identity loads it without a restart, and the new
helper proxies carry exactly this key ID as their issuer.
The remaining proxies switch at their next renewal – all issuers chain
to the same anchor.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show until when the trust anchor in the cluster is valid. Target time: 30 s
- Show when cert-manager will next renew the issuer. Target time: 30 s
- Prove by fingerprint that your
ca.crtis in the cluster. Target time: 45 s - Count the
IssuerUpdatedevents. Target time: 20 s
Solution
kubectl -n linkerd get configmap linkerd-identity-trust-roots \
-o jsonpath='{.data.ca-bundle\.crt}' | step certificate inspect --short - | grep to:
kubectl -n linkerd get certificate linkerd-identity-issuer \
-o jsonpath='{.status.notAfter} {.status.renewalTime}{"\n"}'
step certificate fingerprint ca.crt
kubectl -n linkerd get configmap linkerd-identity-trust-roots \
-o jsonpath='{.data.ca-bundle\.crt}' | step certificate fingerprint -
kubectl get events -n linkerd --field-selector reason=IssuerUpdated --no-headers | wc -lExpected (numbers vary):
to: 2036-11-16T15:51:50Z
2026-11-21T16:02:00Z 2026-11-20T15:02:00Z
8b44a80f0940a58823716aecd2aeac2895e43d94bd968826ec821389f65bab81
8b44a80f0940a58823716aecd2aeac2895e43d94bd968826ec821389f65bab81
3
Save your progress? Sign up for free
Checkpoint
linkerd check 2>&1 | tail -1Expected:
Status check results are √
kubectl -n linkerd get certificate linkerd-identity-issuer
kubectl get bundle linkerd-identity-trust-rootsExpected: both True.
ls ~/linkerd/tag05/ca.crt ~/linkerd/tag05/ca.key
kubectl -n notes get podsExpected: both files present, all Pods
2/2 Running.
Challenge
Before you keep ca.crt and ca.key for the
second cluster: prove that the key matches the certificate and that the
issuer in the cluster is signed by exactly this anchor.
Hint: step can extract the public key
from both files.
Solution
kubectl -n linkerd get secret linkerd-identity-issuer -o jsonpath='{.data.tls\.crt}' \
| base64 -d > issuer-cm.crt
step certificate verify issuer-cm.crt --roots ca.crt && echo "Issuer passt zu ca.crt"
diff <(step crypto key public ca.key) <(step certificate key ca.crt) \
&& echo "ca.key passt zu ca.crt"
rm issuer-cm.crtExpected:
Issuer passt zu ca.crt
ca.key passt zu ca.crt
passt zu means “matches”: both checks succeeded.
Save your progress? Sign up for free
Cleanup
The issuer from Round 2 is still valid for a year and chains to the anchor. Its key is in the Secret and in your upgrade manifests – get rid of it:
kubectl -n linkerd patch secret linkerd-identity-issuer --type=json \
-p '[{"op":"remove","path":"/data/crt.pem"},{"op":"remove","path":"/data/key.pem"}]'
kubectl -n linkerd get secret linkerd-identity-issuer -o jsonpath='{.data}' \
| jq -r 'keys[]'
rm -f issuer.key issuer.crt upgrade-*.yaml cmctl
rm -f step_linux_*.tar.gz cmctl_linux_*.tar.gz
rm -rf step_0.31.0Expected:
secret/linkerd-identity-issuer patched
ca.crt
tls.crt
tls.key
ca.crt and ca.key stay in
~/linkerd/tag05/ – Day 12 needs them.
Interview Note
- How do you replace the trust anchor without an outage? Distribute a bundle of old and new, restart everything, swap the issuer, restart, remove the old anchor, restart.
- What does cert-manager automate, and what not? The issuer, completely. trust-manager distributes the anchor – issuing it, replacing it and restarting the proxies remain manual work.
- Why does multicluster need the same anchor? Proxies verify the chain up to the anchor; only with a shared anchor do workloads trust each other across cluster boundaries.
Enterprise Note
- Anchor key not in the cluster: cert-manager with a Vault or external
CA issuer instead of a
CAissuer with a Secret. - Monitor the expiry of anchor and issuer
(
certmanager_certificate_expiration_ timestamp_seconds), not just the proxy certificates. - Anchor switch as a runbook with a maintenance window: every round restarts every Pod in the mesh, single replicas such as databases go down briefly.
- With Helm instead of the CLI:
identity.externalCA=trueandidentity.issuer.scheme=kubernetes.io/tlsbelong in the values from the start (Day 14).