Skip to content
Hands-on IT IT Drills

Day 5: Certificates

Goal: The mesh trusts your own trust anchor with a ten-year lifetime, the switch ran without TLS errors, and cert-manager renews the issuer on its own. · Duration: approx. 90 min · Prerequisite: Day 4

Briefing

  • Three levels: the trust anchor signs the issuer, the issuer (in linkerd-identity) signs the proxies’ 24-hour certificates (Day 3).
  • On Day 1, linkerd install generated a single self-signed certificate that is both: anchor and issuer, valid for one year, without renewal.
  • You replace the anchor in three rounds: a bundle of old and new, swap the issuer, remove the old anchor. After each round, all proxies restart, because they only read the anchors at startup.
  • Afterwards, cert-manager takes over the issuer (48 hours, renewed automatically) and trust-manager the ConfigMap with the anchor.
  • ca.crt and ca.key stay around as files: the second cluster on Day 12 needs the same anchor.
 ca.crt/ca.key (10 years, file + Secret in cert-manager)
   └─ signs ─> linkerd-identity-issuer (48 h, cert-manager)
                    └─ signs ─> proxy certificates (24 h, linkerd-identity)
 trust-manager: ca.crt ─> ConfigMap linkerd/linkerd-identity-trust-roots

Drill 1: Inventory and tooling

mkdir -p ~/linkerd/tag05 && cd ~/linkerd/tag05
kubectl -n linkerd get configmap linkerd-identity-trust-roots \
  -o jsonpath='{.data.ca-bundle\.crt}' > original-trust.crt
openssl x509 -in original-trust.crt -noout -subject -issuer -enddate

Expected:

subject=CN = identity.linkerd.
issuer=CN = identity.linkerd.
notAfter=Nov 19 11:23:11 2027 GMT

Subject equals issuer: self-signed. The same certificate is also in the Secret linkerd-identity-issuer. In one year, the entire mesh would come to a halt.

For your own certificates, you use the step CLI from Smallstep:

ARCH=$(dpkg --print-architecture)
V=0.31.0
curl -fsSLO \
  https://github.com/smallstep/cli/releases/download/v$V/step_linux_${V}_$ARCH.tar.gz
curl -fsSL https://api.github.com/repos/smallstep/cli/releases/tags/v$V \
  | grep -c "$(sha256sum step_linux_${V}_$ARCH.tar.gz | cut -d' ' -f1)"
tar xzf step_linux_${V}_$ARCH.tar.gz
sudo install -m 0755 step_$V/bin/step /usr/local/bin/step
step version

Expected:

1
Smallstep CLI/0.31.0 (linux/amd64)
Release Date: 2026-09-25T19:53:51Z

The 1 means: the checksum of your file is listed in the release – as with the Linkerd CLI on Day 1.

Save your progress? Sign up for free

Drill 2: Your own trust anchor and issuer

step certificate create root.linkerd.cluster.local ca.crt ca.key \
  --profile root-ca --no-password --insecure --not-after 87600h
step certificate create identity.linkerd.cluster.local issuer.crt issuer.key \
  --profile intermediate-ca --not-after 8760h --no-password --insecure \
  --ca ca.crt --ca-key ca.key
chmod 600 ca.key issuer.key
step certificate inspect ca.crt --short
step certificate inspect issuer.crt --short
step certificate verify issuer.crt --roots ca.crt && echo verify-ok

Expected:

Your certificate has been saved in ca.crt.
Your private key has been saved in ca.key.
Your certificate has been saved in issuer.crt.
Your private key has been saved in issuer.key.
X.509v3 Root CA Certificate (ECDSA P-256) [Serial: 6266...5288]
  Subject:     root.linkerd.cluster.local
  Issuer:      root.linkerd.cluster.local
  Valid from:  2026-11-19T15:51:50Z
          to:  2036-11-16T15:51:50Z
X.509v3 Intermediate CA Certificate (ECDSA P-256) [Serial: 1187...8861]
  Subject:     identity.linkerd.cluster.local
  Issuer:      root.linkerd.cluster.local
  Valid from:  2026-11-19T15:51:50Z
          to:  2027-11-19T15:51:50Z
verify-ok

87,600 hours are ten years. The names follow the Linkerd docs: the issuer is called identity.linkerd.<trust-domain>. --no-password --insecure leaves the keys unencrypted – required for the issuer, acceptable for ca.key only in the lab.

Warning: From now on, ca.key is the most valuable key in the mesh. Whoever has it can forge any identity. In production, it is kept offline or in a vault.

Save your progress? Sign up for free

Drill 3: Round 1 – both anchors

Note the start time; in Drill 6 you use it to measure the entire switch.

date +%s > start.txt
step certificate bundle ca.crt original-trust.crt bundle.crt
grep -c 'BEGIN CERTIFICATE' bundle.crt
linkerd upgrade --identity-trust-anchors-file=./bundle.crt > upgrade-1.yaml
kubectl apply -f upgrade-1.yaml | grep -E 'trust-roots|identity '
kubectl -n linkerd rollout status deploy --timeout=300s
linkerd check --proxy 2>&1 | grep -A4 'match CA'

Expected (abridged):

Your certificate has been saved in bundle.crt.
2
configmap/linkerd-identity-trust-roots configured
...
deployment "linkerd-proxy-injector" successfully rolled out
‼ data plane proxies certificate match CA
    Some pods do not have the current trust bundle and must be restarted:
        * linkerd-viz/metrics-api-6cfdd5799-kzwzj
        * linkerd-viz/tap-854d684c76-s554k
...

The Control Plane has already restarted – linkerd upgrade generates new webhook certificates on every run. All other proxies only know both anchors after a restart:

kubectl -n linkerd-viz rollout restart deploy
kubectl -n notes rollout restart deploy
kubectl -n notes rollout restart statefulset
kubectl -n linkerd-viz rollout status deploy --timeout=300s
kubectl -n notes rollout status deploy/notes-api --timeout=300s
kubectl -n notes rollout status statefulset/postgres --timeout=300s
sleep 20
linkerd check --proxy 2>&1 | grep -E '‼|×|Status'

Expected:

Status check results are √

If a ‼ still appears, an old Pod is stuck terminating. Wait and check again. If you have more namespaces with proxies, restart them the same way.

Save your progress? Sign up for free

Drill 4: Round 2 – new issuer

linkerd upgrade --identity-issuer-certificate-file=./issuer.crt \
  --identity-issuer-key-file=./issuer.key > upgrade-2.yaml
kubectl apply -f upgrade-2.yaml | grep -E 'identity-issuer|identity '
kubectl -n linkerd rollout status deploy --timeout=300s
kubectl get events -n linkerd --field-selector reason=IssuerUpdated

Expected (abridged):

secret/linkerd-identity-issuer configured
deployment.apps/linkerd-identity configured
...
LAST SEEN   TYPE     REASON          OBJECT                        MESSAGE
26s         Normal   IssuerUpdated   deployment/linkerd-identity   Updated identity iss…

From now on, identity issues certificates under the new anchor. This works because every proxy has trusted both anchors since Round 1. Restart all proxies again so that they fetch new certificates:

kubectl -n linkerd-viz rollout restart deploy
kubectl -n notes rollout restart deploy
kubectl -n notes rollout restart statefulset
kubectl -n linkerd-viz rollout status deploy --timeout=300s
kubectl -n notes rollout status deploy/notes-api --timeout=300s
kubectl -n notes rollout status statefulset/postgres --timeout=300s
sleep 15
linkerd identity -n notes -l app=web | grep 'Issuer:' | sort | uniq -c

Expected:

      2         Issuer: CN=identity.linkerd.cluster.local

Before, it said CN=identity.linkerd. – the old, self-signed issuer.

Save your progress? Sign up for free

Drill 5: Round 3 – remove the old anchor

linkerd upgrade --identity-trust-anchors-file=./ca.crt > upgrade-3.yaml
kubectl apply -f upgrade-3.yaml | grep trust-roots
kubectl -n linkerd rollout status deploy --timeout=300s
kubectl -n linkerd-viz rollout restart deploy
kubectl -n notes rollout restart deploy
kubectl -n notes rollout restart statefulset
kubectl -n linkerd-viz rollout status deploy --timeout=300s
kubectl -n notes rollout status deploy/notes-api --timeout=300s
kubectl -n notes rollout status statefulset/postgres --timeout=300s
sleep 20
kubectl -n linkerd get configmap linkerd-identity-trust-roots \
  -o jsonpath='{.data.ca-bundle\.crt}' | step certificate inspect --short -
linkerd check --proxy 2>&1 | grep -E '‼|×|Status'

Expected (abridged):

‼ Rotating the trust anchors will affect existing proxies
See https://linkerd.io/2/tasks/rotating_identity_certificates/ for more information
configmap/linkerd-identity-trust-roots configured
...
X.509v3 Root CA Certificate (ECDSA P-256) [Serial: 6266...5288]
  Subject:     root.linkerd.cluster.local
  Issuer:      root.linkerd.cluster.local
  Valid from:  2026-11-19T15:51:50Z
          to:  2036-11-16T15:51:50Z
Status check results are √

The warning from linkerd upgrade refers to exactly the restart you just did. The old anchor is gone.

Warning: If you skip Round 2 and remove the old anchor directly, you take down the mesh: the proxies then hold certificates that nobody trusts anymore.

Save your progress? Sign up for free

Drill 6: The proof

Three rounds of restarts – what did the load generator get hit with? Its log is gone with every restart, but Prometheus has everything:

kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090 \
  >/dev/null 2>&1 &
sleep 4
R=$(( $(date +%s) - $(cat start.txt) ))s
Q="sum by (dst_deployment, status_code) (increase(response_total{namespace=\"notes\",
 deployment=\"web\", direction=\"outbound\"}[$R]))"
curl -s localhost:9090/api/v1/query --data-urlencode "query=$Q" | jq -r \
  '.data.result[] | .metric.dst_deployment + " " + .metric.status_code + " "
   + (.value[1] | tonumber | round | tostring)'

Expected (numbers vary):

helper 200 425
helper 500 156
notes-api 200 1224
notes-api 500 8

helper delivers its usual 30 percent errors. Not a single 502 or 503 – those would be the proxy’s responses to failed TLS. That leaves eight 500s from notes-api. When did they occur?

Q='sum(increase(response_total{namespace="notes",deployment="web",
 direction="outbound",dst_deployment="notes-api",classification="failure"}[1m]))'
curl -s localhost:9090/api/v1/query_range --data-urlencode "query=$Q" \
  --data-urlencode start=$(cat start.txt) --data-urlencode end=$(date +%s) \
  --data-urlencode step=20 \
  | jq -r '.data.result[0].values[] | (.[0]|todate)+" "+.[1]' | awk '$2>0'
Q='max(kube_pod_start_time{namespace="notes",pod="postgres-0"})'
curl -s localhost:9090/api/v1/query_range --data-urlencode "query=$Q" \
  --data-urlencode start=$(cat start.txt) --data-urlencode end=$(date +%s) \
  --data-urlencode step=30 \
  | jq -r '[.data.result[0].values[][1]] | unique | .[] | tonumber | todate'
kill %1

Expected (abridged):

2026-11-19T15:53:17Z 4
2026-11-19T15:53:37Z 4
2026-11-19T15:55:37Z 4.295
2026-11-19T15:55:57Z 4
2026-11-19T15:56:17Z 3.705
2026-11-19T15:41:16Z
2026-11-19T15:52:47Z
2026-11-19T15:55:19Z
2026-11-19T15:57:29Z

The first start time is the restart from Day 4, followed by the three rounds. The errors coincide with the restarts of postgres-0 in Rounds 1 and 2: a single replica that is gone for a few seconds. This has nothing to do with the certificates – you saw the same thing on Day 4 without a switch. The anchor switch itself ran without an outage.

Save your progress? Sign up for free

Drill 7: trust-manager takes over the anchor

cert-manager v1.21.3 is already running in the lab (namespace cert-manager). If you do not have it: helm install cert-manager oci://quay.io/jetstack/charts/cert-manager --version v1.21.3 -n cert-manager --create-namespace --set crds.enabled=true.

helm install trust-manager oci://quay.io/jetstack/charts/trust-manager \
  --version v0.26.0 -n cert-manager --set app.trust.namespace=cert-manager --wait
kubectl -n cert-manager create secret tls linkerd-trust-anchor \
  --cert=ca.crt --key=ca.key
cat > cert-manager.yaml <<'EOF'
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata: {name: linkerd-identity-issuer}
spec:
  ca: {secretName: linkerd-trust-anchor}
---
apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata: {name: linkerd-identity-trust-roots}
spec:
  sources:
  - secret: {name: linkerd-trust-anchor, key: tls.crt}
  target:
    configMap: {key: ca-bundle.crt}
    namespaceSelector:
      matchLabels: {linkerd.io/is-control-plane: "true"}
EOF
kubectl apply -f cert-manager.yaml
sleep 10
kubectl get clusterissuer linkerd-identity-issuer
kubectl get bundle linkerd-identity-trust-roots
kubectl -n linkerd get configmap linkerd-identity-trust-roots \
  -o jsonpath='{.metadata.ownerReferences[0].kind}{"\n"}'

Expected (abridged):

STATUS: deployed
secret/linkerd-trust-anchor created
clusterissuer.cert-manager.io/linkerd-identity-issuer created
bundle.trust.cert-manager.io/linkerd-identity-trust-roots created
NAME                      READY   AGE
linkerd-identity-issuer   True    11s
NAME                           CONFIGMAP TARGET   SECRET TARGET   SYNCED   REASON   AGE
linkerd-identity-trust-roots   ca-bundle.crt                      True     Synced   11s
Bundle
  • The anchor goes as a Secret into cert-manager. trust-manager may only read from its trust namespace, and the ClusterIssuer also looks for its Secrets there.
  • The Bundle name is the name of the target ConfigMap. trust-manager has taken over the existing ConfigMap; the content remains the same anchor, no restart needed.
  • linkerd install has already set the label linkerd.io/is-control-plane=true.

Save your progress? Sign up for free

Drill 8: cert-manager takes over the issuer

cat > identity-issuer.yaml <<'EOF'
apiVersion: cert-manager.io/v1
kind: Certificate
metadata: {name: linkerd-identity-issuer, namespace: linkerd}
spec:
  issuerRef: {name: linkerd-identity-issuer, kind: ClusterIssuer}
  secretName: linkerd-identity-issuer
  isCA: true
  commonName: identity.linkerd.cluster.local
  duration: 48h0m0s
  renewBefore: 25h0m0s
  privateKey: {rotationPolicy: Always, algorithm: ECDSA}
EOF
kubectl apply -f identity-issuer.yaml
sleep 15
kubectl -n linkerd get certificate
kubectl -n linkerd get secret linkerd-identity-issuer -o jsonpath='{.data}' \
  | jq -r 'keys[]'

Expected:

certificate.cert-manager.io/linkerd-identity-issuer created
NAME                      READY   SECRET                    AGE
linkerd-identity-issuer   True    linkerd-identity-issuer   15s
ca.crt
crt.pem
key.pem
tls.crt
tls.key

cert-manager writes into the existing Secret: tls.* are added, the old crt.pem/key.pem from Round 2 remain. Now you switch Linkerd to the external issuer – only then does identity read the tls.* keys:

linkerd upgrade --set identity.externalCA=true \
  --set identity.issuer.scheme=kubernetes.io/tls > upgrade-4.yaml
grep -c 'name: linkerd-identity-issuer$' upgrade-4.yaml
kubectl apply -f upgrade-4.yaml | grep 'identity '
kubectl -n linkerd rollout status deploy --timeout=300s
kubectl -n linkerd logs deploy/linkerd-identity -c identity | grep 'Issuer cert loaded'
linkerd check 2>&1 | grep -A1 '‼'

Expected (abridged):

0
deployment.apps/linkerd-identity configured
...
level=info msg="Issuer cert loaded" invalid_after=1795276812 ... ttl_seconds=172762
‼ issuer cert is valid for at least 60 days
    issuer certificate will expire on 2026-11-21T16:00:12Z

The manifest no longer contains the issuer Secret or the trust ConfigMap – both now belong to cert-manager and trust-manager. The warning about the 60-day period is intended for a 48-hour issuer; linkerd check still ends with √.

A pitfall awaits at the next linkerd upgrade – so right away:

linkerd upgrade > /dev/null

Expected (wrapped):

--identity-issuer-certificate-file must not be specified if
--identity-external-issuer=true

The CLI remembers all values from earlier upgrades in the Secret linkerd-config-overrides – including the certificate and private key of the issuer from Round 2. Both have to go:

(umask 077; kubectl -n linkerd get secret linkerd-config-overrides \
  -o jsonpath='{.data.linkerd-config-overrides}' | base64 -d > overrides.yaml)
sed -i '/^    tls:/,/^identityTrustAnchorsPEM/{/^identityTrustAnchorsPEM/!d}' \
  overrides.yaml
grep -c 'PRIVATE KEY' overrides.yaml
head -5 overrides.yaml
kubectl -n linkerd patch secret linkerd-config-overrides \
  -p "{\"data\":{\"linkerd-config-overrides\":\"$(base64 -w0 overrides.yaml)\"}}"
rm overrides.yaml
linkerd upgrade > upgrade-5.yaml && echo upgrade-ok
kubectl apply -f upgrade-5.yaml | grep 'configmap/linkerd-config '
kubectl -n linkerd rollout status deploy --timeout=300s

Expected (abridged):

0
identity:
  externalCA: true
  issuer:
    scheme: kubernetes.io/tls
identityTrustAnchorsPEM: |
secret/linkerd-config-overrides patched
upgrade-ok
configmap/linkerd-config configured

The tls: section under issuer: is gone, the anchor remains. Now linkerd upgrade works again, and the result no longer contains a key.

Now for the actual payoff – a renewal during live operation. cmctl triggers it immediately instead of waiting 23 hours:

ARCH=$(dpkg --print-architecture)
curl -fsSLO https://github.com/cert-manager/cmctl/releases/download/v2.6.2/\
cmctl_linux_$ARCH.tar.gz
curl -fsSL https://api.github.com/repos/cert-manager/cmctl/releases/tags/v2.6.2 \
  | grep -c "$(sha256sum cmctl_linux_$ARCH.tar.gz | cut -d' ' -f1)"
tar xzf cmctl_linux_$ARCH.tar.gz cmctl
sudo install -m 0755 cmctl /usr/local/bin/cmctl
cmctl renew -n linkerd linkerd-identity-issuer
sleep 15
kubectl -n linkerd get certificate linkerd-identity-issuer \
  -o jsonpath='{.status.revision} {.status.notAfter}{"\n"}'
kubectl get events -n linkerd --field-selector reason=IssuerUpdated | tail -1
kubectl -n linkerd get secret linkerd-identity-issuer -o jsonpath='{.data.tls\.crt}' \
  | base64 -d | step certificate inspect --format json - \
  | jq -r '.extensions.subject_key_id'
kubectl -n notes rollout restart deploy/helper
kubectl -n notes rollout status deploy/helper --timeout=180s
sleep 5
linkerd identity -n notes -l app=helper | grep -A1 'Authority Key' | grep keyid

Expected (abridged):

1
Manually triggered issuance of Certificate linkerd/linkerd-identity-issuer
2 2026-11-21T16:02:00Z
15s         Normal   IssuerUpdated   deployment/linkerd-identity   Updated identity iss…
ae3e4dc7e07f4657ad799822da17c000b387c4e1
...
                keyid:AE:3E:4D:C7:E0:7F:46:57:AD:79:98:22:DA:17:C0:00:B3:87:C4:E1
                keyid:AE:3E:4D:C7:E0:7F:46:57:AD:79:98:22:DA:17:C0:00:B3:87:C4:E1

A new key (rotationPolicy: Always), identity loads it without a restart, and the new helper proxies carry exactly this key ID as their issuer. The remaining proxies switch at their next renewal – all issuers chain to the same anchor.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show until when the trust anchor in the cluster is valid. Target time: 30 s
  2. Show when cert-manager will next renew the issuer. Target time: 30 s
  3. Prove by fingerprint that your ca.crt is in the cluster. Target time: 45 s
  4. Count the IssuerUpdated events. Target time: 20 s

Solution

kubectl -n linkerd get configmap linkerd-identity-trust-roots \
  -o jsonpath='{.data.ca-bundle\.crt}' | step certificate inspect --short - | grep to:
kubectl -n linkerd get certificate linkerd-identity-issuer \
  -o jsonpath='{.status.notAfter} {.status.renewalTime}{"\n"}'
step certificate fingerprint ca.crt
kubectl -n linkerd get configmap linkerd-identity-trust-roots \
  -o jsonpath='{.data.ca-bundle\.crt}' | step certificate fingerprint -
kubectl get events -n linkerd --field-selector reason=IssuerUpdated --no-headers | wc -l

Expected (numbers vary):

          to:  2036-11-16T15:51:50Z
2026-11-21T16:02:00Z 2026-11-20T15:02:00Z
8b44a80f0940a58823716aecd2aeac2895e43d94bd968826ec821389f65bab81
8b44a80f0940a58823716aecd2aeac2895e43d94bd968826ec821389f65bab81
3

Save your progress? Sign up for free

Checkpoint

linkerd check 2>&1 | tail -1

Expected:

Status check results are √
kubectl -n linkerd get certificate linkerd-identity-issuer
kubectl get bundle linkerd-identity-trust-roots

Expected: both True.

ls ~/linkerd/tag05/ca.crt ~/linkerd/tag05/ca.key
kubectl -n notes get pods

Expected: both files present, all Pods 2/2 Running.

Challenge

Before you keep ca.crt and ca.key for the second cluster: prove that the key matches the certificate and that the issuer in the cluster is signed by exactly this anchor.

Hint: step can extract the public key from both files.

Solution

kubectl -n linkerd get secret linkerd-identity-issuer -o jsonpath='{.data.tls\.crt}' \
  | base64 -d > issuer-cm.crt
step certificate verify issuer-cm.crt --roots ca.crt && echo "Issuer passt zu ca.crt"
diff <(step crypto key public ca.key) <(step certificate key ca.crt) \
  && echo "ca.key passt zu ca.crt"
rm issuer-cm.crt

Expected:

Issuer passt zu ca.crt
ca.key passt zu ca.crt

passt zu means “matches”: both checks succeeded.

Save your progress? Sign up for free

Cleanup

The issuer from Round 2 is still valid for a year and chains to the anchor. Its key is in the Secret and in your upgrade manifests – get rid of it:

kubectl -n linkerd patch secret linkerd-identity-issuer --type=json \
  -p '[{"op":"remove","path":"/data/crt.pem"},{"op":"remove","path":"/data/key.pem"}]'
kubectl -n linkerd get secret linkerd-identity-issuer -o jsonpath='{.data}' \
  | jq -r 'keys[]'
rm -f issuer.key issuer.crt upgrade-*.yaml cmctl
rm -f step_linux_*.tar.gz cmctl_linux_*.tar.gz
rm -rf step_0.31.0

Expected:

secret/linkerd-identity-issuer patched
ca.crt
tls.crt
tls.key

ca.crt and ca.key stay in ~/linkerd/tag05/ – Day 12 needs them.

Interview Note

  • How do you replace the trust anchor without an outage? Distribute a bundle of old and new, restart everything, swap the issuer, restart, remove the old anchor, restart.
  • What does cert-manager automate, and what not? The issuer, completely. trust-manager distributes the anchor – issuing it, replacing it and restarting the proxies remain manual work.
  • Why does multicluster need the same anchor? Proxies verify the chain up to the anchor; only with a shared anchor do workloads trust each other across cluster boundaries.

Enterprise Note

  • Anchor key not in the cluster: cert-manager with a Vault or external CA issuer instead of a CA issuer with a Secret.
  • Monitor the expiry of anchor and issuer (certmanager_certificate_expiration_ timestamp_seconds), not just the proxy certificates.
  • Anchor switch as a runbook with a maintenance window: every round restarts every Pod in the mesh, single replicas such as databases go down briefly.
  • With Helm instead of the CLI: identity.externalCA=true and identity.issuer.scheme=kubernetes.io/tls belong in the values from the start (Day 14).