Skip to content
Hands-on IT IT Drills

Appendix A: Cheat Sheet

The most important commands and snippets from Days 0 to 18, sorted by topic. The days are in parentheses. You replace placeholders in angle brackets; IPs are those of the lab (cp1 172.16.0.126, west 172.16.0.182). Values like lab-key-123 are lab values. All commands run on cp1; without --context they target the cluster ost.

Measuring (Days 0, 4)

Success rate from the load generator – the measuring instrument of the whole book:

kubectl -n notes logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
  | sort | uniq -c
kubectl -n notes logs deploy/last --tail=300 | awk '{print $2, $3}' | sort | uniq -c
kubectl --context west -n tag15 logs deploy/last -c last \
  --since-time=$(cat start.txt) | awk '{print $2, $3}' | sort | uniq -c

Query Prometheus from cp1:

kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090 \
  >/dev/null 2>&1 &
Q='sum by (dst_deployment) (rate(response_total{namespace="notes",
 deployment="web", direction="outbound", classification="success"}[5m]))
 / sum by (dst_deployment) (rate(response_total{namespace="notes",
 deployment="web", direction="outbound"}[5m]))'
curl -s localhost:9090/api/v1/query --data-urlencode "query=$Q" \
  | jq -r '.data.result[] | .metric.dst_deployment + " " + .value[1]'
kill %1

Installation and check (Day 1)

Fetch the CLI architecture-neutrally and verify it against the GitHub API digest:

ARCH=$(dpkg --print-architecture)
V=edge-26.11.2
curl -fsSLO \
  https://github.com/linkerd/linkerd2/releases/download/$V/linkerd2-cli-$V-linux-$ARCH
sha256sum linkerd2-cli-$V-linux-$ARCH
curl -fsSL https://api.github.com/repos/linkerd/linkerd2/releases/tags/$V \
  | grep -o '"digest": "sha256:[0-9a-f]*"'
sudo install -m 0755 linkerd2-cli-$V-linux-$ARCH /usr/local/bin/linkerd
linkerd version --client

CRDs first, then the Control Plane:

linkerd check --pre
kubectl get crd httproutes.gateway.networking.k8s.io -o \
  jsonpath='{.metadata.annotations.gateway\.networking\.k8s\.io/bundle-version}{"\n"}'
linkerd install --crds > crds.yaml
kubectl apply -f crds.yaml
linkerd install > control-plane.yaml
kubectl apply -f control-plane.yaml
kubectl -n linkerd rollout status deploy --timeout=300s
linkerd check
linkerd version --short
linkerd install --ha --ignore-cluster > ha.yaml   # for comparison only
Command Checks Day
linkerd check --pre cluster before installation 1
linkerd check Control Plane, certificates, webhooks 1, 5
linkerd check --proxy -n <ns> proxies of a namespace 2, 6
linkerd check --proxy -n <ns> --wait 30s the same, without hanging for 5 min 16
linkerd viz check viz extension 4
linkerd multicluster check extension and links 12
  • Linkerd does not ship the Gateway API CRDs (in the lab: Cilium or k3s).
  • The Control Plane: linkerd-identity (CA), linkerd-destination (discovery and policy), linkerd-proxy-injector (webhook). The proxy runs as a native sidecar (init container with restartPolicy: Always).

Injection (Day 2)

kubectl annotate namespace notes linkerd.io/inject=enabled
kubectl -n notes rollout restart deploy/notes-api deploy/helper deploy/web deploy/last
kubectl -n notes rollout restart statefulset/postgres
kubectl -n notes get pods                     # 2/2 = proxy present
kubectl -n notes get pods -o custom-columns=\
NAME:.metadata.name,PROXY:.metadata.annotations.linkerd\\.io/proxy-version
linkerd version --proxy -n notes
kubectl -n notes annotate service postgres config.linkerd.io/opaque-ports=5432
linkerd inject plain.yaml                     # only set the annotation
linkerd inject --manual plain.yaml            # insert the proxy container directly
kubectl -n tag02 patch deploy ohne -p \
  '{"spec":{"template":{"metadata":{"annotations":{"linkerd.io/inject":"disabled"}}}}}'

One identity per service – via dedicated ServiceAccounts:

for a in web notes-api helper last postgres; do
  kubectl -n notes create serviceaccount $a
done
for d in web notes-api helper last; do
  kubectl -n notes set serviceaccount deploy/$d $d
done
kubectl -n notes set serviceaccount statefulset/postgres postgres
kubectl -n notes logs deploy/web -c linkerd-proxy | grep 'Local identity'
  • The namespace annotation only affects new Pods: rollout restart.
  • With the native sidecar, init containers run in the mesh. With config.beta.linkerd.io/proxy-enable-native-sidecar: "false" they run before the proxy and thus without mTLS – under default-deny they fail (Day 6).
  • Server-first protocols (SMTP, MySQL) and Postgres need opaque-ports; otherwise, with server-first, protocol detection waits 10 s (Day 16).

Identity and mTLS (Day 3)

linkerd identity -n notes -l app=web | grep -E 'POD|Issuer|Not |Subject:|DNS:'
kubectl -n notes logs deploy/web -c linkerd-proxy | grep 'Certified identity'
kubectl -n linkerd logs deploy/linkerd-identity -c identity | grep issued | tail -1
kubectl -n linkerd get configmap linkerd-config -o jsonpath='{.data.values}' \
  | grep -E 'issuanceLifetime|clockSkewAllowance'
linkerd diagnostics proxy-metrics -n notes deploy/web | grep '^control_identity_cert'

Who talks mTLS, who talks plaintext (client_id, no_tls_reason):

linkerd diagnostics proxy-metrics -n notes pod/postgres-0 \
  | grep '^tcp_open_total{direction="inbound",peer="src"' \
  | grep -oE '(target_port|client_id|no_tls_reason)="[^"]*"|[0-9]+$' | paste - - -

Look on the wire (ephemeral container with tcpdump):

P=$(kubectl -n notes get pod -l app=notes-api -o jsonpath='{.items[0].metadata.name}')
kubectl -n notes debug $P --image=docker.io/nicolaka/netshoot:v0.15 \
  --profile=netadmin -c sniff -- sleep 1800
kubectl -n notes exec $P -c sniff -- timeout 10 \
  tcpdump -i eth0 -nn -A 'tcp port 4143' 2>/dev/null | grep -c 'lab-key'
  • Meshed traffic arrives on port 4143, TLS 1.3, SNI = target identity.
  • Identity: <sa>.<ns>.serviceaccount.identity.linkerd.cluster.local.
  • kubectl debug only works on Pods, not on deploy/….

viz and Grafana (Day 4)

P=http://kps-kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090
linkerd viz install --set prometheus.enabled=false --set prometheusUrl=$P \
  --set grafana.url=kps-grafana.monitoring.svc.cluster.local > viz.yaml
kubectl apply -f viz.yaml
linkerd viz check
Command Shows Day
linkerd viz stat -n notes deploy success rate, RPS, latency 4
linkerd viz stat -n notes deploy/helper --from deploy/web one edge 4
linkerd viz tap -n notes deploy/helper --path /unstable live requests 4
linkerd viz top -n notes deploy/web most frequent paths (needs a TTY) 4
linkerd viz edges -n notes deploy edges with identities 4
linkerd viz routes -n notes deploy/web routes 4
linkerd viz authz -n notes deploy/helper hits per Server/rule 6
linkerd viz stat-outbound -n tag07 deploy/web $PROM per route and backend 7
linkerd viz dashboard --show url web UI 4
ssh -L 50750:localhost:50750 [email protected] linkerd viz dashboard --show url
PROM=--prometheusURL=http://localhost:9090    # stat-outbound/-inbound via port-forward
kubectl -n monitoring create configmap linkerd-deployment-dashboard \
  --from-file=linkerd-deployment.json
kubectl -n monitoring label configmap linkerd-deployment-dashboard grafana_dashboard=1
  • Without prometheusUrl, viz install aborts when prometheus.enabled=false.
  • Proxy metrics reach Prometheus via the PodMonitor linkerd-proxy (port linkerd-admin, 4191).
  • tap needs restarted Pods (viz.linkerd.io/tap-enabled) and does not show denied (403) requests.

Certificates (Day 5)

Trust anchor (10 years) and issuer (1 year) with step:

step certificate create root.linkerd.cluster.local ca.crt ca.key \
  --profile root-ca --no-password --insecure --not-after 87600h
step certificate create identity.linkerd.cluster.local issuer.crt issuer.key \
  --profile intermediate-ca --not-after 8760h --no-password --insecure \
  --ca ca.crt --ca-key ca.key
step certificate verify issuer.crt --roots ca.crt && echo verify-ok
step certificate fingerprint ca.crt
kubectl -n linkerd get configmap linkerd-identity-trust-roots \
  -o jsonpath='{.data.ca-bundle\.crt}' | step certificate inspect --short -

Anchor rotation in three rounds; after each round, restart CP, viz and workloads:

step certificate bundle ca.crt original-trust.crt bundle.crt
linkerd upgrade --identity-trust-anchors-file=./bundle.crt > upgrade-1.yaml
linkerd upgrade --identity-issuer-certificate-file=./issuer.crt \
  --identity-issuer-key-file=./issuer.key > upgrade-2.yaml
linkerd upgrade --identity-trust-anchors-file=./ca.crt > upgrade-3.yaml
kubectl get events -n linkerd --field-selector reason=IssuerUpdated

cert-manager and trust-manager take over:

helm install trust-manager oci://quay.io/jetstack/charts/trust-manager \
  --version v0.26.0 -n cert-manager --set app.trust.namespace=cert-manager --wait
kubectl -n cert-manager create secret tls linkerd-trust-anchor \
  --cert=ca.crt --key=ca.key
linkerd upgrade --set identity.externalCA=true \
  --set identity.issuer.scheme=kubernetes.io/tls > upgrade-4.yaml
cmctl renew -n linkerd linkerd-identity-issuer
kubectl -n linkerd get certificate linkerd-identity-issuer \
  -o jsonpath='{.status.notAfter} {.status.renewalTime}{"\n"}'
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata: {name: linkerd-identity-issuer}
spec:
  ca: {secretName: linkerd-trust-anchor}
---
apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata: {name: linkerd-identity-trust-roots}
spec:
  sources:
  - secret: {name: linkerd-trust-anchor, key: tls.crt}
  target:
    configMap: {key: ca-bundle.crt}
    namespaceSelector:
      matchLabels: {linkerd.io/is-control-plane: "true"}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata: {name: linkerd-identity-issuer, namespace: linkerd}
spec:
  issuerRef: {name: linkerd-identity-issuer, kind: ClusterIssuer}
  secretName: linkerd-identity-issuer
  isCA: true
  commonName: identity.linkerd.cluster.local
  duration: 48h0m0s
  renewBefore: 25h0m0s
  privateKey: {rotationPolicy: Always, algorithm: ECDSA}
  • Keep ca.key safe: Days 12 and 14 use it to sign the issuers for west.
  • If linkerd upgrade fails after switching to externalCA, the old issuer values are still in linkerd-config-overrides – remove them, then try again.
  • The warning “issuer cert is valid for at least 60 days” is expected with 48 h.

Policies (Day 6)

apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata: {name: notes-api-http, namespace: notes}
spec:
  podSelector: {matchLabels: {app: notes-api}}
  port: http
  proxyProtocol: HTTP/1
  accessPolicy: deny          # audit to observe, all-unauthenticated (Day 8)
---
apiVersion: policy.linkerd.io/v1alpha1
kind: MeshTLSAuthentication
metadata: {name: web, namespace: notes}
spec:
  identityRefs: [{kind: ServiceAccount, name: web}]
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: notes-api-von-web, namespace: notes}
spec:
  targetRef: {group: policy.linkerd.io, kind: Server, name: notes-api-http}
  requiredAuthenticationRefs:
  - {group: policy.linkerd.io, kind: MeshTLSAuthentication, name: web}

For clients without a proxy (Prometheus, kubelet, Cilium gateway), by network:

apiVersion: policy.linkerd.io/v1alpha1
kind: NetworkAuthentication
metadata: {name: pod-netz, namespace: notes}
spec:
  networks:
  - cidr: 10.0.0.0/16
kubectl annotate namespace notes config.linkerd.io/default-inbound-policy=deny
kubectl -n notes get server,authorizationpolicy,meshtlsauthentication,\
networkauthentication
kubectl -n notes get pod -l app=helper -o yaml | grep -m1 -A1 INBOUND_DEFAULT_POLICY
linkerd diagnostics policy -n notes po/<pod> 8080 -o json \
  | jq -r '.authorizations[].labels.name'
kubectl -n notes logs deploy/helper -c linkerd-proxy --since=2m \
  | grep -m1 'Request denied' | grep -oE 'server.name=[^ ]*|Name\("[^"]*'
linkerd diagnostics proxy-metrics -n notes po/postgres-0 \
  | grep '^inbound_tcp_authz_deny_total' | grep -oE '(client_id|tls)="[^"]*"'
  • Procedure: Server with audit, create rules, evaluate the audit, switch to deny, finally default-deny for the namespace (restart the Pods).
  • Default-deny also blocks Prometheus on 4191: allow a Server proxy-admin (podSelector: {}, port linkerd-admin) with a NetworkAuthentication.
  • Without an HTTPRoute on the Server, probes are allowed automatically. Once a route is attached to the Server, the probes need their own route plus a rule for the kubelet IPs.
  • Denied: HTTP with 403, opaque with a closed connection.

Routes, retries and timeouts (Day 7)

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: helper-unstable
  namespace: tag07
  annotations:
    retry.linkerd.io/http: 5xx
    retry.linkerd.io/limit: "2"
spec:
  parentRefs:
  - {group: core, kind: Service, name: helper, port: 80}
  rules:
  - matches:
    - path: {type: PathPrefix, value: /unstable}
      method: GET             # don't retry POST
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: helper-delay
  namespace: tag07
  annotations:
    timeout.linkerd.io/request: 1s
spec:
  parentRefs:
  - {group: core, kind: Service, name: helper, port: 80}
  rules:
  - matches:
    - path: {type: PathPrefix, value: /delay}
kubectl -n tag07 get httproute helper-unstable -o yaml | grep -E 'controllerName|reason'
kubectl -n tag07 annotate httproute helper-unstable retry.linkerd.io/limit=3 --overwrite
kubectl -n tag07 patch httproute helper-delay --type=json \
  -p '[{"op":"add","path":"/spec/rules/0/timeouts","value":{"request":"3s"}}]'
linkerd diagnostics proxy-metrics -n tag07 deploy/last \
  | grep -E '^outbound_http_route_retry_(requests|limit_exceeded)_total' \
  | sed -E 's/\{.*\}//'
  • First route on the Service: all other paths get 404. Create a catch-all route with PathPrefix /.
  • Retries also apply to POST if the route does not restrict the method.
  • No retry budget, only limit; a ServiceProfile for the same Service overrides the annotations. The timeout annotation beats rules[].timeouts.request.
  • A request timeout is not retried; retry.linkerd.io/timeout makes slow attempts retryable.

Circuit breaking and rate limiting (Day 8)

kubectl -n tag08 annotate service helper \
  balancer.linkerd.io/failure-accrual=consecutive \
  balancer.linkerd.io/failure-accrual-consecutive-max-failures=3 \
  balancer.linkerd.io/failure-accrual-consecutive-min-penalty=10s \
  balancer.linkerd.io/failure-accrual-consecutive-max-penalty=1m
linkerd viz stat -n tag08 po -l app=helper --from deploy/web
linkerd diagnostics proxy-metrics -n tag08 deploy/web \
  | grep '^outbound_http_balancer_endpoints.*pending' | grep -oE '[0-9]+$'
apiVersion: policy.linkerd.io/v1alpha1
kind: HTTPLocalRateLimitPolicy
metadata: {name: helper-rl, namespace: tag08}
spec:
  targetRef: {group: policy.linkerd.io, kind: Server, name: helper-http}
  total:
    requestsPerSecond: 20
  identity:
    requestsPerSecond: 1
  overrides:
  - requestsPerSecond: 4
    clientRefs:
    - {kind: ServiceAccount, name: web}
kubectl -n tag08 get httplocalratelimitpolicy
kubectl -n tag08 logs deploy/helper -c linkerd-proxy --since=1m \
  | grep -o 'total rate limit exceeded: [0-9]*rps' | tail -1
linkerd diagnostics proxy-metrics -n tag08 deploy/helper \
  | grep '^inbound_http_local_ratelimit' \
  | grep -oE 'ratelimit_scope="[^"]*"|client_id="[a-z]+|\} [0-9]+' | paste - - -
  • Failure accrual blocks individual Pods; if all are blocked, you get 503.
  • Rate limits apply per Pod, response 429. Requests denied via identity also consume the total budget – choose total generously.

Traffic shifting (Day 9)

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: notes-api-canary, namespace: tag09}
spec:
  parentRefs:
  - {group: "", kind: Service, name: notes-api, port: 80}
  rules:
  - matches:
    - headers: [{name: x-canary, value: "true"}]
    backendRefs:
    - {name: notes-api-v2, port: 80}
  - backendRefs:
    - {name: notes-api, port: 80, weight: 90}
    - {name: notes-api-v2, port: 80, weight: 10}
W=/spec/rules/0/backendRefs
kubectl -n tag09 patch httproute notes-api-canary --type=json -p \
  "[{\"op\":\"replace\",\"path\":\"$W/0/weight\",\"value\":50},
    {\"op\":\"replace\",\"path\":\"$W/1/weight\",\"value\":50}]"
kubectl -n tag09 get httproute notes-api-canary \
  -o jsonpath='{.spec.rules[*].backendRefs[*].weight}{"\n"}'
kubectl -n tag09 delete httproute notes-api-canary      # rollback in seconds
linkerd viz stat -n tag09 deploy --from deploy/last
kubectl -n tag09 delete endpointslice -l io.cilium.gateway/owning-gateway=notes-api
  • Linkerd and Cilium both accept the route with a Service parent: Linkerd splits meshed clients, Cilium Envoy splits unmeshed clients (server: envoy).
  • Rule without backendRefs: Linkerd uses the parent Service, Cilium’s Envoy answers unmeshed clients with 500 (Day 18, Round 1).
  • To finish: Service selector to v2, delete the route, scale down v1.

Ingress (Day 10)

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata: {name: notes, namespace: tag10}
spec:
  gatewayClassName: cilium
  listeners:
  - {name: http, protocol: HTTP, port: 80}
  - name: https
    protocol: HTTPS
    port: 443
    hostname: notes.lab.internal
    tls:
      mode: Terminate
      certificateRefs: [{name: notes-tls}]
NP=$(kubectl -n tag10 get svc cilium-gateway-notes \
  -o jsonpath='{.spec.ports[?(@.port==443)].nodePort}')
curl -s -w ' %{http_code}\n' --cacert lab-ca.crt \
  --resolve notes.lab.internal:$NP:172.16.0.126 https://notes.lab.internal:$NP/api/notes
kubectl get ciliumnodes \
  -o jsonpath='{range .items[*]}{.metadata.name} {.spec.ingress.ipv4}{"\n"}{end}'
cmctl renew -n tag10 notes-tls
Entry point Mesh view Rule in the policy
Cilium gateway no_tls_from_remote, ingress IPs of the nodes NetworkAuthentication
nginx edge in the mesh mTLS with identity edge MeshTLSAuthentication

Egress (Day 11)

apiVersion: policy.linkerd.io/v1alpha1
kind: EgressNetwork
metadata: {name: internet, namespace: tag11}
spec:
  trafficPolicy: Deny
---
apiVersion: gateway.networking.k8s.io/v1
kind: TLSRoute
metadata: {name: example-com, namespace: tag11}
spec:
  parentRefs:
  - {group: policy.linkerd.io, kind: EgressNetwork, name: internet, port: 443}
  hostnames: [example.com]
  rules:
  - backendRefs:
    - {group: policy.linkerd.io, kind: EgressNetwork, name: internet}
kubectl -n tag11 patch egressnetwork internet --type=merge \
  -p '{"spec":{"trafficPolicy":"Deny"}}'
kubectl -n tag11 patch tlsroute example-com --type=json \
  -p '[{"op":"add","path":"/spec/hostnames/-","value":"linkerd.io"}]'
kubectl -n tag11 get httproute,tlsroute
  • Without a TLSRoute, the proxy sees HTTPS only as tcp with an IP; with a TLSRoute, as tls with SNI. HTTPRoutes on the EgressNetwork filter plaintext HTTP by method and path.
  • Hostnames in metrics: config.linkerd.io/proxy-metrics-hostname-labels: "true".
  • Policy only affects new connections: restart keep-alive clients after Deny.
  • Unmeshed Pods bypass the egress policy.

Multicluster (Days 12, 13)

Second cluster with a shared trust anchor:

step certificate create identity.linkerd.cluster.local \
  west-issuer.crt west-issuer.key \
  --profile intermediate-ca --not-after 8760h --no-password --insecure \
  --ca ../tag05/ca.crt --ca-key ../tag05/ca.key
linkerd --context west install --crds | kubectl --context west apply -f -
linkerd --context west install \
  --identity-trust-anchors-file ../tag05/ca.crt \
  --identity-issuer-certificate-file west-issuer.crt \
  --identity-issuer-key-file west-issuer.key > west-cp.yaml
kubectl --context west apply -f west-cp.yaml

Extension, link, export:

linkerd --context west multicluster install > mc-west.yaml
kubectl --context west apply -f mc-west.yaml
linkerd --context ost multicluster install -f mc-ost-values.yaml > mc-ost.yaml
kubectl --context ost apply -f mc-ost.yaml
linkerd --context west multicluster link-gen --cluster-name west > link-west.yaml
kubectl --context ost apply -f link-west.yaml
kubectl --context west -n tag12 label service helper mirror.linkerd.io/exported=true
linkerd --context ost multicluster gateways
linkerd --context ost viz stat -n tag12 deploy/last --to svc/helper-west
kubectl --context ost get svc -A -l mirror.linkerd.io/mirrored-service=true
gateway:
  enabled: false              # ost: no gateway, controller only
controllers:
- link:
    ref:
      name: west
Label on the Service (source cluster) Result in the target cluster Day
mirror.linkerd.io/exported=true <svc>-west, traffic via gateway 12
mirror.linkerd.io/exported=remote-discovery <svc>-west, directly to Pod IPs 13
mirror.linkerd.io/federated=member <svc>-federated from all clusters 13
linkerd --context ost diagnostics endpoints echo-federated.tag13.svc.cluster.local:80
kubectl --context ost -n linkerd-multicluster delete link west
kubectl --context ost -n linkerd-multicluster delete secret cluster-credentials-west
kubectl --context ost -n linkerd delete secret cluster-credentials-west
  • The mirror namespace must exist in the reading cluster.
  • NodePort gateway: set controllers[].gateway.probe.port to the probe NodePort before the link is created.
  • remote-discovery and federated need a flat network (Pod IPs reachable).
  • linkerd multicluster unlink no longer exists.
  • Gateway gone: the client proxy puts the backend into failfast after ~3 s, the controller empties the endpoints after ~9 s. The return takes longer (backoff in the controller’s proxy, up to 60 s between two attempts).

Helm, HA and upgrade (Days 14, 15)

helm repo add linkerd-edge https://helm.linkerd.io/edge
helm search repo linkerd-edge/linkerd-control-plane --versions | head -4
kubectl --context west -n linkerd create configmap linkerd-identity-trust-roots \
  --from-file=ca-bundle.crt=../tag05/ca.crt
kubectl --context west -n linkerd create secret generic linkerd-identity-issuer \
  --type=kubernetes.io/tls --from-file=tls.crt=issuer.crt \
  --from-file=tls.key=issuer.key --from-file=ca.crt=../tag05/ca.crt
helm --kube-context west install linkerd-crds linkerd-edge/linkerd-crds \
  --version 2026.11.1 -n linkerd
helm --kube-context west install linkerd-control-plane \
  linkerd-edge/linkerd-control-plane --version 2026.11.1 -n linkerd \
  -f values-ha.yaml -f values-west.yaml
identity:
  externalCA: true            # trust anchor from the ConfigMap
  issuer:
    scheme: kubernetes.io/tls # issuer from the Secret
controllerReplicas: 2
enablePodAntiAffinity: false  # single node only
identityTrustAnchorsPEM: |    # for multicluster check
  -----BEGIN CERTIFICATE-----
  ...

Upgrade: release notes, preview, CRDs, Control Plane, Data Plane:

diff <(helm show values linkerd-edge/linkerd-control-plane --version 2026.11.1) \
  <(helm show values linkerd-edge/linkerd-control-plane --version 2026.11.2)
helm --kube-context west get manifest linkerd-control-plane -n linkerd > alt.yaml
helm --kube-context west upgrade linkerd-crds linkerd-edge/linkerd-crds \
  --version 2026.11.2 -n linkerd
helm --kube-context west upgrade linkerd-control-plane \
  linkerd-edge/linkerd-control-plane --version 2026.11.2 -n linkerd \
  -f ../tag14/values-ha.yaml -f ../tag14/values-west.yaml
kubectl --context west -n tag15 rollout restart deploy/helper
helm --kube-context west history linkerd-control-plane -n linkerd
helm --kube-context west rollback linkerd-control-plane 2 -n linkerd
  • The namespace linkerd must exist with the CLI’s labels before the Helm installation; the issuer Secret needs tls.crt, tls.key and ca.crt.
  • values-ha.yaml (from the chart) sets hard anti-affinity, PDBs and proxy requests for all workloads.
  • Control Plane gone: existing proxies keep working. Injector gone: new Pods fail with FailedCreate (fail policy).
  • Installed via Helm, change via Helm – no longer with linkerd upgrade.
  • With externalCA, set identityTrustAnchorsPEM anyway (same content as the ConfigMap); otherwise linkerd multicluster check reports × clusters share trust anchors once a link exists.
  • Pin the proxy version per workload: config.linkerd.io/proxy-version.

Troubleshooting (Day 16)

From the outside in: load generator, linkerd check, metrics, the proxy’s policy, logs last.

linkerd diagnostics policy -n tag16 svc/helper 80 -o json \
  | jq -c '.protocol.Kind.Detect.http1.routes[].metadata.Kind'
linkerd diagnostics policy -n tag16 svc/banner 2525 -o json \
  | jq -c '.protocol.Kind | keys'
linkerd diagnostics endpoints helper.tag16.svc.cluster.local:80
linkerd viz edges -n tag16 deploy -o wide
kubectl -n tag16 get pod -l app=helper -o custom-columns=NAME:.metadata.name,\
PROXY-RESTARTS:.status.initContainerStatuses[1].restartCount,\
GRUND:.status.initContainerStatuses[1].lastState.terminated.reason
kubectl -n tag16 patch deploy web -p '{"spec":{"template":{"metadata":
  {"annotations":{"config.linkerd.io/proxy-log-level":"warn,linkerd=debug"}}}}}'
Symptom First command Common cause
Pod 1/1 instead of 2/2 proxy-version column of the Pods not restarted
Everything 403 proxy log, grep 'Request denied' no Server/no rule
Single client 403 linkerd viz edges -n <ns> deploy -o wide wrong ServiceAccount
Exactly 10 s delay linkerd diagnostics policy svc/<name> <port> opaque port missing
Init:Error, DB closed proxy-metrics for port_skipped skip port, old sidecar
Retry/timeout has no effect linkerd diagnostics policy svc/<name> <port> wrong parentRef port
503 from the client proxy linkerd diagnostics endpoints <fqdn>:<port> no endpoints
Proxy restarts lastState of the init containers OOMKilled
Pod not Ready kubectl get events route on the Server (Day 6)
TLS errors, no identity linkerd check (identity section) issuer expired, clock
viz shows only - proxy log, linkerd diagnostics policy proxy-admin missing
Rollout hangs silently kubectl get events (FailedCreate) proxy request > limit
Everything 504 header l5d-proxy-error (curl -si) timeout annotation too tight
429 for one client proxy log rate limit exceeded override with wrong namespace
a few percent 5xx viz stat-outbound per backend retries hide the canary
  • An HTTPRoute with the wrong port is still Accepted.
  • linkerd check --proxy without --wait hangs for up to 5 min if no proxies run.
  • An ephemeral container inherits the identity of its Pod.
  • The last three rows come from the Competition (Day 18, Rounds 4 to 6).

Measuring cost (Day 17)

Same setup without and with a proxy, fixed rate, JSON to stdout:

kubectl -n tag17 exec deploy/last-mit -- fortio load -qps 100 -t 60s -c 4 \
  -p 50,90,99 -json - http://server-mit/get 2>/dev/null > mit-100.json
jq -c '[(.ActualQPS|round),
  (.DurationHistogram.Percentiles[] | .Value*10000|round/10)]' mit-100.json
kubectl -n tag17 top pod --containers
kubectl -n tag17 get pods -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass
kubectl -n tag17 logs deploy/server-mit -c linkerd-proxy | grep runtime
R='{"spec":{"template":{"metadata":{"annotations":{
  "config.linkerd.io/proxy-cpu-request":"200m",
  "config.linkerd.io/proxy-memory-request":"20Mi"}}}}}'
kubectl -n tag17 patch deploy server-mit -p "$R"
  • Without requests, the proxy (and the whole Pod) is BestEffort – on full nodes, latency rises sharply.
  • A CPU limit without a request sets the request equal to the limit (Pod stays Pending). The rounded-up limit determines the worker threads (LINKERD2_PROXY_CORES_MAX, at most the node’s CPUs), the request CORES_MIN and the deprecated CORES.
  • kubectl top shows the working set (3 Mi); the proxy reports its RSS (22 MB).

Important annotations

Annotation On Effect Day
linkerd.io/inject: enabled / disabled Namespace, Pod proxy yes/no 2
config.linkerd.io/opaque-ports Service, Pod no protocol detection 2, 16
config.beta.linkerd.io/proxy-enable-native-sidecar Pod "false": old sidecar 2, 6
config.linkerd.io/default-inbound-policy Namespace deny: default-deny 6
retry.linkerd.io/http, …/limit, …/timeout HTTPRoute retries 7
timeout.linkerd.io/request (response, idle) HTTPRoute timeout 7
balancer.linkerd.io/failure-accrual Service circuit breaking 8
config.linkerd.io/proxy-metrics-hostname-labels Pod hostnames in metrics 11
config.linkerd.io/proxy-cpu-request Pod proxy CPU request 14, 17
config.linkerd.io/proxy-cpu-limit Pod CPU limit, worker threads 17
config.linkerd.io/proxy-version Pod pin proxy version 15
config.linkerd.io/skip-outbound-ports Pod port bypasses the proxy 16
config.linkerd.io/proxy-log-level Pod proxy log level 16
config.linkerd.io/proxy-memory-limit, …-request Pod proxy memory 16

“Pod” means: in the Pod template of the Deployment or StatefulSet. All proxy annotations take effect only after the Pods are restarted.