Appendix A: Cheat Sheet
The most important commands and snippets from Days 0 to 18, sorted by
topic. The days are in parentheses. You replace placeholders in angle
brackets; IPs are those of the lab (cp1 172.16.0.126,
west 172.16.0.182). Values like lab-key-123
are lab values. All commands run on cp1; without
--context they target the cluster ost.
Measuring (Days 0, 4)
Success rate from the load generator – the measuring instrument of the whole book:
kubectl -n notes logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
| sort | uniq -c
kubectl -n notes logs deploy/last --tail=300 | awk '{print $2, $3}' | sort | uniq -c
kubectl --context west -n tag15 logs deploy/last -c last \
--since-time=$(cat start.txt) | awk '{print $2, $3}' | sort | uniq -cQuery Prometheus from cp1:
kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090 \
>/dev/null 2>&1 &
Q='sum by (dst_deployment) (rate(response_total{namespace="notes",
deployment="web", direction="outbound", classification="success"}[5m]))
/ sum by (dst_deployment) (rate(response_total{namespace="notes",
deployment="web", direction="outbound"}[5m]))'
curl -s localhost:9090/api/v1/query --data-urlencode "query=$Q" \
| jq -r '.data.result[] | .metric.dst_deployment + " " + .value[1]'
kill %1Installation and check (Day 1)
Fetch the CLI architecture-neutrally and verify it against the GitHub API digest:
ARCH=$(dpkg --print-architecture)
V=edge-26.11.2
curl -fsSLO \
https://github.com/linkerd/linkerd2/releases/download/$V/linkerd2-cli-$V-linux-$ARCH
sha256sum linkerd2-cli-$V-linux-$ARCH
curl -fsSL https://api.github.com/repos/linkerd/linkerd2/releases/tags/$V \
| grep -o '"digest": "sha256:[0-9a-f]*"'
sudo install -m 0755 linkerd2-cli-$V-linux-$ARCH /usr/local/bin/linkerd
linkerd version --clientCRDs first, then the Control Plane:
linkerd check --pre
kubectl get crd httproutes.gateway.networking.k8s.io -o \
jsonpath='{.metadata.annotations.gateway\.networking\.k8s\.io/bundle-version}{"\n"}'
linkerd install --crds > crds.yaml
kubectl apply -f crds.yaml
linkerd install > control-plane.yaml
kubectl apply -f control-plane.yaml
kubectl -n linkerd rollout status deploy --timeout=300s
linkerd check
linkerd version --short
linkerd install --ha --ignore-cluster > ha.yaml # for comparison only| Command | Checks | Day |
|---|---|---|
linkerd check --pre |
cluster before installation | 1 |
linkerd check |
Control Plane, certificates, webhooks | 1, 5 |
linkerd check --proxy -n <ns> |
proxies of a namespace | 2, 6 |
linkerd check --proxy -n <ns> --wait 30s |
the same, without hanging for 5 min | 16 |
linkerd viz check |
viz extension | 4 |
linkerd multicluster check |
extension and links | 12 |
- Linkerd does not ship the Gateway API CRDs (in the lab: Cilium or k3s).
- The Control Plane:
linkerd-identity(CA),linkerd-destination(discovery and policy),linkerd-proxy-injector(webhook). The proxy runs as a native sidecar (init container withrestartPolicy: Always).
Injection (Day 2)
kubectl annotate namespace notes linkerd.io/inject=enabled
kubectl -n notes rollout restart deploy/notes-api deploy/helper deploy/web deploy/last
kubectl -n notes rollout restart statefulset/postgres
kubectl -n notes get pods # 2/2 = proxy present
kubectl -n notes get pods -o custom-columns=\
NAME:.metadata.name,PROXY:.metadata.annotations.linkerd\\.io/proxy-version
linkerd version --proxy -n notes
kubectl -n notes annotate service postgres config.linkerd.io/opaque-ports=5432
linkerd inject plain.yaml # only set the annotation
linkerd inject --manual plain.yaml # insert the proxy container directly
kubectl -n tag02 patch deploy ohne -p \
'{"spec":{"template":{"metadata":{"annotations":{"linkerd.io/inject":"disabled"}}}}}'One identity per service – via dedicated ServiceAccounts:
for a in web notes-api helper last postgres; do
kubectl -n notes create serviceaccount $a
done
for d in web notes-api helper last; do
kubectl -n notes set serviceaccount deploy/$d $d
done
kubectl -n notes set serviceaccount statefulset/postgres postgres
kubectl -n notes logs deploy/web -c linkerd-proxy | grep 'Local identity'- The namespace annotation only affects new Pods:
rollout restart. - With the native sidecar, init containers run in the mesh. With
config.beta.linkerd.io/proxy-enable-native-sidecar: "false"they run before the proxy and thus without mTLS – under default-deny they fail (Day 6). - Server-first protocols (SMTP, MySQL) and Postgres need
opaque-ports; otherwise, with server-first, protocol detection waits 10 s (Day 16).
Identity and mTLS (Day 3)
linkerd identity -n notes -l app=web | grep -E 'POD|Issuer|Not |Subject:|DNS:'
kubectl -n notes logs deploy/web -c linkerd-proxy | grep 'Certified identity'
kubectl -n linkerd logs deploy/linkerd-identity -c identity | grep issued | tail -1
kubectl -n linkerd get configmap linkerd-config -o jsonpath='{.data.values}' \
| grep -E 'issuanceLifetime|clockSkewAllowance'
linkerd diagnostics proxy-metrics -n notes deploy/web | grep '^control_identity_cert'Who talks mTLS, who talks plaintext (client_id,
no_tls_reason):
linkerd diagnostics proxy-metrics -n notes pod/postgres-0 \
| grep '^tcp_open_total{direction="inbound",peer="src"' \
| grep -oE '(target_port|client_id|no_tls_reason)="[^"]*"|[0-9]+$' | paste - - -Look on the wire (ephemeral container with tcpdump):
P=$(kubectl -n notes get pod -l app=notes-api -o jsonpath='{.items[0].metadata.name}')
kubectl -n notes debug $P --image=docker.io/nicolaka/netshoot:v0.15 \
--profile=netadmin -c sniff -- sleep 1800
kubectl -n notes exec $P -c sniff -- timeout 10 \
tcpdump -i eth0 -nn -A 'tcp port 4143' 2>/dev/null | grep -c 'lab-key'- Meshed traffic arrives on port 4143, TLS 1.3, SNI = target identity.
- Identity:
<sa>.<ns>.serviceaccount.identity.linkerd.cluster.local. kubectl debugonly works on Pods, not ondeploy/….
viz and Grafana (Day 4)
P=http://kps-kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090
linkerd viz install --set prometheus.enabled=false --set prometheusUrl=$P \
--set grafana.url=kps-grafana.monitoring.svc.cluster.local > viz.yaml
kubectl apply -f viz.yaml
linkerd viz check| Command | Shows | Day |
|---|---|---|
linkerd viz stat -n notes deploy |
success rate, RPS, latency | 4 |
linkerd viz stat -n notes deploy/helper --from deploy/web |
one edge | 4 |
linkerd viz tap -n notes deploy/helper --path /unstable |
live requests | 4 |
linkerd viz top -n notes deploy/web |
most frequent paths (needs a TTY) | 4 |
linkerd viz edges -n notes deploy |
edges with identities | 4 |
linkerd viz routes -n notes deploy/web |
routes | 4 |
linkerd viz authz -n notes deploy/helper |
hits per Server/rule | 6 |
linkerd viz stat-outbound -n tag07 deploy/web $PROM |
per route and backend | 7 |
linkerd viz dashboard --show url |
web UI | 4 |
ssh -L 50750:localhost:50750 [email protected] linkerd viz dashboard --show url
PROM=--prometheusURL=http://localhost:9090 # stat-outbound/-inbound via port-forward
kubectl -n monitoring create configmap linkerd-deployment-dashboard \
--from-file=linkerd-deployment.json
kubectl -n monitoring label configmap linkerd-deployment-dashboard grafana_dashboard=1- Without
prometheusUrl,viz installaborts whenprometheus.enabled=false. - Proxy metrics reach Prometheus via the PodMonitor
linkerd-proxy(portlinkerd-admin, 4191). tapneeds restarted Pods (viz.linkerd.io/tap-enabled) and does not show denied (403) requests.
Certificates (Day 5)
Trust anchor (10 years) and issuer (1 year) with
step:
step certificate create root.linkerd.cluster.local ca.crt ca.key \
--profile root-ca --no-password --insecure --not-after 87600h
step certificate create identity.linkerd.cluster.local issuer.crt issuer.key \
--profile intermediate-ca --not-after 8760h --no-password --insecure \
--ca ca.crt --ca-key ca.key
step certificate verify issuer.crt --roots ca.crt && echo verify-ok
step certificate fingerprint ca.crt
kubectl -n linkerd get configmap linkerd-identity-trust-roots \
-o jsonpath='{.data.ca-bundle\.crt}' | step certificate inspect --short -Anchor rotation in three rounds; after each round, restart CP, viz and workloads:
step certificate bundle ca.crt original-trust.crt bundle.crt
linkerd upgrade --identity-trust-anchors-file=./bundle.crt > upgrade-1.yaml
linkerd upgrade --identity-issuer-certificate-file=./issuer.crt \
--identity-issuer-key-file=./issuer.key > upgrade-2.yaml
linkerd upgrade --identity-trust-anchors-file=./ca.crt > upgrade-3.yaml
kubectl get events -n linkerd --field-selector reason=IssuerUpdatedcert-manager and trust-manager take over:
helm install trust-manager oci://quay.io/jetstack/charts/trust-manager \
--version v0.26.0 -n cert-manager --set app.trust.namespace=cert-manager --wait
kubectl -n cert-manager create secret tls linkerd-trust-anchor \
--cert=ca.crt --key=ca.key
linkerd upgrade --set identity.externalCA=true \
--set identity.issuer.scheme=kubernetes.io/tls > upgrade-4.yaml
cmctl renew -n linkerd linkerd-identity-issuer
kubectl -n linkerd get certificate linkerd-identity-issuer \
-o jsonpath='{.status.notAfter} {.status.renewalTime}{"\n"}'apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata: {name: linkerd-identity-issuer}
spec:
ca: {secretName: linkerd-trust-anchor}
---
apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata: {name: linkerd-identity-trust-roots}
spec:
sources:
- secret: {name: linkerd-trust-anchor, key: tls.crt}
target:
configMap: {key: ca-bundle.crt}
namespaceSelector:
matchLabels: {linkerd.io/is-control-plane: "true"}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata: {name: linkerd-identity-issuer, namespace: linkerd}
spec:
issuerRef: {name: linkerd-identity-issuer, kind: ClusterIssuer}
secretName: linkerd-identity-issuer
isCA: true
commonName: identity.linkerd.cluster.local
duration: 48h0m0s
renewBefore: 25h0m0s
privateKey: {rotationPolicy: Always, algorithm: ECDSA}- Keep
ca.keysafe: Days 12 and 14 use it to sign the issuers forwest. - If
linkerd upgradefails after switching toexternalCA, the old issuer values are still inlinkerd-config-overrides– remove them, then try again. - The warning “issuer cert is valid for at least 60 days” is expected with 48 h.
Policies (Day 6)
apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata: {name: notes-api-http, namespace: notes}
spec:
podSelector: {matchLabels: {app: notes-api}}
port: http
proxyProtocol: HTTP/1
accessPolicy: deny # audit to observe, all-unauthenticated (Day 8)
---
apiVersion: policy.linkerd.io/v1alpha1
kind: MeshTLSAuthentication
metadata: {name: web, namespace: notes}
spec:
identityRefs: [{kind: ServiceAccount, name: web}]
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: notes-api-von-web, namespace: notes}
spec:
targetRef: {group: policy.linkerd.io, kind: Server, name: notes-api-http}
requiredAuthenticationRefs:
- {group: policy.linkerd.io, kind: MeshTLSAuthentication, name: web}For clients without a proxy (Prometheus, kubelet, Cilium gateway), by network:
apiVersion: policy.linkerd.io/v1alpha1
kind: NetworkAuthentication
metadata: {name: pod-netz, namespace: notes}
spec:
networks:
- cidr: 10.0.0.0/16kubectl annotate namespace notes config.linkerd.io/default-inbound-policy=deny
kubectl -n notes get server,authorizationpolicy,meshtlsauthentication,\
networkauthentication
kubectl -n notes get pod -l app=helper -o yaml | grep -m1 -A1 INBOUND_DEFAULT_POLICY
linkerd diagnostics policy -n notes po/<pod> 8080 -o json \
| jq -r '.authorizations[].labels.name'
kubectl -n notes logs deploy/helper -c linkerd-proxy --since=2m \
| grep -m1 'Request denied' | grep -oE 'server.name=[^ ]*|Name\("[^"]*'
linkerd diagnostics proxy-metrics -n notes po/postgres-0 \
| grep '^inbound_tcp_authz_deny_total' | grep -oE '(client_id|tls)="[^"]*"'- Procedure: Server with
audit, create rules, evaluate the audit, switch todeny, finally default-deny for the namespace (restart the Pods). - Default-deny also blocks Prometheus on 4191: allow a Server
proxy-admin(podSelector: {}, portlinkerd-admin) with a NetworkAuthentication. - Without an HTTPRoute on the Server, probes are allowed automatically. Once a route is attached to the Server, the probes need their own route plus a rule for the kubelet IPs.
- Denied: HTTP with 403, opaque with a closed connection.
Routes, retries and timeouts (Day 7)
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: helper-unstable
namespace: tag07
annotations:
retry.linkerd.io/http: 5xx
retry.linkerd.io/limit: "2"
spec:
parentRefs:
- {group: core, kind: Service, name: helper, port: 80}
rules:
- matches:
- path: {type: PathPrefix, value: /unstable}
method: GET # don't retry POST
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: helper-delay
namespace: tag07
annotations:
timeout.linkerd.io/request: 1s
spec:
parentRefs:
- {group: core, kind: Service, name: helper, port: 80}
rules:
- matches:
- path: {type: PathPrefix, value: /delay}kubectl -n tag07 get httproute helper-unstable -o yaml | grep -E 'controllerName|reason'
kubectl -n tag07 annotate httproute helper-unstable retry.linkerd.io/limit=3 --overwrite
kubectl -n tag07 patch httproute helper-delay --type=json \
-p '[{"op":"add","path":"/spec/rules/0/timeouts","value":{"request":"3s"}}]'
linkerd diagnostics proxy-metrics -n tag07 deploy/last \
| grep -E '^outbound_http_route_retry_(requests|limit_exceeded)_total' \
| sed -E 's/\{.*\}//'- First route on the Service: all other paths get 404. Create a
catch-all route with
PathPrefix /. - Retries also apply to POST if the route does not restrict the method.
- No retry budget, only
limit; a ServiceProfile for the same Service overrides the annotations. The timeout annotation beatsrules[].timeouts.request. - A request timeout is not retried;
retry.linkerd.io/timeoutmakes slow attempts retryable.
Circuit breaking and rate limiting (Day 8)
kubectl -n tag08 annotate service helper \
balancer.linkerd.io/failure-accrual=consecutive \
balancer.linkerd.io/failure-accrual-consecutive-max-failures=3 \
balancer.linkerd.io/failure-accrual-consecutive-min-penalty=10s \
balancer.linkerd.io/failure-accrual-consecutive-max-penalty=1m
linkerd viz stat -n tag08 po -l app=helper --from deploy/web
linkerd diagnostics proxy-metrics -n tag08 deploy/web \
| grep '^outbound_http_balancer_endpoints.*pending' | grep -oE '[0-9]+$'apiVersion: policy.linkerd.io/v1alpha1
kind: HTTPLocalRateLimitPolicy
metadata: {name: helper-rl, namespace: tag08}
spec:
targetRef: {group: policy.linkerd.io, kind: Server, name: helper-http}
total:
requestsPerSecond: 20
identity:
requestsPerSecond: 1
overrides:
- requestsPerSecond: 4
clientRefs:
- {kind: ServiceAccount, name: web}kubectl -n tag08 get httplocalratelimitpolicy
kubectl -n tag08 logs deploy/helper -c linkerd-proxy --since=1m \
| grep -o 'total rate limit exceeded: [0-9]*rps' | tail -1
linkerd diagnostics proxy-metrics -n tag08 deploy/helper \
| grep '^inbound_http_local_ratelimit' \
| grep -oE 'ratelimit_scope="[^"]*"|client_id="[a-z]+|\} [0-9]+' | paste - - -- Failure accrual blocks individual Pods; if all are blocked, you get 503.
- Rate limits apply per Pod, response 429. Requests denied via
identityalso consume thetotalbudget – choosetotalgenerously.
Traffic shifting (Day 9)
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: notes-api-canary, namespace: tag09}
spec:
parentRefs:
- {group: "", kind: Service, name: notes-api, port: 80}
rules:
- matches:
- headers: [{name: x-canary, value: "true"}]
backendRefs:
- {name: notes-api-v2, port: 80}
- backendRefs:
- {name: notes-api, port: 80, weight: 90}
- {name: notes-api-v2, port: 80, weight: 10}W=/spec/rules/0/backendRefs
kubectl -n tag09 patch httproute notes-api-canary --type=json -p \
"[{\"op\":\"replace\",\"path\":\"$W/0/weight\",\"value\":50},
{\"op\":\"replace\",\"path\":\"$W/1/weight\",\"value\":50}]"
kubectl -n tag09 get httproute notes-api-canary \
-o jsonpath='{.spec.rules[*].backendRefs[*].weight}{"\n"}'
kubectl -n tag09 delete httproute notes-api-canary # rollback in seconds
linkerd viz stat -n tag09 deploy --from deploy/last
kubectl -n tag09 delete endpointslice -l io.cilium.gateway/owning-gateway=notes-api- Linkerd and Cilium both accept the route with a
Service parent: Linkerd splits meshed clients, Cilium Envoy splits
unmeshed clients (
server: envoy). - Rule without
backendRefs: Linkerd uses the parent Service, Cilium’s Envoy answers unmeshed clients with500(Day 18, Round 1). - To finish: Service selector to v2, delete the route, scale down v1.
Ingress (Day 10)
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata: {name: notes, namespace: tag10}
spec:
gatewayClassName: cilium
listeners:
- {name: http, protocol: HTTP, port: 80}
- name: https
protocol: HTTPS
port: 443
hostname: notes.lab.internal
tls:
mode: Terminate
certificateRefs: [{name: notes-tls}]NP=$(kubectl -n tag10 get svc cilium-gateway-notes \
-o jsonpath='{.spec.ports[?(@.port==443)].nodePort}')
curl -s -w ' %{http_code}\n' --cacert lab-ca.crt \
--resolve notes.lab.internal:$NP:172.16.0.126 https://notes.lab.internal:$NP/api/notes
kubectl get ciliumnodes \
-o jsonpath='{range .items[*]}{.metadata.name} {.spec.ingress.ipv4}{"\n"}{end}'
cmctl renew -n tag10 notes-tls| Entry point | Mesh view | Rule in the policy |
|---|---|---|
| Cilium gateway | no_tls_from_remote, ingress IPs of the nodes |
NetworkAuthentication |
nginx edge in the mesh |
mTLS with identity edge |
MeshTLSAuthentication |
Egress (Day 11)
apiVersion: policy.linkerd.io/v1alpha1
kind: EgressNetwork
metadata: {name: internet, namespace: tag11}
spec:
trafficPolicy: Deny
---
apiVersion: gateway.networking.k8s.io/v1
kind: TLSRoute
metadata: {name: example-com, namespace: tag11}
spec:
parentRefs:
- {group: policy.linkerd.io, kind: EgressNetwork, name: internet, port: 443}
hostnames: [example.com]
rules:
- backendRefs:
- {group: policy.linkerd.io, kind: EgressNetwork, name: internet}kubectl -n tag11 patch egressnetwork internet --type=merge \
-p '{"spec":{"trafficPolicy":"Deny"}}'
kubectl -n tag11 patch tlsroute example-com --type=json \
-p '[{"op":"add","path":"/spec/hostnames/-","value":"linkerd.io"}]'
kubectl -n tag11 get httproute,tlsroute- Without a TLSRoute, the proxy sees HTTPS only as
tcpwith an IP; with a TLSRoute, astlswith SNI. HTTPRoutes on the EgressNetwork filter plaintext HTTP by method and path. - Hostnames in metrics:
config.linkerd.io/proxy-metrics-hostname-labels: "true". - Policy only affects new connections: restart keep-alive clients
after
Deny. - Unmeshed Pods bypass the egress policy.
Multicluster (Days 12, 13)
Second cluster with a shared trust anchor:
step certificate create identity.linkerd.cluster.local \
west-issuer.crt west-issuer.key \
--profile intermediate-ca --not-after 8760h --no-password --insecure \
--ca ../tag05/ca.crt --ca-key ../tag05/ca.key
linkerd --context west install --crds | kubectl --context west apply -f -
linkerd --context west install \
--identity-trust-anchors-file ../tag05/ca.crt \
--identity-issuer-certificate-file west-issuer.crt \
--identity-issuer-key-file west-issuer.key > west-cp.yaml
kubectl --context west apply -f west-cp.yamlExtension, link, export:
linkerd --context west multicluster install > mc-west.yaml
kubectl --context west apply -f mc-west.yaml
linkerd --context ost multicluster install -f mc-ost-values.yaml > mc-ost.yaml
kubectl --context ost apply -f mc-ost.yaml
linkerd --context west multicluster link-gen --cluster-name west > link-west.yaml
kubectl --context ost apply -f link-west.yaml
kubectl --context west -n tag12 label service helper mirror.linkerd.io/exported=true
linkerd --context ost multicluster gateways
linkerd --context ost viz stat -n tag12 deploy/last --to svc/helper-west
kubectl --context ost get svc -A -l mirror.linkerd.io/mirrored-service=truegateway:
enabled: false # ost: no gateway, controller only
controllers:
- link:
ref:
name: west| Label on the Service (source cluster) | Result in the target cluster | Day |
|---|---|---|
mirror.linkerd.io/exported=true |
<svc>-west, traffic via gateway |
12 |
mirror.linkerd.io/exported=remote-discovery |
<svc>-west, directly to Pod IPs |
13 |
mirror.linkerd.io/federated=member |
<svc>-federated from all clusters |
13 |
linkerd --context ost diagnostics endpoints echo-federated.tag13.svc.cluster.local:80
kubectl --context ost -n linkerd-multicluster delete link west
kubectl --context ost -n linkerd-multicluster delete secret cluster-credentials-west
kubectl --context ost -n linkerd delete secret cluster-credentials-west- The mirror namespace must exist in the reading cluster.
- NodePort gateway: set
controllers[].gateway.probe.portto the probe NodePort before the link is created. remote-discoveryandfederatedneed a flat network (Pod IPs reachable).linkerd multicluster unlinkno longer exists.- Gateway gone: the client proxy puts the backend into failfast after ~3 s, the controller empties the endpoints after ~9 s. The return takes longer (backoff in the controller’s proxy, up to 60 s between two attempts).
Helm, HA and upgrade (Days 14, 15)
helm repo add linkerd-edge https://helm.linkerd.io/edge
helm search repo linkerd-edge/linkerd-control-plane --versions | head -4
kubectl --context west -n linkerd create configmap linkerd-identity-trust-roots \
--from-file=ca-bundle.crt=../tag05/ca.crt
kubectl --context west -n linkerd create secret generic linkerd-identity-issuer \
--type=kubernetes.io/tls --from-file=tls.crt=issuer.crt \
--from-file=tls.key=issuer.key --from-file=ca.crt=../tag05/ca.crt
helm --kube-context west install linkerd-crds linkerd-edge/linkerd-crds \
--version 2026.11.1 -n linkerd
helm --kube-context west install linkerd-control-plane \
linkerd-edge/linkerd-control-plane --version 2026.11.1 -n linkerd \
-f values-ha.yaml -f values-west.yamlidentity:
externalCA: true # trust anchor from the ConfigMap
issuer:
scheme: kubernetes.io/tls # issuer from the Secret
controllerReplicas: 2
enablePodAntiAffinity: false # single node only
identityTrustAnchorsPEM: | # for multicluster check
-----BEGIN CERTIFICATE-----
...Upgrade: release notes, preview, CRDs, Control Plane, Data Plane:
diff <(helm show values linkerd-edge/linkerd-control-plane --version 2026.11.1) \
<(helm show values linkerd-edge/linkerd-control-plane --version 2026.11.2)
helm --kube-context west get manifest linkerd-control-plane -n linkerd > alt.yaml
helm --kube-context west upgrade linkerd-crds linkerd-edge/linkerd-crds \
--version 2026.11.2 -n linkerd
helm --kube-context west upgrade linkerd-control-plane \
linkerd-edge/linkerd-control-plane --version 2026.11.2 -n linkerd \
-f ../tag14/values-ha.yaml -f ../tag14/values-west.yaml
kubectl --context west -n tag15 rollout restart deploy/helper
helm --kube-context west history linkerd-control-plane -n linkerd
helm --kube-context west rollback linkerd-control-plane 2 -n linkerd- The namespace
linkerdmust exist with the CLI’s labels before the Helm installation; the issuer Secret needstls.crt,tls.keyandca.crt. values-ha.yaml(from the chart) sets hard anti-affinity, PDBs and proxy requests for all workloads.- Control Plane gone: existing proxies keep working. Injector gone:
new Pods fail with
FailedCreate(fail policy). - Installed via Helm, change via Helm – no longer with
linkerd upgrade. - With
externalCA, setidentityTrustAnchorsPEManyway (same content as the ConfigMap); otherwiselinkerd multicluster checkreports× clusters share trust anchorsonce a link exists. - Pin the proxy version per workload:
config.linkerd.io/proxy-version.
Troubleshooting (Day 16)
From the outside in: load generator, linkerd check,
metrics, the proxy’s policy, logs last.
linkerd diagnostics policy -n tag16 svc/helper 80 -o json \
| jq -c '.protocol.Kind.Detect.http1.routes[].metadata.Kind'
linkerd diagnostics policy -n tag16 svc/banner 2525 -o json \
| jq -c '.protocol.Kind | keys'
linkerd diagnostics endpoints helper.tag16.svc.cluster.local:80
linkerd viz edges -n tag16 deploy -o wide
kubectl -n tag16 get pod -l app=helper -o custom-columns=NAME:.metadata.name,\
PROXY-RESTARTS:.status.initContainerStatuses[1].restartCount,\
GRUND:.status.initContainerStatuses[1].lastState.terminated.reason
kubectl -n tag16 patch deploy web -p '{"spec":{"template":{"metadata":
{"annotations":{"config.linkerd.io/proxy-log-level":"warn,linkerd=debug"}}}}}'| Symptom | First command | Common cause |
|---|---|---|
Pod 1/1 instead of 2/2 |
proxy-version column of the Pods |
not restarted |
Everything 403 |
proxy log, grep 'Request denied' |
no Server/no rule |
Single client 403 |
linkerd viz edges -n <ns> deploy -o wide |
wrong ServiceAccount |
| Exactly 10 s delay | linkerd diagnostics policy svc/<name> <port> |
opaque port missing |
Init:Error, DB closed |
proxy-metrics for port_skipped |
skip port, old sidecar |
| Retry/timeout has no effect | linkerd diagnostics policy svc/<name> <port> |
wrong parentRef port |
503 from the client proxy |
linkerd diagnostics endpoints <fqdn>:<port> |
no endpoints |
| Proxy restarts | lastState of the init containers |
OOMKilled |
| Pod not Ready | kubectl get events |
route on the Server (Day 6) |
| TLS errors, no identity | linkerd check (identity section) |
issuer expired, clock |
viz shows only - |
proxy log, linkerd diagnostics policy |
proxy-admin missing |
| Rollout hangs silently | kubectl get events (FailedCreate) |
proxy request > limit |
Everything 504 |
header l5d-proxy-error (curl -si) |
timeout annotation too tight |
429 for one client |
proxy log rate limit exceeded |
override with wrong namespace |
a few percent 5xx |
viz stat-outbound per backend |
retries hide the canary |
- An HTTPRoute with the wrong port is still
Accepted. linkerd check --proxywithout--waithangs for up to 5 min if no proxies run.- An ephemeral container inherits the identity of its Pod.
- The last three rows come from the Competition (Day 18, Rounds 4 to 6).
Measuring cost (Day 17)
Same setup without and with a proxy, fixed rate, JSON to stdout:
kubectl -n tag17 exec deploy/last-mit -- fortio load -qps 100 -t 60s -c 4 \
-p 50,90,99 -json - http://server-mit/get 2>/dev/null > mit-100.json
jq -c '[(.ActualQPS|round),
(.DurationHistogram.Percentiles[] | .Value*10000|round/10)]' mit-100.json
kubectl -n tag17 top pod --containers
kubectl -n tag17 get pods -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass
kubectl -n tag17 logs deploy/server-mit -c linkerd-proxy | grep runtimeR='{"spec":{"template":{"metadata":{"annotations":{
"config.linkerd.io/proxy-cpu-request":"200m",
"config.linkerd.io/proxy-memory-request":"20Mi"}}}}}'
kubectl -n tag17 patch deploy server-mit -p "$R"- Without requests, the proxy (and the whole Pod) is BestEffort – on full nodes, latency rises sharply.
- A CPU limit without a request sets the request equal to the limit
(Pod stays
Pending). The rounded-up limit determines the worker threads (LINKERD2_PROXY_CORES_MAX, at most the node’s CPUs), the requestCORES_MINand the deprecatedCORES. kubectl topshows the working set (3 Mi); the proxy reports its RSS (22 MB).
Important annotations
| Annotation | On | Effect | Day |
|---|---|---|---|
linkerd.io/inject: enabled / disabled |
Namespace, Pod | proxy yes/no | 2 |
config.linkerd.io/opaque-ports |
Service, Pod | no protocol detection | 2, 16 |
config.beta.linkerd.io/proxy-enable-native-sidecar |
Pod | "false": old sidecar |
2, 6 |
config.linkerd.io/default-inbound-policy |
Namespace | deny: default-deny |
6 |
retry.linkerd.io/http, …/limit,
…/timeout |
HTTPRoute | retries | 7 |
timeout.linkerd.io/request (response,
idle) |
HTTPRoute | timeout | 7 |
balancer.linkerd.io/failure-accrual |
Service | circuit breaking | 8 |
config.linkerd.io/proxy-metrics-hostname-labels |
Pod | hostnames in metrics | 11 |
config.linkerd.io/proxy-cpu-request |
Pod | proxy CPU request | 14, 17 |
config.linkerd.io/proxy-cpu-limit |
Pod | CPU limit, worker threads | 17 |
config.linkerd.io/proxy-version |
Pod | pin proxy version | 15 |
config.linkerd.io/skip-outbound-ports |
Pod | port bypasses the proxy | 16 |
config.linkerd.io/proxy-log-level |
Pod | proxy log level | 16 |
config.linkerd.io/proxy-memory-limit,
…-request |
Pod | proxy memory | 16 |
“Pod” means: in the Pod template of the Deployment or StatefulSet. All proxy annotations take effect only after the Pods are restarted.