Day 18: Competition – The Broken Mesh
Goal: In eight rounds you find and fix planted mesh faults from across the book – against the clock, with proof before and after. · Duration: approx. 90 min · Prerequisite: Days 0–17
Briefing
- In
tag18you build a complete copy ofnotes: in the mesh, with default-deny, the rules from Day 6 and the retries from Day 7. - Eight rounds, all topics: injection, opaque ports, authorization, timeouts, rate limits, traffic shifting, egress and certificates. The rounds build on each other: play them in order.
- Tools: load generator,
linkerd viz stat/authz/stat-outbound,linkerd diagnostics policy/proxy-metrics,linkerd identity, proxy logs, events. - Every round ends with a proof command: broken before, fixed after.
Round Symptom Target time
1 Everything 403, only /flaky 500 3 min
2 /flaky only 502 5 min
3 First 403, then 503 on /api/notes 7 min
4 /api/notes entirely 504 4 min
5 Every second call 429 4 min
6 A few percent 500 after a release 5 min
7 /api/upstream fails 5 min
8 Questions from the auditors (read only) 4 min
Rules of Play
- Per round, the referee runs the referee commands – a colleague, or you yourself without looking: copy the block, paste it into a second terminal, close the window. Then start the clock.
- Find the cause, fix it, prove it with one command. Only then stop the clock.
- Fix the cause, not the symptom: whoever turns off default-deny,
deletes a rule instead of correcting it, opens policies with
all-unauthenticatedor deletes Pods that are not broken gets zero points. - Only read the Hint once the target time has run out.
- As always, the load generator is your measuring instrument. The goal
of every round: back to 100 percent
200on/api/notes,/flakyalmost only200(retries), unless the round says otherwise. In the following, the command for this is simply called measurement:
kubectl -n tag18 logs deploy/last -c last --since=40s | awk '{print $2, $3}' \
| sort | uniq -cPlaying Field
The manifests from Day 0 and the rules from Days 6 and 7 are already
on cp1. sed rewrites them to the new namespace
and halves the replicas – one Pod per service is enough for the
competition.
mkdir -p ~/linkerd/tag18 && cd ~/linkerd/tag18
kubectl create namespace tag18
kubectl annotate namespace tag18 linkerd.io/inject=enabled \
config.linkerd.io/default-inbound-policy=deny
kubectl -n tag18 create secret generic notes-db \
--from-literal=password=lab-db-123 \
--from-literal=url='postgres://notes:lab-db-123@postgres:5432/notes?sslmode=disable'
kubectl -n tag18 create secret generic notes-api-key --from-literal=key=lab-key-123
for a in web notes-api helper last postgres; do
kubectl -n tag18 create serviceaccount $a
done
for f in tag06/servers tag06/authz tag06/admin tag07/route-retry tag07/route-rest; do
sed 's/namespace: notes/namespace: tag18/; s/namespace: tag07/namespace: tag18/' \
~/linkerd/$f.yaml; echo ---
done > policy.yaml
for f in postgres notes-api helper web last; do
sed 's/namespace: notes/namespace: tag18/; s/\.notes\.svc/.tag18.svc/
s/replicas: 2/replicas: 1/' ~/linkerd/tag00/$f.yaml; echo ---
done > app.yaml
grep -c '^kind:' policy.yaml app.yaml
kubectl apply -f policy.yaml -f app.yaml | wc -lExpected (abridged):
namespace/tag18 created
...
policy.yaml:16
app.yaml:10
26
Now, as on Day 2, the dedicated ServiceAccounts and the opaque port,
plus a target on the internet for notes-api (Day 11):
kubectl -n tag18 annotate service postgres config.linkerd.io/opaque-ports=5432
kubectl -n tag18 set serviceaccount statefulset/postgres postgres
for d in web notes-api helper last; do
kubectl -n tag18 set serviceaccount deploy/$d $d
done
kubectl -n tag18 set env deploy/notes-api NOTES_UPSTREAM=https://example.com
kubectl -n tag18 rollout status statefulset/postgres --timeout=180s
kubectl -n tag18 rollout status deploy --timeout=240s
sleep 60
kubectl -n tag18 get podsExpected (abridged):
NAME READY STATUS RESTARTS AGE
helper-79d57fbbbb-kn8rl 2/2 Running 0 100s
last-78895758f9-br8k7 2/2 Running 0 99s
notes-api-6ff48b7779-mmss9 2/2 Running 0 99s
postgres-0 2/2 Running 0 84s
web-7d94ccc84d-9k2bv 2/2 Running 0 102s
The first Pods still start with the ServiceAccount
default and briefly fail on default-deny – after
set serviceaccount they are replaced. The baseline:
kubectl -n tag18 logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
| sort | uniq -c
linkerd viz stat -n tag18 deploy
kubectl -n tag18 exec deploy/last -c last -- curl -s http://web/api/upstream; echoExpected (abridged, numbers vary):
80 /api/notes 200
38 /flaky 200
1 /flaky 500
NAME MESHED SUCCESS RPS LATENCY_P50 LATENCY_P95 LATENCY_P99 ...
helper 1/1 78.57% 1.4rps 3ms 12ms 18ms ...
last 1/1 100.00% 0.3rps 1ms 19ms 20ms ...
notes-api 1/1 100.00% 1.8rps 4ms 20ms 90ms ...
web 1/1 100.00% 2.4rps 15ms 86ms 97ms ...
{"status":200,"url":"https://example.com"}
This is the healthy playing field. Remember the numbers.
Round 1: Everything Locked
Situation: After a “small update” to
web, the load generator reports nothing but errors.
Supposedly nobody touched the policies. Bring web back.
Referee commands:
kubectl -n tag18 patch deploy web -p '{"spec":{"template":{"metadata":
{"annotations":{"linkerd.io/inject":"disabled"}}}}}'
kubectl -n tag18 rollout status deploy/web --timeout=120sTarget time: 3 min
Hint: Count the containers in the web
Pod.
Solution to Round 1
sleep 50
kubectl -n tag18 logs deploy/last -c last --since=40s | awk '{print $2, $3}' \
| sort | uniq -c
kubectl -n tag18 get podsExpected (abridged):
58 /api/notes 403
28 /flaky 500
NAME READY STATUS RESTARTS AGE
helper-79d57fbbbb-kn8rl 2/2 Running 0 2m44s
...
web-5f7fb96f68-4nq9w 1/1 Running 0 53s
web has only one container – no proxy. The proof from
the other side:
kubectl -n tag18 logs deploy/notes-api -c linkerd-proxy --since=1m \
| grep 'Request denied' | tail -1 \
| grep -oE 'server.name=[^ ]*|client.tls=[^ ]*|client.ip=[^ ]*'
kubectl -n tag18 get pod -l app=web -o wide
kubectl -n tag18 get deploy web \
-o jsonpath='{.spec.template.metadata.annotations}{"\n"}'Expected (abridged):
server.name=notes-api-http
client.tls=None(NoClientHello)
client.ip=10.0.1.179
NAME READY STATUS RESTARTS AGE IP NODE ...
web-5f7fb96f68-4nq9w 1/1 Running 0 77s 10.0.1.179 worker1 ...
{"linkerd.io/inject":"disabled"}
No proxy, no mTLS; no mTLS, no identity – default-deny and the rule
notes-api-von-web reject the request (Day 6). And why
500 instead of 403 on /flaky?
kubectl -n tag18 exec deploy/web -c web -- curl -si \
'http://helper.tag18.svc.cluster.local/unstable?failure_rate=0.3' \
| grep -iE '^HTTP|^server'Expected:
HTTP/1.1 500 Internal Server Error
server: envoy
server: envoy: HTTPRoutes are attached to
helper (retries from Day 7). Unmeshed clients therefore end
up at Cilium, not at Linkerd (Day 9). But why 500? A look
at Cilium’s Envoy configuration and at the other side:
kubectl -n tag18 get ciliumenvoyconfig helper \
-o jsonpath='{..directResponse.status}{"\n"}'
kubectl -n tag18 logs deploy/helper -c linkerd-proxy --since=1m \
| grep -c 'Request denied'Expected:
500 500
0
Envoy answers both routes itself with 500
(directResponse, empty body), and the helper
proxy has not denied a single call – the request never leaves Envoy. The
reason is in the routes from Day 7: no rule has
backendRefs. Linkerd sends such a rule to the Service
itself; Cilium follows the Gateway API literally and answers a rule
without a backend with 500. With backendRefs,
Envoy would forward, and then the helper proxy denies:
403, still with server: envoy (cross-checked).
Two status codes, one cause – web has no proxy. The
annotation has to go:
kubectl -n tag18 patch deploy web --type=json -p '[{"op":"remove",
"path":"/spec/template/metadata/annotations/linkerd.io~1inject"}]'
kubectl -n tag18 rollout status deploy/web --timeout=120s
sleep 50
kubectl -n tag18 get pods -l app=web
kubectl -n tag18 logs deploy/last -c last --since=40s | awk '{print $2, $3}' \
| sort | uniq -cExpected (numbers vary):
NAME READY STATUS RESTARTS AGE
web-7d94ccc84d-5rwlr 2/2 Running 0 57s
58 /api/notes 200
28 /flaky 200
1 /flaky 500
Save your progress? Sign up for free
Round 2: Faster, but Broken
Situation: A colleague read that opaque connections
have “less overhead” and used them to speed up helper.
Since then, /flaky has been completely dead. Make
/flaky as reliable as before.
Referee commands:
kubectl -n tag18 annotate service helper config.linkerd.io/opaque-ports=80Target time: 5 min
Hint: The helper proxy rejects
web, even though web is allowed. Ask the
web proxy which protocol it assumes for the Service.
Solution to Round 2
sleep 60
kubectl -n tag18 logs deploy/last -c last --since=60s | awk '{print $2, $3}' \
| sort | uniq -c
linkerd viz stat -n tag18 deploy
kubectl -n tag18 logs deploy/web -c web --since=30s | grep error | tail -1 \
| grep -oE 'upstream prematurely[^,]*'Expected (abridged, numbers vary):
86 /api/notes 200
43 /flaky 502
NAME MESHED SUCCESS RPS LATENCY_P50 LATENCY_P95 LATENCY_P99 ...
helper 1/1 100.00% 0.4rps 1ms 10ms 10ms ...
last 1/1 100.00% 0.3rps 1ms 19ms 20ms ...
notes-api 1/1 100.00% 1.8rps 3ms 110ms 182ms ...
web 1/1 70.67% 2.5rps 18ms 96ms 175ms ...
upstream prematurely closed connection while reading response header from upstream
helper only sees the probes (0.4 rps); the HTTP requests
from web no longer arrive as HTTP at all. nginx reports a
connection that closes without a response. Who closes it?
kubectl -n tag18 logs deploy/helper -c linkerd-proxy --since=1m \
| grep 'Connection denied' | tail -1 \
| grep -oE 'policy::[a-z]+|server.name=[^ ]*|Name\("[^"]*'
linkerd diagnostics policy -n tag18 svc/helper 80 -o json \
| jq -c '.protocol.Kind | keys'
kubectl -n tag18 get service helper -o jsonpath='{.metadata.annotations}{"\n"}' \
| cut -c1-80
kubectl -n tag18 get server helper-http -o jsonpath='{.spec.proxyProtocol}{"\n"}'Expected:
policy::tcp
server.name=helper-http
Name("web.tag18.serviceaccount.identity.linkerd.cluster.local
["Opaque"]
{"config.linkerd.io/opaque-ports":"80","kubectl.kubernetes.io/last-applied-confi
HTTP/1
The client proxy now sends the traffic as an opaque TCP stream. But
the Server helper-http is declared as HTTP/1 –
the TCP check (policy::tcp) closes the connection, even
though the identity web is correct. And even if it got
through: opaque also means no routes, no retries, no HTTP metrics. HTTP
never belongs on an opaque port:
kubectl -n tag18 annotate service helper config.linkerd.io/opaque-ports-
sleep 5
linkerd diagnostics policy -n tag18 svc/helper 80 -o json \
| jq -c '.protocol.Kind | keys'
sleep 55
kubectl -n tag18 logs deploy/last -c last --since=50s | awk '{print $2, $3}' \
| sort | uniq -cExpected (numbers vary):
service/helper annotated
["Detect"]
62 /api/notes 200
31 /flaky 200
1 /flaky 500
Save your progress? Sign up for free
Round 3: The Hardening
Situation: The security department wants
notes-api to receive read-only access only. To do so, a
colleague built a route and moved the existing rule onto it. Since then,
/api/notes no longer works. Fix it so that the hardening
stays: POST must be rejected.
Referee commands:
kubectl apply -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: notes-api-lesen, namespace: tag18}
spec:
parentRefs:
- {group: policy.linkerd.io, kind: Server, name: notes-api-http}
rules:
- matches:
- method: GET
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: notes-api-von-web, namespace: tag18}
spec:
targetRef: {group: gateway.networking.k8s.io, kind: HTTPRoute, name: notes-api-lesen}
requiredAuthenticationRefs:
- {group: policy.linkerd.io, kind: MeshTLSAuthentication, name: notes-api}
EOFTarget time: 7 min
Hint: Two faults. Measure again right after the first fix – and remember what Day 6 says about routes on Servers.
Solution to Round 3
sleep 20
kubectl -n tag18 logs deploy/last -c last --since=15s | awk '{print $2, $3}' \
| sort | uniq -c
sleep 50
kubectl -n tag18 logs deploy/last -c last --since=20s | awk '{print $2, $3}' \
| sort | uniq -c
kubectl -n tag18 get pods -l app=notes-apiExpected (abridged):
24 /api/notes 403
12 /flaky 200
30 /api/notes 503
15 /flaky 200
1 /flaky 500
NAME READY STATUS RESTARTS AGE
notes-api-6ff48b7779-mmss9 1/2 Running 0 31m
First 403, half a minute later 503 and a
Pod that is no longer Ready. Who is being rejected?
kubectl -n tag18 get events --field-selector reason=Unhealthy \
--sort-by=.lastTimestamp | tail -1
kubectl -n tag18 logs deploy/notes-api -c linkerd-proxy --since=1m \
| grep 'Request denied' \
| grep -oE 'route.name=[^ ]*|client.ip=[^ ]*|Name\("[^"]*|client.tls=None' \
| paste - - - | sort | uniq -c
linkerd viz authz -n tag18 deploy/notes-api | grep notes-api-lesen
kubectl -n tag18 get authorizationpolicy notes-api-von-web \
-o jsonpath='{.spec.requiredAuthenticationRefs}{"\n"}'Expected (abridged):
49s Warning Unhealthy pod/notes-api-6ff48b7779-mmss9 Readiness probe failed: HTTP
probe failed with statuscode: 403
17 route.name=notes-api-lesen Name("web.tag18.serviceaccount.identity.lin...
7 route.name=notes-api-lesen client.tls=None client.ip=10.0.2.241
notes-api-lesen notes-api-http 0.3rps 0.00% ...
notes-api-lesen notes-api-http authorizationpolicy/notes-api-von-web - - ...
[{"group":"policy.linkerd.io","kind":"MeshTLSAuthentication","name":"notes-api"}]
Fault 1: The rule requires the identity notes-api –
copied from postgres-von-notes-api. Fault 2: As soon as a
route is attached to the Server, Linkerd no longer allows the probes
automatically. The kubelet (10.0.2.241, without TLS) gets
403, the Pod drops out of the Service, web
gets 503 (Day 6, Challenge). First the identity:
kubectl -n tag18 patch authorizationpolicy notes-api-von-web --type=json \
-p '[{"op":"replace","path":"/spec/requiredAuthenticationRefs/0/name","value":"web"}]'
sleep 15
kubectl -n tag18 logs deploy/last -c last --since=10s | awk '{print $2, $3}' \
| sort | uniq -cExpected:
14 /api/notes 503
6 /flaky 200
1 /flaky 500
Still 503 – after all, the Pod is not Ready. Then the
probe, with its own route and the kubelet addresses of the nodes (Day
6):
cat > probe.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: notes-api-probe, namespace: tag18}
spec:
parentRefs:
- {group: policy.linkerd.io, kind: Server, name: notes-api-http}
rules:
- matches:
- path: {type: Exact, value: /healthz}
method: GET
---
apiVersion: policy.linkerd.io/v1alpha1
kind: NetworkAuthentication
metadata: {name: kubelet, namespace: tag18}
spec:
networks:
- cidr: 10.0.0.88/32
- cidr: 10.0.1.116/32
- cidr: 10.0.2.241/32
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: notes-api-probe-von-kubelet, namespace: tag18}
spec:
targetRef: {group: gateway.networking.k8s.io, kind: HTTPRoute, name: notes-api-probe}
requiredAuthenticationRefs:
- {group: policy.linkerd.io, kind: NetworkAuthentication, name: kubelet}
EOF
kubectl apply -f probe.yaml
sleep 40
kubectl -n tag18 get pods -l app=notes-api
kubectl -n tag18 logs deploy/last -c last --since=20s | awk '{print $2, $3}' \
| sort | uniq -c
kubectl -n tag18 exec deploy/web -c web -- curl -s -o /dev/null -w '%{http_code}\n' \
-X POST -H 'X-API-Key: lab-key-123' -d '{"text":"x"}' http://notes-api/notesExpected:
httproute.gateway.networking.k8s.io/notes-api-probe created
networkauthentication.policy.linkerd.io/kubelet created
authorizationpolicy.policy.linkerd.io/notes-api-probe-von-kubelet created
NAME READY STATUS RESTARTS AGE
notes-api-6ff48b7779-mmss9 2/2 Running 0 32m
30 /api/notes 200
15 /flaky 200
404
GET works, POST matches no route and gets
404 – the hardening holds. The exact route
/healthz beats the general GET route, so the
probe is governed by its own rule.
Save your progress? Sign up for free
Round 4: Made Robust
Situation: To make notes-api “more
robust”, the team added retries and a generous timeout of five seconds.
Since then, every call fails. Keep the retries and the five-second
timeout.
Referee commands:
kubectl apply -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: notes-api-robust
namespace: tag18
annotations:
retry.linkerd.io/http: 5xx
retry.linkerd.io/limit: "2"
timeout.linkerd.io/request: 5ms
spec:
parentRefs:
- {group: core, kind: Service, name: notes-api, port: 80}
rules:
- matches:
- path: {type: PathPrefix, value: /}
timeouts: {request: 5s}
EOFTarget time: 4 min
Hint: For errors of its own, the proxy writes a header stating the reason.
Solution to Round 4
sleep 60
kubectl -n tag18 logs deploy/last -c last --since=50s | awk '{print $2, $3}' \
| sort | uniq -c
kubectl -n tag18 exec deploy/web -c web -- curl -si -H 'X-API-Key: lab-key-123' \
http://notes-api/notes | grep -E '^HTTP|^l5d'Expected:
64 /api/notes 504
30 /flaky 200
2 /flaky 500
HTTP/1.1 504 Gateway Timeout
l5d-proxy-error: logical service 10.109.85.58:80: route HTTPRoute.tag18.notes-api-robus
t: backend Service.tag18.notes-api:80: Service.tag18.notes-api:80: timed out waiting f
or response headers: request timeout: 5ms
l5d-proxy-error names the route and the reason: five
milliseconds. Which setting wins is shown by the
proxy’s view; the rate per route by stat-outbound (Day
7):
linkerd diagnostics policy -n tag18 svc/notes-api 80 -o json \
| jq -c '.protocol.Kind.Detect.http1.routes[]
| {name: .metadata.Kind.Resource.name, timeouts: .rules[0].timeouts}'
kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090 \
>/dev/null 2>&1 &
sleep 4
PROM=--prometheusURL=http://localhost:9090
linkerd viz stat-outbound -n tag18 deploy/web $PROM | grep -A1 notes-api
kill %1Expected (abridged):
{"name":"notes-api-robust","timeouts":{"request":{"nanos":5000000}}}
web notes-api:80 notes-api-robust HTTPRoute ... 0.00% ... 100.00% 0.00%
└──────────────► notes-api:80 ... 0.00% ... 100.00%
The annotation beats rules[].timeouts.request (Day 7).
The TIMEOUTS column shows 100 percent, RETRIES
0: without retry.linkerd.io/timeout, an expired timeout is
not retried. Remove the annotation; the rule with 5 s stays:
kubectl -n tag18 annotate httproute notes-api-robust timeout.linkerd.io/request-
sleep 5
linkerd diagnostics policy -n tag18 svc/notes-api 80 -o json \
| jq -c '.protocol.Kind.Detect.http1.routes[]
| {name: .metadata.Kind.Resource.name, timeouts: .rules[0].timeouts}'
sleep 50
kubectl -n tag18 logs deploy/last -c last --since=45s | awk '{print $2, $3}' \
| sort | uniq -cExpected (numbers vary):
httproute.gateway.networking.k8s.io/notes-api-robust annotated
{"name":"notes-api-robust","timeouts":{"request":{"seconds":5}}}
69 /api/notes 200
33 /flaky 200
1 /flaky 500
Save your progress? Sign up for free
Round 5: The Brake
Situation: To guard against noisy clients,
notes-api gets a rate limit: at most one request per second
per identity, ten for web as the most important customer.
The template comes from notes. Since then, every second
request of the load generator is rejected. The limit is supposed to
stay.
Referee commands:
kubectl apply -f - <<'EOF'
apiVersion: policy.linkerd.io/v1alpha1
kind: HTTPLocalRateLimitPolicy
metadata: {name: notes-api-rl, namespace: tag18}
spec:
targetRef: {group: policy.linkerd.io, kind: Server, name: notes-api-http}
total:
requestsPerSecond: 50
identity:
requestsPerSecond: 1
overrides:
- requestsPerSecond: 10
clientRefs:
- {kind: ServiceAccount, name: web, namespace: notes}
EOFTarget time: 4 min
Hint: Which bucket is empty – total or
identity?
Solution to Round 5
sleep 60
kubectl -n tag18 logs deploy/last -c last --since=50s | awk '{print $2, $3}' \
| sort | uniq -c
kubectl -n tag18 logs deploy/notes-api -c linkerd-proxy --since=1m \
| grep -oE '[a-z]+ rate limit exceeded: [0-9]+rps' | sort | uniq -c
linkerd diagnostics proxy-metrics -n tag18 deploy/notes-api \
| grep '^inbound_http_local_ratelimit' \
| grep -oE 'ratelimit_scope="[^"]*"|client_id="[a-z.0-9-]+|\} [0-9]+' | paste - - -
kubectl -n tag18 get httplocalratelimitpolicy notes-api-rl \
-o jsonpath='{.spec.overrides}{"\n"}'Expected (abridged, numbers vary):
39 /api/notes 200
39 /api/notes 429
37 /flaky 200
1 /flaky 500
96 identity rate limit exceeded: 1rps
ratelimit_scope="identity" client_id="web.tag18.serviceaccount.identity.linker... } 58
[{"clientRefs":[{"kind":"ServiceAccount","name":"web","namespace":"notes"}],"reque...
Rejected in the scope identity, specifically
web.tag18 – but the override applies to the ServiceAccount
web in notes. For web from
tag18, the limit for everyone therefore applies: one
request per second, while last sends around 1.5. Without
namespace, the namespace of the policy applies; here it is
set explicitly:
kubectl -n tag18 patch httplocalratelimitpolicy notes-api-rl --type=json \
-p '[{"op":"replace","path":"/spec/overrides/0/clientRefs/0/namespace",
"value":"tag18"}]'
sleep 60
kubectl -n tag18 logs deploy/last -c last --since=50s | awk '{print $2, $3}' \
| sort | uniq -c
kubectl -n tag18 logs deploy/notes-api -c linkerd-proxy --since=40s \
| grep -c 'rate limit exceeded'Expected (numbers vary):
httplocalratelimitpolicy.policy.linkerd.io/notes-api-rl patched
75 /api/notes 200
36 /flaky 200
1 /flaky 500
0
Save your progress? Sign up for free
Round 6: The Release
Situation: Version 2 of notes-api goes
live as a canary, with half of the traffic. The team reports: “Works,
just a few percent errors, surely that’s the retries.” Bring
/api/notes to 100 percent without deleting version 2 – the
team wants to debug it.
Referee commands:
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata: {name: v2-conf, namespace: tag18}
data:
default.conf: |
server {
listen 8080;
location = /healthz { return 200 "ok\n"; }
location / { return 500 "v2 kaputt\n"; }
}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: notes-api-v2, namespace: tag18}
spec:
replicas: 1
selector: {matchLabels: {app: notes-api-v2}}
template:
metadata: {labels: {app: notes-api-v2}}
spec:
serviceAccountName: notes-api
containers:
- name: notes-api
image: docker.io/nginxinc/nginx-unprivileged:1.31.7-alpine
ports: [{containerPort: 8080, name: http}]
readinessProbe:
httpGet: {path: /healthz, port: http}
volumeMounts: [{name: conf, mountPath: /etc/nginx/conf.d}]
volumes: [{name: conf, configMap: {name: v2-conf}}]
---
apiVersion: v1
kind: Service
metadata: {name: notes-api-v2, namespace: tag18}
spec:
selector: {app: notes-api-v2}
ports: [{port: 80, targetPort: http, name: http}]
---
apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata: {name: notes-api-v2-http, namespace: tag18}
spec:
podSelector: {matchLabels: {app: notes-api-v2}}
port: http
proxyProtocol: HTTP/1
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: notes-api-v2-von-web, namespace: tag18}
spec:
targetRef: {group: policy.linkerd.io, kind: Server, name: notes-api-v2-http}
requiredAuthenticationRefs:
- {group: policy.linkerd.io, kind: MeshTLSAuthentication, name: web}
EOF
kubectl -n tag18 rollout status deploy/notes-api-v2 --timeout=120s
kubectl -n tag18 patch httproute notes-api-robust --type=json -p '[{"op":"add",
"path":"/spec/rules/0/backendRefs","value":[{"name":"notes-api","port":80,
"weight":50},{"name":"notes-api-v2","port":80,"weight":50}]}]'Target time: 5 min
Hint: stat-outbound shows every backend
of a route separately.
Solution to Round 6
sleep 90
kubectl -n tag18 logs deploy/last -c last --since=80s | awk '{print $2, $3}' \
| sort | uniq -c
linkerd viz stat -n tag18 deploy | grep -E 'NAME|notes-api'Expected (abridged, numbers vary):
101 /api/notes 200
11 /api/notes 500
55 /flaky 200
1 /flaky 500
NAME MESHED SUCCESS RPS LATENCY_P50 LATENCY_P95 LATENCY_P99 ...
notes-api 1/1 100.00% 1.7rps 5ms 32ms 38ms ...
notes-api-v2 1/1 28.57% 1.4rps 1ms 6ms 9ms ...
Around ten percent errors – and notes-api-v2 with 28
percent success, and that only thanks to the probes. Per backend of the
route:
kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090 \
>/dev/null 2>&1 &
sleep 4
PROM=--prometheusURL=http://localhost:9090
linkerd viz stat-outbound -n tag18 deploy/web $PROM | grep -A2 notes-api
kill %1
kubectl -n tag18 get httproute notes-api-robust \
-o jsonpath='{range .spec.rules[0].backendRefs[*]}{.name} {.weight}{"\n"}{end}'Expected (abridged):
web notes-api:80 notes-api-robust HTTPRoute 90.70% 1.43 ... 0.00% 37.23%
├──────────────► notes-api:80 100.00% 1.30 ... 0.00%
└──────────────► notes-api-v2:80 0.00% 0.98 ... 0.00%
notes-api 50
notes-api-v2 50
Version 2 fails 100 percent of the time. That the route still reaches 90 percent is due to the retries from Round 4: a failure on v2 is retried and lands on v1 with a probability of 50 percent. Only when all three attempts hit v2 (0.5³ = 12.5 percent) does the user see the error. Retries hide a broken canary. Roll back via the weight (Day 9):
kubectl -n tag18 patch httproute notes-api-robust --type=json -p '[{"op":"replace",
"path":"/spec/rules/0/backendRefs/1/weight","value":0}]'
sleep 75
kubectl -n tag18 logs deploy/last -c last --since=65s | awk '{print $2, $3}' \
| sort | uniq -c
kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090 \
>/dev/null 2>&1 &
sleep 4
linkerd viz stat-outbound -n tag18 deploy/web $PROM -t 30s | grep -A2 notes-api
kill %1Expected (abridged, numbers vary):
102 /api/notes 200
50 /flaky 200
1 /flaky 500
web notes-api:80 notes-api-robust HTTPRoute 100.00% 1.50 ... 0.00% 0.00%
├──────────────► notes-api:80 100.00% 1.50 ... 0.00%
└──────────────► notes-api-v2:80 NaN% 0.00 ... NaN%
v2 keeps running and no longer gets anything. Had you deleted the v2 Deployment instead, you would also have seen 100 percent (a backend without endpoints drops out of the distribution) – but that is exactly what the team did not want.
Save your progress? Sign up for free
Round 7: The Firewall in the Mesh
Situation: The security department blocks all
outgoing traffic from tag18. Only example.com
via HTTPS is supposed to be allowed; notes-api needs it for
/api/upstream. But that now returns 502. Allow
exactly example.com – nothing else.
Referee commands:
kubectl apply -f - <<'EOF'
apiVersion: policy.linkerd.io/v1alpha1
kind: EgressNetwork
metadata: {name: internet, namespace: tag18}
spec:
trafficPolicy: Deny
---
apiVersion: gateway.networking.k8s.io/v1
kind: TLSRoute
metadata: {name: example-com, namespace: tag18}
spec:
parentRefs:
- {group: policy.linkerd.io, kind: EgressNetwork, name: internet, port: 443}
hostnames: [www.example.com]
rules:
- backendRefs:
- {group: policy.linkerd.io, kind: EgressNetwork, name: internet}
EOFTarget time: 5 min
Hint: Compare the name that notes-api
calls with the name that the route allows.
Solution to Round 7
kubectl -n tag18 exec deploy/last -c last -- curl -s -w ' %{http_code}\n' \
http://web/api/upstream
linkerd diagnostics proxy-metrics -n tag18 deploy/notes-api \
| grep -E '^outbound_t(cp|ls)_route_open_total' \
| grep -oE 'parent_name="[^"]*"|route_name="[^"]*"|\} [0-9]+' | paste - - -Expected (abridged):
{"error":"Get \"https://example.com\": EOF"} 502
parent_name="postgres" route_name="opaq" } 70
parent_name="egress-fallback" route_name="egress-fallback" } 1
parent_name="internet" route_name="tls-egress-deny" } 3
The first connection (baseline) still went through
egress-fallback; now everything lands on the default route
tls-egress-deny. Your own TLSRoute does not count at all.
Why?
kubectl -n tag18 get tlsroute example-com \
-o jsonpath='{.spec.hostnames} {.status.parents[0].conditions[0].reason}{"\n"}'
kubectl -n tag18 set env deploy/notes-api -c notes-api --list | grep UPSTREAMExpected:
["www.example.com"] Accepted
NOTES_UPSTREAM=https://example.com
The route is Accepted but does not match: the SNI name
must match exactly (Day 11). www.example.com is a different
host than example.com.
kubectl -n tag18 patch tlsroute example-com --type=merge \
-p '{"spec":{"hostnames":["example.com"]}}'
sleep 5
kubectl -n tag18 exec deploy/last -c last -- curl -s -w ' %{http_code}\n' \
http://web/api/upstream
for u in https://example.com https://linkerd.io; do
kubectl -n tag18 exec deploy/last -c last -- curl -s -o /dev/null \
-w "$u %{http_code}\n" $u
done
linkerd diagnostics proxy-metrics -n tag18 deploy/notes-api \
| grep -E '^outbound_tls_route_open_total' \
| grep -oE 'route_name="[^"]*"|\} [0-9]+' | paste - -Expected:
tlsroute.gateway.networking.k8s.io/example-com patched
{"status":200,"url":"https://example.com"}
200
https://example.com 200
https://linkerd.io 000
command terminated with exit code 35
route_name="example-com" } 1
route_name="tls-egress-deny" } 3
example.com works, linkerd.io stays blocked
– for last too, because the EgressNetwork applies to all
clients in the namespace.
Save your progress? Sign up for free
Round 8: The Auditors Ask
Situation: You don’t break certificates in a competition – that would hit the entire mesh (Day 16). Instead, the auditors want three answers, each backed by a command. Read only, change nothing.
- What identity does
webhave intag18, who issued it, and until when is the certificate valid? - When does the issuer expire, and when is it renewed? Do Pods have to restart for that?
- Is exactly the trust anchor from
~/linkerd/tag05/ca.crtdistributed in the cluster? What doeslinkerd checksay about it?
Target time: 4 min
Hint: Day 3 and Day 5, speed drills.
Solution to Round 8
linkerd identity -n tag18 -l app=web | grep -E 'Subject:|Issuer:|Not After'
kubectl -n linkerd get certificate linkerd-identity-issuer \
-o jsonpath='{.status.notAfter} {.status.renewalTime}{"\n"}'
kubectl -n linkerd get configmap linkerd-identity-trust-roots \
-o jsonpath='{.data.ca-bundle\.crt}' | step certificate fingerprint -
step certificate fingerprint ~/linkerd/tag05/ca.crt
linkerd check 2>&1 | grep -A2 '‼ issuer'Expected (dates depend on your run):
Issuer: CN=identity.linkerd.cluster.local
Not After : Nov 20 20:40:11 2026 UTC
Subject: CN=web.tag18.serviceaccount.identity.linkerd.cluster.local
2026-11-21T16:02:00Z 2026-11-20T15:02:00Z
8b44a80f0940a58823716aecd2aeac2895e43d94bd968826ec821389f65bab81
8b44a80f0940a58823716aecd2aeac2895e43d94bd968826ec821389f65bab81
‼ issuer cert is valid for at least 60 days
issuer certificate will expire on 2026-11-21T16:02:00Z
see https://linkerd.io/2/checks/#l5d-identity-issuer-cert-not-expiring-soon ...
web.tag18.serviceaccount.identity.linkerd.cluster.local, issued by the issueridentity.linkerd.cluster.local, valid for 24 hours from Pod start – the proxy renews it itself.- The issuer is valid for 48 hours; cert-manager renews it 25 hours
before expiry.
identityreloads it without a restart (IssuerUpdated), and the proxies get their next certificates from the new issuer – no Pod has to restart. - Yes, both fingerprints are identical. The warning from
checkis expected with a 48-hour issuer (Day 5) and is not a finding.
Save your progress? Sign up for free
Scoring
| Round | Fault | Target time | Points |
|---|---|---|---|
| 1 | linkerd.io/inject: disabled on the Pod template of
web |
3 min | 2 |
| 2 | opaque port on the HTTP Service helper |
5 min | 2 |
| 3 | wrong identity in the rule + probes without a route | 7 min | 3 |
| 4 | timeout annotation 5ms beats
timeouts.request |
4 min | 2 |
| 5 | rate limit override for an SA from the wrong namespace | 4 min | 2 |
| 6 | canary at 50 percent on a broken version, hidden by retries | 5 min | 2 |
| 7 | TLSRoute for www.example.com instead of
example.com |
5 min | 2 |
| 8 | three questions on identity, issuer, trust anchor | 4 min | 3 |
You only get full points within the target time, without a hint
and with a proof command; otherwise half (round down).
In Round 8 there is one point per backed-up answer. Whoever removes the
symptom instead of the cause (default-deny off,
accessPolicy: all-unauthenticated, deleting the hardening
or the rate limit, deleting the route instead of keeping the retries,
deleting v2, trafficPolicy: Allow) gets 0 for the round.
Overall target time: 37 min, maximum: 18 points.
| Points | Assessment |
|---|---|
| 15–18 | ready for the final exam (Appendix B) |
| 10–14 | good; repeat the days of the rounds you missed |
| 0–9 | work through Days 2, 6, 7, 8, 9 and 11 once more |
Challenge
After a change to notes-api, the rollout is stuck. The
users notice nothing, the load generator shows 100 percent. Finish the
rollout without loosening the rules for postgres. Referee
commands:
kubectl -n tag18 patch deploy notes-api -p '{"spec":{"template":{"metadata":
{"annotations":{"config.beta.linkerd.io/proxy-enable-native-sidecar":"false"}}}}}'Hint: Look at the order of the init containers in the new Pod.
Solution
sleep 60
kubectl -n tag18 get pods -l app=notes-api
P=$(kubectl -n tag18 get pod -l app=notes-api \
--sort-by=.metadata.creationTimestamp -o name | tail -1)
kubectl -n tag18 get $P \
-o jsonpath='{range .spec.initContainers[*]}{.name} {.restartPolicy}{"\n"}{end}'
kubectl -n tag18 logs $P -c migrate --tail=1 | cut -c1-80
linkerd diagnostics proxy-metrics -n tag18 po/postgres-0 \
| grep '^inbound_tcp_authz_deny_total' \
| grep -oE '(srv_name|no_tls_reason)="[^"]*"|\} [0-9]+' | paste - - -
kubectl -n tag18 get deploy notes-api \
-o jsonpath='{.spec.template.metadata.annotations}{"\n"}'Expected (abridged):
NAME READY STATUS RESTARTS AGE
notes-api-54c8fb75bf-xdxbt 0/2 Init:Error 3 (43s ago) 60s
notes-api-6ff48b7779-mmss9 2/2 Running 0 33m
migrate
linkerd-init
time=2026-11-19T21:10:10.890Z level=ERROR msg="database unreachable" source=env
srv_name="postgres" no_tls_reason="port_skipped" } 7
{"config.beta.linkerd.io/proxy-enable-native-sidecar":"false"}
migrate runs before linkerd-init, and there
is no longer a proxy as a native sidecar
(restartPolicy: Always). The migration talks plaintext to
Postgres, and default-deny closes the connection
(port_skipped, Days 2 and 6). The annotation has to go:
A='config.beta.linkerd.io~1proxy-enable-native-sidecar'
kubectl -n tag18 patch deploy notes-api --type=json -p '[{"op":"remove",
"path":"/spec/template/metadata/annotations/'$A'"}]'
kubectl -n tag18 rollout status deploy/notes-api --timeout=180s
kubectl -n tag18 get pods -l app=notes-apiExpected:
deployment "notes-api" successfully rolled out
NAME READY STATUS RESTARTS AGE
notes-api-6ff48b7779-mmss9 2/2 Running 0 34m
The template is identical to the old one again, so Kubernetes simply scales the old ReplicaSet back up – the same Pod keeps running.
Save your progress? Sign up for free
Cleanup
kubectl delete namespace tag18 --timeout=180s
kubectl get endpointslices -A | grep cilium-gateway | grep tag18Expected: namespace "tag18" deleted,
then no output. Together with the namespace, the PVC, the EgressNetwork
and the EndpointSlices cilium-gateway-helper and
cilium-gateway-notes-api, which Cilium created for the
HTTPRoutes on the Services, disappear. The files in
~/linkerd/tag18 remain as a template for the final
exam.
Checkpoint
kubectl get ns | grep -c tag18
kubectl get pv | grep -c tag18Expected: 0 twice.
kubectl -n notes get podsExpected: all eight Pods 2/2 Running –
you did not touch notes today.
kubectl -n notes logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
| sort | uniq -c
linkerd check 2>&1 | tail -1Expected (numbers vary):
86 /api/notes 200
29 /flaky 200
14 /flaky 500
Status check results are √
Interview Note
- A service suddenly returns 403 – where do you start? At the
server proxy: the proxy log
Request deniedshows the Server, route and client identity.client.tls=Nonemeans: client without a proxy. A wrong identity means: check the ServiceAccount or the rule. - Why does a broken canary look harmless in the dashboard?
The client proxy’s retries repeat failures on the other backend. The
route is almost green while the backend sits at 0 percent – read
stat-outboundper backend, not just the route. - What do you check after every policy or route change?
Readiness of the target Pods (probes on Servers with routes), the
proxy’s view with
linkerd diagnostics policy, and a measurement with fresh connections.
Enterprise Note
- Play game days like this one regularly with the on-call team – the solutions become runbooks.
- Allow or forbid annotations that affect the mesh
(
inject,opaque-ports,proxy-enable-native-sidecar, timeouts) via admission policy, not via review. - Manage policies, routes and rate limits from Git with review and a test namespace that has default-deny like production. Check copied templates for namespace and identities.
- Evaluate canary releases automatically (Flagger, Argo Rollouts) – with the error rate per backend and the retry rate as abort criteria.
- Maintain egress allowances as a list of hostnames and cross-check
them with metrics (
tls-egress-deny) to see which blocked targets are being requested.