Skip to content
Hands-on IT IT Drills

Day 18: Competition – The Broken Mesh

Goal: In eight rounds you find and fix planted mesh faults from across the book – against the clock, with proof before and after. · Duration: approx. 90 min · Prerequisite: Days 0–17

Briefing

  • In tag18 you build a complete copy of notes: in the mesh, with default-deny, the rules from Day 6 and the retries from Day 7.
  • Eight rounds, all topics: injection, opaque ports, authorization, timeouts, rate limits, traffic shifting, egress and certificates. The rounds build on each other: play them in order.
  • Tools: load generator, linkerd viz stat/authz/stat-outbound, linkerd diagnostics policy/proxy-metrics, linkerd identity, proxy logs, events.
  • Every round ends with a proof command: broken before, fixed after.
Round  Symptom                                   Target time
1      Everything 403, only /flaky 500           3 min
2      /flaky only 502                           5 min
3      First 403, then 503 on /api/notes         7 min
4      /api/notes entirely 504                   4 min
5      Every second call 429                     4 min
6      A few percent 500 after a release         5 min
7      /api/upstream fails                       5 min
8      Questions from the auditors (read only)   4 min

Rules of Play

  • Per round, the referee runs the referee commands – a colleague, or you yourself without looking: copy the block, paste it into a second terminal, close the window. Then start the clock.
  • Find the cause, fix it, prove it with one command. Only then stop the clock.
  • Fix the cause, not the symptom: whoever turns off default-deny, deletes a rule instead of correcting it, opens policies with all-unauthenticated or deletes Pods that are not broken gets zero points.
  • Only read the Hint once the target time has run out.
  • As always, the load generator is your measuring instrument. The goal of every round: back to 100 percent 200 on /api/notes, /flaky almost only 200 (retries), unless the round says otherwise. In the following, the command for this is simply called measurement:
kubectl -n tag18 logs deploy/last -c last --since=40s | awk '{print $2, $3}' \
  | sort | uniq -c

Playing Field

The manifests from Day 0 and the rules from Days 6 and 7 are already on cp1. sed rewrites them to the new namespace and halves the replicas – one Pod per service is enough for the competition.

mkdir -p ~/linkerd/tag18 && cd ~/linkerd/tag18
kubectl create namespace tag18
kubectl annotate namespace tag18 linkerd.io/inject=enabled \
  config.linkerd.io/default-inbound-policy=deny
kubectl -n tag18 create secret generic notes-db \
  --from-literal=password=lab-db-123 \
  --from-literal=url='postgres://notes:lab-db-123@postgres:5432/notes?sslmode=disable'
kubectl -n tag18 create secret generic notes-api-key --from-literal=key=lab-key-123
for a in web notes-api helper last postgres; do
  kubectl -n tag18 create serviceaccount $a
done
for f in tag06/servers tag06/authz tag06/admin tag07/route-retry tag07/route-rest; do
  sed 's/namespace: notes/namespace: tag18/; s/namespace: tag07/namespace: tag18/' \
    ~/linkerd/$f.yaml; echo ---
done > policy.yaml
for f in postgres notes-api helper web last; do
  sed 's/namespace: notes/namespace: tag18/; s/\.notes\.svc/.tag18.svc/
    s/replicas: 2/replicas: 1/' ~/linkerd/tag00/$f.yaml; echo ---
done > app.yaml
grep -c '^kind:' policy.yaml app.yaml
kubectl apply -f policy.yaml -f app.yaml | wc -l

Expected (abridged):

namespace/tag18 created
...
policy.yaml:16
app.yaml:10
26

Now, as on Day 2, the dedicated ServiceAccounts and the opaque port, plus a target on the internet for notes-api (Day 11):

kubectl -n tag18 annotate service postgres config.linkerd.io/opaque-ports=5432
kubectl -n tag18 set serviceaccount statefulset/postgres postgres
for d in web notes-api helper last; do
  kubectl -n tag18 set serviceaccount deploy/$d $d
done
kubectl -n tag18 set env deploy/notes-api NOTES_UPSTREAM=https://example.com
kubectl -n tag18 rollout status statefulset/postgres --timeout=180s
kubectl -n tag18 rollout status deploy --timeout=240s
sleep 60
kubectl -n tag18 get pods

Expected (abridged):

NAME                         READY   STATUS    RESTARTS   AGE
helper-79d57fbbbb-kn8rl      2/2     Running   0          100s
last-78895758f9-br8k7        2/2     Running   0          99s
notes-api-6ff48b7779-mmss9   2/2     Running   0          99s
postgres-0                   2/2     Running   0          84s
web-7d94ccc84d-9k2bv         2/2     Running   0          102s

The first Pods still start with the ServiceAccount default and briefly fail on default-deny – after set serviceaccount they are replaced. The baseline:

kubectl -n tag18 logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
  | sort | uniq -c
linkerd viz stat -n tag18 deploy
kubectl -n tag18 exec deploy/last -c last -- curl -s http://web/api/upstream; echo

Expected (abridged, numbers vary):

     80 /api/notes 200
     38 /flaky 200
      1 /flaky 500
NAME        MESHED   SUCCESS      RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99 ...
helper         1/1    78.57%   1.4rps           3ms          12ms          18ms ...
last           1/1   100.00%   0.3rps           1ms          19ms          20ms ...
notes-api      1/1   100.00%   1.8rps           4ms          20ms          90ms ...
web            1/1   100.00%   2.4rps          15ms          86ms          97ms ...
{"status":200,"url":"https://example.com"}

This is the healthy playing field. Remember the numbers.

Round 1: Everything Locked

Situation: After a “small update” to web, the load generator reports nothing but errors. Supposedly nobody touched the policies. Bring web back.

Referee commands:

kubectl -n tag18 patch deploy web -p '{"spec":{"template":{"metadata":
  {"annotations":{"linkerd.io/inject":"disabled"}}}}}'
kubectl -n tag18 rollout status deploy/web --timeout=120s

Target time: 3 min

Hint: Count the containers in the web Pod.

Solution to Round 1

sleep 50
kubectl -n tag18 logs deploy/last -c last --since=40s | awk '{print $2, $3}' \
  | sort | uniq -c
kubectl -n tag18 get pods

Expected (abridged):

     58 /api/notes 403
     28 /flaky 500
NAME                         READY   STATUS    RESTARTS   AGE
helper-79d57fbbbb-kn8rl      2/2     Running   0          2m44s
...
web-5f7fb96f68-4nq9w         1/1     Running   0          53s

web has only one container – no proxy. The proof from the other side:

kubectl -n tag18 logs deploy/notes-api -c linkerd-proxy --since=1m \
  | grep 'Request denied' | tail -1 \
  | grep -oE 'server.name=[^ ]*|client.tls=[^ ]*|client.ip=[^ ]*'
kubectl -n tag18 get pod -l app=web -o wide
kubectl -n tag18 get deploy web \
  -o jsonpath='{.spec.template.metadata.annotations}{"\n"}'

Expected (abridged):

server.name=notes-api-http
client.tls=None(NoClientHello)
client.ip=10.0.1.179
NAME                   READY   STATUS    RESTARTS   AGE   IP           NODE      ...
web-5f7fb96f68-4nq9w   1/1     Running   0          77s   10.0.1.179   worker1   ...
{"linkerd.io/inject":"disabled"}

No proxy, no mTLS; no mTLS, no identity – default-deny and the rule notes-api-von-web reject the request (Day 6). And why 500 instead of 403 on /flaky?

kubectl -n tag18 exec deploy/web -c web -- curl -si \
  'http://helper.tag18.svc.cluster.local/unstable?failure_rate=0.3' \
  | grep -iE '^HTTP|^server'

Expected:

HTTP/1.1 500 Internal Server Error
server: envoy

server: envoy: HTTPRoutes are attached to helper (retries from Day 7). Unmeshed clients therefore end up at Cilium, not at Linkerd (Day 9). But why 500? A look at Cilium’s Envoy configuration and at the other side:

kubectl -n tag18 get ciliumenvoyconfig helper \
  -o jsonpath='{..directResponse.status}{"\n"}'
kubectl -n tag18 logs deploy/helper -c linkerd-proxy --since=1m \
  | grep -c 'Request denied'

Expected:

500 500
0

Envoy answers both routes itself with 500 (directResponse, empty body), and the helper proxy has not denied a single call – the request never leaves Envoy. The reason is in the routes from Day 7: no rule has backendRefs. Linkerd sends such a rule to the Service itself; Cilium follows the Gateway API literally and answers a rule without a backend with 500. With backendRefs, Envoy would forward, and then the helper proxy denies: 403, still with server: envoy (cross-checked). Two status codes, one cause – web has no proxy. The annotation has to go:

kubectl -n tag18 patch deploy web --type=json -p '[{"op":"remove",
  "path":"/spec/template/metadata/annotations/linkerd.io~1inject"}]'
kubectl -n tag18 rollout status deploy/web --timeout=120s
sleep 50
kubectl -n tag18 get pods -l app=web
kubectl -n tag18 logs deploy/last -c last --since=40s | awk '{print $2, $3}' \
  | sort | uniq -c

Expected (numbers vary):

NAME                   READY   STATUS    RESTARTS   AGE
web-7d94ccc84d-5rwlr   2/2     Running   0          57s
     58 /api/notes 200
     28 /flaky 200
      1 /flaky 500

Save your progress? Sign up for free

Round 2: Faster, but Broken

Situation: A colleague read that opaque connections have “less overhead” and used them to speed up helper. Since then, /flaky has been completely dead. Make /flaky as reliable as before.

Referee commands:

kubectl -n tag18 annotate service helper config.linkerd.io/opaque-ports=80

Target time: 5 min

Hint: The helper proxy rejects web, even though web is allowed. Ask the web proxy which protocol it assumes for the Service.

Solution to Round 2

sleep 60
kubectl -n tag18 logs deploy/last -c last --since=60s | awk '{print $2, $3}' \
  | sort | uniq -c
linkerd viz stat -n tag18 deploy
kubectl -n tag18 logs deploy/web -c web --since=30s | grep error | tail -1 \
  | grep -oE 'upstream prematurely[^,]*'

Expected (abridged, numbers vary):

     86 /api/notes 200
     43 /flaky 502
NAME        MESHED   SUCCESS      RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99 ...
helper         1/1   100.00%   0.4rps           1ms          10ms          10ms ...
last           1/1   100.00%   0.3rps           1ms          19ms          20ms ...
notes-api      1/1   100.00%   1.8rps           3ms         110ms         182ms ...
web            1/1    70.67%   2.5rps          18ms          96ms         175ms ...
upstream prematurely closed connection while reading response header from upstream

helper only sees the probes (0.4 rps); the HTTP requests from web no longer arrive as HTTP at all. nginx reports a connection that closes without a response. Who closes it?

kubectl -n tag18 logs deploy/helper -c linkerd-proxy --since=1m \
  | grep 'Connection denied' | tail -1 \
  | grep -oE 'policy::[a-z]+|server.name=[^ ]*|Name\("[^"]*'
linkerd diagnostics policy -n tag18 svc/helper 80 -o json \
  | jq -c '.protocol.Kind | keys'
kubectl -n tag18 get service helper -o jsonpath='{.metadata.annotations}{"\n"}' \
  | cut -c1-80
kubectl -n tag18 get server helper-http -o jsonpath='{.spec.proxyProtocol}{"\n"}'

Expected:

policy::tcp
server.name=helper-http
Name("web.tag18.serviceaccount.identity.linkerd.cluster.local
["Opaque"]
{"config.linkerd.io/opaque-ports":"80","kubectl.kubernetes.io/last-applied-confi
HTTP/1

The client proxy now sends the traffic as an opaque TCP stream. But the Server helper-http is declared as HTTP/1 – the TCP check (policy::tcp) closes the connection, even though the identity web is correct. And even if it got through: opaque also means no routes, no retries, no HTTP metrics. HTTP never belongs on an opaque port:

kubectl -n tag18 annotate service helper config.linkerd.io/opaque-ports-
sleep 5
linkerd diagnostics policy -n tag18 svc/helper 80 -o json \
  | jq -c '.protocol.Kind | keys'
sleep 55
kubectl -n tag18 logs deploy/last -c last --since=50s | awk '{print $2, $3}' \
  | sort | uniq -c

Expected (numbers vary):

service/helper annotated
["Detect"]
     62 /api/notes 200
     31 /flaky 200
      1 /flaky 500

Save your progress? Sign up for free

Round 3: The Hardening

Situation: The security department wants notes-api to receive read-only access only. To do so, a colleague built a route and moved the existing rule onto it. Since then, /api/notes no longer works. Fix it so that the hardening stays: POST must be rejected.

Referee commands:

kubectl apply -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: notes-api-lesen, namespace: tag18}
spec:
  parentRefs:
  - {group: policy.linkerd.io, kind: Server, name: notes-api-http}
  rules:
  - matches:
    - method: GET
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: notes-api-von-web, namespace: tag18}
spec:
  targetRef: {group: gateway.networking.k8s.io, kind: HTTPRoute, name: notes-api-lesen}
  requiredAuthenticationRefs:
  - {group: policy.linkerd.io, kind: MeshTLSAuthentication, name: notes-api}
EOF

Target time: 7 min

Hint: Two faults. Measure again right after the first fix – and remember what Day 6 says about routes on Servers.

Solution to Round 3

sleep 20
kubectl -n tag18 logs deploy/last -c last --since=15s | awk '{print $2, $3}' \
  | sort | uniq -c
sleep 50
kubectl -n tag18 logs deploy/last -c last --since=20s | awk '{print $2, $3}' \
  | sort | uniq -c
kubectl -n tag18 get pods -l app=notes-api

Expected (abridged):

     24 /api/notes 403
     12 /flaky 200
     30 /api/notes 503
     15 /flaky 200
      1 /flaky 500
NAME                         READY   STATUS    RESTARTS   AGE
notes-api-6ff48b7779-mmss9   1/2     Running   0          31m

First 403, half a minute later 503 and a Pod that is no longer Ready. Who is being rejected?

kubectl -n tag18 get events --field-selector reason=Unhealthy \
  --sort-by=.lastTimestamp | tail -1
kubectl -n tag18 logs deploy/notes-api -c linkerd-proxy --since=1m \
  | grep 'Request denied' \
  | grep -oE 'route.name=[^ ]*|client.ip=[^ ]*|Name\("[^"]*|client.tls=None' \
  | paste - - - | sort | uniq -c
linkerd viz authz -n tag18 deploy/notes-api | grep notes-api-lesen
kubectl -n tag18 get authorizationpolicy notes-api-von-web \
  -o jsonpath='{.spec.requiredAuthenticationRefs}{"\n"}'

Expected (abridged):

49s  Warning  Unhealthy  pod/notes-api-6ff48b7779-mmss9  Readiness probe failed: HTTP
probe failed with statuscode: 403
     17 route.name=notes-api-lesen  Name("web.tag18.serviceaccount.identity.lin...
      7 route.name=notes-api-lesen  client.tls=None  client.ip=10.0.2.241
notes-api-lesen  notes-api-http                                    0.3rps    0.00% ...
notes-api-lesen  notes-api-http  authorizationpolicy/notes-api-von-web     -       - ...
[{"group":"policy.linkerd.io","kind":"MeshTLSAuthentication","name":"notes-api"}]

Fault 1: The rule requires the identity notes-api – copied from postgres-von-notes-api. Fault 2: As soon as a route is attached to the Server, Linkerd no longer allows the probes automatically. The kubelet (10.0.2.241, without TLS) gets 403, the Pod drops out of the Service, web gets 503 (Day 6, Challenge). First the identity:

kubectl -n tag18 patch authorizationpolicy notes-api-von-web --type=json \
  -p '[{"op":"replace","path":"/spec/requiredAuthenticationRefs/0/name","value":"web"}]'
sleep 15
kubectl -n tag18 logs deploy/last -c last --since=10s | awk '{print $2, $3}' \
  | sort | uniq -c

Expected:

     14 /api/notes 503
      6 /flaky 200
      1 /flaky 500

Still 503 – after all, the Pod is not Ready. Then the probe, with its own route and the kubelet addresses of the nodes (Day 6):

cat > probe.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: notes-api-probe, namespace: tag18}
spec:
  parentRefs:
  - {group: policy.linkerd.io, kind: Server, name: notes-api-http}
  rules:
  - matches:
    - path: {type: Exact, value: /healthz}
      method: GET
---
apiVersion: policy.linkerd.io/v1alpha1
kind: NetworkAuthentication
metadata: {name: kubelet, namespace: tag18}
spec:
  networks:
  - cidr: 10.0.0.88/32
  - cidr: 10.0.1.116/32
  - cidr: 10.0.2.241/32
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: notes-api-probe-von-kubelet, namespace: tag18}
spec:
  targetRef: {group: gateway.networking.k8s.io, kind: HTTPRoute, name: notes-api-probe}
  requiredAuthenticationRefs:
  - {group: policy.linkerd.io, kind: NetworkAuthentication, name: kubelet}
EOF
kubectl apply -f probe.yaml
sleep 40
kubectl -n tag18 get pods -l app=notes-api
kubectl -n tag18 logs deploy/last -c last --since=20s | awk '{print $2, $3}' \
  | sort | uniq -c
kubectl -n tag18 exec deploy/web -c web -- curl -s -o /dev/null -w '%{http_code}\n' \
  -X POST -H 'X-API-Key: lab-key-123' -d '{"text":"x"}' http://notes-api/notes

Expected:

httproute.gateway.networking.k8s.io/notes-api-probe created
networkauthentication.policy.linkerd.io/kubelet created
authorizationpolicy.policy.linkerd.io/notes-api-probe-von-kubelet created
NAME                         READY   STATUS    RESTARTS   AGE
notes-api-6ff48b7779-mmss9   2/2     Running   0          32m
     30 /api/notes 200
     15 /flaky 200
404

GET works, POST matches no route and gets 404 – the hardening holds. The exact route /healthz beats the general GET route, so the probe is governed by its own rule.

Save your progress? Sign up for free

Round 4: Made Robust

Situation: To make notes-api “more robust”, the team added retries and a generous timeout of five seconds. Since then, every call fails. Keep the retries and the five-second timeout.

Referee commands:

kubectl apply -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: notes-api-robust
  namespace: tag18
  annotations:
    retry.linkerd.io/http: 5xx
    retry.linkerd.io/limit: "2"
    timeout.linkerd.io/request: 5ms
spec:
  parentRefs:
  - {group: core, kind: Service, name: notes-api, port: 80}
  rules:
  - matches:
    - path: {type: PathPrefix, value: /}
    timeouts: {request: 5s}
EOF

Target time: 4 min

Hint: For errors of its own, the proxy writes a header stating the reason.

Solution to Round 4

sleep 60
kubectl -n tag18 logs deploy/last -c last --since=50s | awk '{print $2, $3}' \
  | sort | uniq -c
kubectl -n tag18 exec deploy/web -c web -- curl -si -H 'X-API-Key: lab-key-123' \
  http://notes-api/notes | grep -E '^HTTP|^l5d'

Expected:

     64 /api/notes 504
     30 /flaky 200
      2 /flaky 500
HTTP/1.1 504 Gateway Timeout
l5d-proxy-error: logical service 10.109.85.58:80: route HTTPRoute.tag18.notes-api-robus
t: backend Service.tag18.notes-api:80: Service.tag18.notes-api:80: timed out waiting f
or response headers: request timeout: 5ms

l5d-proxy-error names the route and the reason: five milliseconds. Which setting wins is shown by the proxy’s view; the rate per route by stat-outbound (Day 7):

linkerd diagnostics policy -n tag18 svc/notes-api 80 -o json \
  | jq -c '.protocol.Kind.Detect.http1.routes[]
  | {name: .metadata.Kind.Resource.name, timeouts: .rules[0].timeouts}'
kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090 \
  >/dev/null 2>&1 &
sleep 4
PROM=--prometheusURL=http://localhost:9090
linkerd viz stat-outbound -n tag18 deploy/web $PROM | grep -A1 notes-api
kill %1

Expected (abridged):

{"name":"notes-api-robust","timeouts":{"request":{"nanos":5000000}}}
web   notes-api:80  notes-api-robust  HTTPRoute    ...  0.00%  ...  100.00%    0.00%
                    └──────────────►  notes-api:80 ...  0.00%  ...  100.00%

The annotation beats rules[].timeouts.request (Day 7). The TIMEOUTS column shows 100 percent, RETRIES 0: without retry.linkerd.io/timeout, an expired timeout is not retried. Remove the annotation; the rule with 5 s stays:

kubectl -n tag18 annotate httproute notes-api-robust timeout.linkerd.io/request-
sleep 5
linkerd diagnostics policy -n tag18 svc/notes-api 80 -o json \
  | jq -c '.protocol.Kind.Detect.http1.routes[]
  | {name: .metadata.Kind.Resource.name, timeouts: .rules[0].timeouts}'
sleep 50
kubectl -n tag18 logs deploy/last -c last --since=45s | awk '{print $2, $3}' \
  | sort | uniq -c

Expected (numbers vary):

httproute.gateway.networking.k8s.io/notes-api-robust annotated
{"name":"notes-api-robust","timeouts":{"request":{"seconds":5}}}
     69 /api/notes 200
     33 /flaky 200
      1 /flaky 500

Save your progress? Sign up for free

Round 5: The Brake

Situation: To guard against noisy clients, notes-api gets a rate limit: at most one request per second per identity, ten for web as the most important customer. The template comes from notes. Since then, every second request of the load generator is rejected. The limit is supposed to stay.

Referee commands:

kubectl apply -f - <<'EOF'
apiVersion: policy.linkerd.io/v1alpha1
kind: HTTPLocalRateLimitPolicy
metadata: {name: notes-api-rl, namespace: tag18}
spec:
  targetRef: {group: policy.linkerd.io, kind: Server, name: notes-api-http}
  total:
    requestsPerSecond: 50
  identity:
    requestsPerSecond: 1
  overrides:
  - requestsPerSecond: 10
    clientRefs:
    - {kind: ServiceAccount, name: web, namespace: notes}
EOF

Target time: 4 min

Hint: Which bucket is empty – total or identity?

Solution to Round 5

sleep 60
kubectl -n tag18 logs deploy/last -c last --since=50s | awk '{print $2, $3}' \
  | sort | uniq -c
kubectl -n tag18 logs deploy/notes-api -c linkerd-proxy --since=1m \
  | grep -oE '[a-z]+ rate limit exceeded: [0-9]+rps' | sort | uniq -c
linkerd diagnostics proxy-metrics -n tag18 deploy/notes-api \
  | grep '^inbound_http_local_ratelimit' \
  | grep -oE 'ratelimit_scope="[^"]*"|client_id="[a-z.0-9-]+|\} [0-9]+' | paste - - -
kubectl -n tag18 get httplocalratelimitpolicy notes-api-rl \
  -o jsonpath='{.spec.overrides}{"\n"}'

Expected (abridged, numbers vary):

     39 /api/notes 200
     39 /api/notes 429
     37 /flaky 200
      1 /flaky 500
     96 identity rate limit exceeded: 1rps
ratelimit_scope="identity"  client_id="web.tag18.serviceaccount.identity.linker... } 58
[{"clientRefs":[{"kind":"ServiceAccount","name":"web","namespace":"notes"}],"reque...

Rejected in the scope identity, specifically web.tag18 – but the override applies to the ServiceAccount web in notes. For web from tag18, the limit for everyone therefore applies: one request per second, while last sends around 1.5. Without namespace, the namespace of the policy applies; here it is set explicitly:

kubectl -n tag18 patch httplocalratelimitpolicy notes-api-rl --type=json \
  -p '[{"op":"replace","path":"/spec/overrides/0/clientRefs/0/namespace",
  "value":"tag18"}]'
sleep 60
kubectl -n tag18 logs deploy/last -c last --since=50s | awk '{print $2, $3}' \
  | sort | uniq -c
kubectl -n tag18 logs deploy/notes-api -c linkerd-proxy --since=40s \
  | grep -c 'rate limit exceeded'

Expected (numbers vary):

httplocalratelimitpolicy.policy.linkerd.io/notes-api-rl patched
     75 /api/notes 200
     36 /flaky 200
      1 /flaky 500
0

Save your progress? Sign up for free

Round 6: The Release

Situation: Version 2 of notes-api goes live as a canary, with half of the traffic. The team reports: “Works, just a few percent errors, surely that’s the retries.” Bring /api/notes to 100 percent without deleting version 2 – the team wants to debug it.

Referee commands:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata: {name: v2-conf, namespace: tag18}
data:
  default.conf: |
    server {
      listen 8080;
      location = /healthz { return 200 "ok\n"; }
      location / { return 500 "v2 kaputt\n"; }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: notes-api-v2, namespace: tag18}
spec:
  replicas: 1
  selector: {matchLabels: {app: notes-api-v2}}
  template:
    metadata: {labels: {app: notes-api-v2}}
    spec:
      serviceAccountName: notes-api
      containers:
      - name: notes-api
        image: docker.io/nginxinc/nginx-unprivileged:1.31.7-alpine
        ports: [{containerPort: 8080, name: http}]
        readinessProbe:
          httpGet: {path: /healthz, port: http}
        volumeMounts: [{name: conf, mountPath: /etc/nginx/conf.d}]
      volumes: [{name: conf, configMap: {name: v2-conf}}]
---
apiVersion: v1
kind: Service
metadata: {name: notes-api-v2, namespace: tag18}
spec:
  selector: {app: notes-api-v2}
  ports: [{port: 80, targetPort: http, name: http}]
---
apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata: {name: notes-api-v2-http, namespace: tag18}
spec:
  podSelector: {matchLabels: {app: notes-api-v2}}
  port: http
  proxyProtocol: HTTP/1
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: notes-api-v2-von-web, namespace: tag18}
spec:
  targetRef: {group: policy.linkerd.io, kind: Server, name: notes-api-v2-http}
  requiredAuthenticationRefs:
  - {group: policy.linkerd.io, kind: MeshTLSAuthentication, name: web}
EOF
kubectl -n tag18 rollout status deploy/notes-api-v2 --timeout=120s
kubectl -n tag18 patch httproute notes-api-robust --type=json -p '[{"op":"add",
  "path":"/spec/rules/0/backendRefs","value":[{"name":"notes-api","port":80,
  "weight":50},{"name":"notes-api-v2","port":80,"weight":50}]}]'

Target time: 5 min

Hint: stat-outbound shows every backend of a route separately.

Solution to Round 6

sleep 90
kubectl -n tag18 logs deploy/last -c last --since=80s | awk '{print $2, $3}' \
  | sort | uniq -c
linkerd viz stat -n tag18 deploy | grep -E 'NAME|notes-api'

Expected (abridged, numbers vary):

    101 /api/notes 200
     11 /api/notes 500
     55 /flaky 200
      1 /flaky 500
NAME           MESHED   SUCCESS      RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99 ...
notes-api         1/1   100.00%   1.7rps           5ms          32ms          38ms ...
notes-api-v2      1/1    28.57%   1.4rps           1ms           6ms           9ms ...

Around ten percent errors – and notes-api-v2 with 28 percent success, and that only thanks to the probes. Per backend of the route:

kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090 \
  >/dev/null 2>&1 &
sleep 4
PROM=--prometheusURL=http://localhost:9090
linkerd viz stat-outbound -n tag18 deploy/web $PROM | grep -A2 notes-api
kill %1
kubectl -n tag18 get httproute notes-api-robust \
  -o jsonpath='{range .spec.rules[0].backendRefs[*]}{.name} {.weight}{"\n"}{end}'

Expected (abridged):

web   notes-api:80  notes-api-robust  HTTPRoute       90.70%  1.43  ...  0.00%   37.23%
                    ├──────────────►  notes-api:80   100.00%  1.30  ...  0.00%
                    └──────────────►  notes-api-v2:80  0.00%  0.98  ...  0.00%
notes-api 50
notes-api-v2 50

Version 2 fails 100 percent of the time. That the route still reaches 90 percent is due to the retries from Round 4: a failure on v2 is retried and lands on v1 with a probability of 50 percent. Only when all three attempts hit v2 (0.5³ = 12.5 percent) does the user see the error. Retries hide a broken canary. Roll back via the weight (Day 9):

kubectl -n tag18 patch httproute notes-api-robust --type=json -p '[{"op":"replace",
  "path":"/spec/rules/0/backendRefs/1/weight","value":0}]'
sleep 75
kubectl -n tag18 logs deploy/last -c last --since=65s | awk '{print $2, $3}' \
  | sort | uniq -c
kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090 \
  >/dev/null 2>&1 &
sleep 4
linkerd viz stat-outbound -n tag18 deploy/web $PROM -t 30s | grep -A2 notes-api
kill %1

Expected (abridged, numbers vary):

    102 /api/notes 200
     50 /flaky 200
      1 /flaky 500
web   notes-api:80  notes-api-robust  HTTPRoute      100.00%  1.50  ...  0.00%    0.00%
                    ├──────────────►  notes-api:80   100.00%  1.50  ...  0.00%
                    └──────────────►  notes-api-v2:80   NaN%  0.00  ...   NaN%

v2 keeps running and no longer gets anything. Had you deleted the v2 Deployment instead, you would also have seen 100 percent (a backend without endpoints drops out of the distribution) – but that is exactly what the team did not want.

Save your progress? Sign up for free

Round 7: The Firewall in the Mesh

Situation: The security department blocks all outgoing traffic from tag18. Only example.com via HTTPS is supposed to be allowed; notes-api needs it for /api/upstream. But that now returns 502. Allow exactly example.com – nothing else.

Referee commands:

kubectl apply -f - <<'EOF'
apiVersion: policy.linkerd.io/v1alpha1
kind: EgressNetwork
metadata: {name: internet, namespace: tag18}
spec:
  trafficPolicy: Deny
---
apiVersion: gateway.networking.k8s.io/v1
kind: TLSRoute
metadata: {name: example-com, namespace: tag18}
spec:
  parentRefs:
  - {group: policy.linkerd.io, kind: EgressNetwork, name: internet, port: 443}
  hostnames: [www.example.com]
  rules:
  - backendRefs:
    - {group: policy.linkerd.io, kind: EgressNetwork, name: internet}
EOF

Target time: 5 min

Hint: Compare the name that notes-api calls with the name that the route allows.

Solution to Round 7

kubectl -n tag18 exec deploy/last -c last -- curl -s -w ' %{http_code}\n' \
  http://web/api/upstream
linkerd diagnostics proxy-metrics -n tag18 deploy/notes-api \
  | grep -E '^outbound_t(cp|ls)_route_open_total' \
  | grep -oE 'parent_name="[^"]*"|route_name="[^"]*"|\} [0-9]+' | paste - - -

Expected (abridged):

{"error":"Get \"https://example.com\": EOF"} 502
parent_name="postgres"         route_name="opaq"             } 70
parent_name="egress-fallback"  route_name="egress-fallback"  } 1
parent_name="internet"         route_name="tls-egress-deny"  } 3

The first connection (baseline) still went through egress-fallback; now everything lands on the default route tls-egress-deny. Your own TLSRoute does not count at all. Why?

kubectl -n tag18 get tlsroute example-com \
  -o jsonpath='{.spec.hostnames} {.status.parents[0].conditions[0].reason}{"\n"}'
kubectl -n tag18 set env deploy/notes-api -c notes-api --list | grep UPSTREAM

Expected:

["www.example.com"] Accepted
NOTES_UPSTREAM=https://example.com

The route is Accepted but does not match: the SNI name must match exactly (Day 11). www.example.com is a different host than example.com.

kubectl -n tag18 patch tlsroute example-com --type=merge \
  -p '{"spec":{"hostnames":["example.com"]}}'
sleep 5
kubectl -n tag18 exec deploy/last -c last -- curl -s -w ' %{http_code}\n' \
  http://web/api/upstream
for u in https://example.com https://linkerd.io; do
  kubectl -n tag18 exec deploy/last -c last -- curl -s -o /dev/null \
    -w "$u %{http_code}\n" $u
done
linkerd diagnostics proxy-metrics -n tag18 deploy/notes-api \
  | grep -E '^outbound_tls_route_open_total' \
  | grep -oE 'route_name="[^"]*"|\} [0-9]+' | paste - -

Expected:

tlsroute.gateway.networking.k8s.io/example-com patched
{"status":200,"url":"https://example.com"}
 200
https://example.com 200
https://linkerd.io 000
command terminated with exit code 35
route_name="example-com"      } 1
route_name="tls-egress-deny"  } 3

example.com works, linkerd.io stays blocked – for last too, because the EgressNetwork applies to all clients in the namespace.

Save your progress? Sign up for free

Round 8: The Auditors Ask

Situation: You don’t break certificates in a competition – that would hit the entire mesh (Day 16). Instead, the auditors want three answers, each backed by a command. Read only, change nothing.

  1. What identity does web have in tag18, who issued it, and until when is the certificate valid?
  2. When does the issuer expire, and when is it renewed? Do Pods have to restart for that?
  3. Is exactly the trust anchor from ~/linkerd/tag05/ca.crt distributed in the cluster? What does linkerd check say about it?

Target time: 4 min

Hint: Day 3 and Day 5, speed drills.

Solution to Round 8

linkerd identity -n tag18 -l app=web | grep -E 'Subject:|Issuer:|Not After'
kubectl -n linkerd get certificate linkerd-identity-issuer \
  -o jsonpath='{.status.notAfter} {.status.renewalTime}{"\n"}'
kubectl -n linkerd get configmap linkerd-identity-trust-roots \
  -o jsonpath='{.data.ca-bundle\.crt}' | step certificate fingerprint -
step certificate fingerprint ~/linkerd/tag05/ca.crt
linkerd check 2>&1 | grep -A2 '‼ issuer'

Expected (dates depend on your run):

        Issuer: CN=identity.linkerd.cluster.local
            Not After : Nov 20 20:40:11 2026 UTC
        Subject: CN=web.tag18.serviceaccount.identity.linkerd.cluster.local
2026-11-21T16:02:00Z 2026-11-20T15:02:00Z
8b44a80f0940a58823716aecd2aeac2895e43d94bd968826ec821389f65bab81
8b44a80f0940a58823716aecd2aeac2895e43d94bd968826ec821389f65bab81
‼ issuer cert is valid for at least 60 days
    issuer certificate will expire on 2026-11-21T16:02:00Z
    see https://linkerd.io/2/checks/#l5d-identity-issuer-cert-not-expiring-soon ...
  1. web.tag18.serviceaccount.identity.linkerd.cluster.local, issued by the issuer identity.linkerd.cluster.local, valid for 24 hours from Pod start – the proxy renews it itself.
  2. The issuer is valid for 48 hours; cert-manager renews it 25 hours before expiry. identity reloads it without a restart (IssuerUpdated), and the proxies get their next certificates from the new issuer – no Pod has to restart.
  3. Yes, both fingerprints are identical. The warning from check is expected with a 48-hour issuer (Day 5) and is not a finding.

Save your progress? Sign up for free

Scoring

Round Fault Target time Points
1 linkerd.io/inject: disabled on the Pod template of web 3 min 2
2 opaque port on the HTTP Service helper 5 min 2
3 wrong identity in the rule + probes without a route 7 min 3
4 timeout annotation 5ms beats timeouts.request 4 min 2
5 rate limit override for an SA from the wrong namespace 4 min 2
6 canary at 50 percent on a broken version, hidden by retries 5 min 2
7 TLSRoute for www.example.com instead of example.com 5 min 2
8 three questions on identity, issuer, trust anchor 4 min 3

You only get full points within the target time, without a hint and with a proof command; otherwise half (round down). In Round 8 there is one point per backed-up answer. Whoever removes the symptom instead of the cause (default-deny off, accessPolicy: all-unauthenticated, deleting the hardening or the rate limit, deleting the route instead of keeping the retries, deleting v2, trafficPolicy: Allow) gets 0 for the round. Overall target time: 37 min, maximum: 18 points.

Points Assessment
15–18 ready for the final exam (Appendix B)
10–14 good; repeat the days of the rounds you missed
0–9 work through Days 2, 6, 7, 8, 9 and 11 once more

Challenge

After a change to notes-api, the rollout is stuck. The users notice nothing, the load generator shows 100 percent. Finish the rollout without loosening the rules for postgres. Referee commands:

kubectl -n tag18 patch deploy notes-api -p '{"spec":{"template":{"metadata":
  {"annotations":{"config.beta.linkerd.io/proxy-enable-native-sidecar":"false"}}}}}'

Hint: Look at the order of the init containers in the new Pod.

Solution

sleep 60
kubectl -n tag18 get pods -l app=notes-api
P=$(kubectl -n tag18 get pod -l app=notes-api \
  --sort-by=.metadata.creationTimestamp -o name | tail -1)
kubectl -n tag18 get $P \
  -o jsonpath='{range .spec.initContainers[*]}{.name} {.restartPolicy}{"\n"}{end}'
kubectl -n tag18 logs $P -c migrate --tail=1 | cut -c1-80
linkerd diagnostics proxy-metrics -n tag18 po/postgres-0 \
  | grep '^inbound_tcp_authz_deny_total' \
  | grep -oE '(srv_name|no_tls_reason)="[^"]*"|\} [0-9]+' | paste - - -
kubectl -n tag18 get deploy notes-api \
  -o jsonpath='{.spec.template.metadata.annotations}{"\n"}'

Expected (abridged):

NAME                         READY   STATUS       RESTARTS      AGE
notes-api-54c8fb75bf-xdxbt   0/2     Init:Error   3 (43s ago)   60s
notes-api-6ff48b7779-mmss9   2/2     Running      0             33m
migrate
linkerd-init
time=2026-11-19T21:10:10.890Z level=ERROR msg="database unreachable" source=env
srv_name="postgres"  no_tls_reason="port_skipped"  } 7
{"config.beta.linkerd.io/proxy-enable-native-sidecar":"false"}

migrate runs before linkerd-init, and there is no longer a proxy as a native sidecar (restartPolicy: Always). The migration talks plaintext to Postgres, and default-deny closes the connection (port_skipped, Days 2 and 6). The annotation has to go:

A='config.beta.linkerd.io~1proxy-enable-native-sidecar'
kubectl -n tag18 patch deploy notes-api --type=json -p '[{"op":"remove",
  "path":"/spec/template/metadata/annotations/'$A'"}]'
kubectl -n tag18 rollout status deploy/notes-api --timeout=180s
kubectl -n tag18 get pods -l app=notes-api

Expected:

deployment "notes-api" successfully rolled out
NAME                         READY   STATUS    RESTARTS   AGE
notes-api-6ff48b7779-mmss9   2/2     Running   0          34m

The template is identical to the old one again, so Kubernetes simply scales the old ReplicaSet back up – the same Pod keeps running.

Save your progress? Sign up for free

Cleanup

kubectl delete namespace tag18 --timeout=180s
kubectl get endpointslices -A | grep cilium-gateway | grep tag18

Expected: namespace "tag18" deleted, then no output. Together with the namespace, the PVC, the EgressNetwork and the EndpointSlices cilium-gateway-helper and cilium-gateway-notes-api, which Cilium created for the HTTPRoutes on the Services, disappear. The files in ~/linkerd/tag18 remain as a template for the final exam.

Checkpoint

kubectl get ns | grep -c tag18
kubectl get pv | grep -c tag18

Expected: 0 twice.

kubectl -n notes get pods

Expected: all eight Pods 2/2 Running – you did not touch notes today.

kubectl -n notes logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
  | sort | uniq -c
linkerd check 2>&1 | tail -1

Expected (numbers vary):

     86 /api/notes 200
     29 /flaky 200
     14 /flaky 500
Status check results are √

Interview Note

  • A service suddenly returns 403 – where do you start? At the server proxy: the proxy log Request denied shows the Server, route and client identity. client.tls=None means: client without a proxy. A wrong identity means: check the ServiceAccount or the rule.
  • Why does a broken canary look harmless in the dashboard? The client proxy’s retries repeat failures on the other backend. The route is almost green while the backend sits at 0 percent – read stat-outbound per backend, not just the route.
  • What do you check after every policy or route change? Readiness of the target Pods (probes on Servers with routes), the proxy’s view with linkerd diagnostics policy, and a measurement with fresh connections.

Enterprise Note

  • Play game days like this one regularly with the on-call team – the solutions become runbooks.
  • Allow or forbid annotations that affect the mesh (inject, opaque-ports, proxy-enable-native-sidecar, timeouts) via admission policy, not via review.
  • Manage policies, routes and rate limits from Git with review and a test namespace that has default-deny like production. Check copied templates for namespace and identities.
  • Evaluate canary releases automatically (Flagger, Argo Rollouts) – with the error rate per backend and the retry rate as abort criteria.
  • Maintain egress allowances as a list of hostnames and cross-check them with metrics (tls-egress-deny) to see which blocked targets are being requested.