Skip to content
Hands-on IT IT Drills

Appendix B: Final Exam

Goal: You solve 16 practical tasks from across the book in 120 minutes and grade yourself. · Duration: 120 min · Prerequisite: Days 0–18

Rules of Play

  • Time: 120 minutes, set a timer. Each task has a weight in percent, 100 % in total.
  • There is no certificate. Your own yardstick: from 70 %, the craft is solid.
  • Allowed: linkerd.io/2-edge, gateway-api.sigs.k8s.io, --help, kubectl explain and your own files under ~/linkerd/tagNN as templates. Not allowed: this book during the exam.
  • Everything runs on cp1 in the namespace pruefung, files in ~/linkerd/pruefung. Tasks 13 and 15 also address west (--context west).
  • You do not touch notes. No cluster-wide changes: no linkerd upgrade, no helm upgrade, no reinstalling or removing an extension. The only exception is the link in task 13 – it disappears again during cleanup.
  • Order: tasks 1 to 6 in sequence, task 4 before task 7, task 7 before task 10. From task 8 on, the default-deny from task 6 applies. Otherwise the order is up to you.
  • The solutions are at the end. Only read them once the time is up.

Preparation

A copy of the app from the Day 0 manifests, one replica per service, still without the mesh. In addition, notes-api with a target on the internet (Day 11) and a service statistik that a colleague built. Every ten seconds, it counts the notes directly in Postgres:

mkdir -p ~/linkerd/pruefung && cd ~/linkerd/pruefung
kubectl create namespace pruefung
kubectl -n pruefung create secret generic notes-db \
  --from-literal=password=lab-db-123 \
  --from-literal=url='postgres://notes:lab-db-123@postgres:5432/notes?sslmode=disable'
kubectl -n pruefung create secret generic notes-api-key --from-literal=key=lab-key-123
for f in postgres notes-api helper web last; do
  sed 's/namespace: notes/namespace: pruefung/; s/\.notes\.svc/.pruefung.svc/
    s/replicas: 2/replicas: 1/' ~/linkerd/tag00/$f.yaml; echo ---
done > app.yaml
kubectl -n pruefung create serviceaccount statistik
cat > statistik.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: statistik, namespace: pruefung}
spec:
  replicas: 1
  selector: {matchLabels: {app: statistik}}
  template:
    metadata:
      labels: {app: statistik}
      annotations: {config.linkerd.io/skip-outbound-ports: "5432"}
    spec:
      serviceAccountName: statistik
      containers:
      - name: statistik
        image: docker.io/library/postgres:18.7-alpine
        env:
        - {name: PGCONNECT_TIMEOUT, value: "3"}
        - name: PGPASSWORD
          valueFrom: {secretKeyRef: {name: notes-db, key: password}}
        command: [sh, -c]
        args:
        - |
          while true; do
            n=$(psql -h postgres -U notes -tAc 'select count(*) from notes' 2>&1)
            echo "$(date +%T) $n"
            sleep 10
          done
EOF
kubectl apply -f app.yaml -f statistik.yaml
kubectl -n pruefung set env deploy/notes-api NOTES_UPSTREAM=https://example.com
kubectl -n pruefung rollout status statefulset/postgres --timeout=180s
kubectl -n pruefung rollout status deploy --timeout=240s

For task 8, a broken notes-api is ready – only as a file, not yet applied. It answers the readiness probe, and everything else with 500:

cat > kaputt.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata: {name: kaputt-conf, namespace: pruefung}
data:
  default.conf: |
    server {
      listen 8080;
      location = /healthz { return 200 "ok\n"; }
      location / { return 500 "kaputt\n"; }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: notes-api-kaputt, namespace: pruefung}
spec:
  replicas: 1
  selector: {matchLabels: {app: notes-api, variante: kaputt}}
  template:
    metadata: {labels: {app: notes-api, variante: kaputt}}
    spec:
      serviceAccountName: notes-api
      containers:
      - name: notes-api
        image: docker.io/nginxinc/nginx-unprivileged:1.31.7-alpine
        ports: [{containerPort: 8080, name: http}]
        readinessProbe:
          httpGet: {path: /healthz, port: http}
        volumeMounts: [{name: conf, mountPath: /etc/nginx/conf.d}]
      volumes: [{name: conf, configMap: {name: kaputt-conf}}]
EOF

After one minute, the baseline:

kubectl -n pruefung logs deploy/last --since=35s | awk '{print $2, $3}' | sort | uniq -c
kubectl -n pruefung logs deploy/statistik --tail=2

Expected (numbers vary):

     58 /api/notes 200
     20 /flaky 200
      9 /flaky 500
21:23:34 0
21:23:44 0

The 0 is the number of notes in the new database. Start the clock.

Task 1: Inventory (5 %)

Write four lines to ~/linkerd/pruefung/01.txt: the version of the control plane, the number of Linkerd CRDs in the cluster, the last line of linkerd check and the names of the installed extensions (one line, separated by spaces).

Task 2: Into the Mesh (7 %)

Bring pruefung into the mesh. Each of the five app services gets its own ServiceAccount with its name; statistik already has one. Afterwards, all Pods run with a proxy (2/2), and linkerd check --proxy -n pruefung reports no error and no warning about opaque ports.

Task 3: Who Talks to Postgres? (6 %)

Without tcpdump: write to 03.txt the identity with which notes-api arrives at postgres-0 (line 1), and the number of plaintext connections that postgres-0 has accepted on port 5432 so far (line 2). Also show with one command that all HTTP edges in pruefung run over mTLS.

Task 4: Golden Metrics (6 %)

Solve before task 7. Write to 04.txt the success rate of helper as seen from web, as a decimal number – once according to linkerd viz (line 1), once via PromQL over five minutes directly from Prometheus (line 2).

Task 5: The Auditors Ask About Certificates (5 %)

Read only. Write to 05.txt: when the trust anchor in the cluster expires, when the issuer expires and when it is renewed, and two fingerprints proving that exactly the anchor from ~/linkerd/tag05/ca.crt is distributed in the cluster.

Task 6: Default-Deny (8 %)

In pruefung, only those who must may talk: last → web → notes-api → postgres and web → helper. New ports without a rule are blocked. Probes and the Prometheus scrapes (up = 1 for every proxy in pruefung) keep working, and the load generator shows the same rates as before. Use a Pod fremd (with a proxy, ServiceAccount default) to prove that it gets 403 from helper.

Task 7: Retries and Timeouts (7 %)

/flaky succeeds at least 95 percent of the time: the client proxy of web retries GET requests to /unstable on 5xx at most twice, other methods never. All other paths of helper stay reachable. /slow aborts with 504 after one second.

Task 8: A Pod That Lies (6 %)

Start the broken Pod with kubectl apply -f kaputt.yaml. It is ready, but only returns 500. Without deleting it, scaling it or changing its labels: every client should avoid it after three consecutive failures for at least 30 seconds, at most two minutes. Afterwards, /api/notes is back above 99 percent. Prove per Pod who still gets traffic from web.

Task 9: A Brake for the Report (7 %)

Create a Deployment bericht (curlimages/curl:8.23.0, sleep infinity, its own ServiceAccount bericht). bericht may call helper directly, but at most 2 requests per second and Pod. web may send up to 20, and each helper Pod accepts at most 50 in total. Proof: 40 fast calls from bericht end mostly with 429, and the load generator does not see a single one.

Task 10: Canary (6 %)

A new version helper-v2 (go-httpbin 2.26.0, label app: helper-v2, ServiceAccount helper, Service helper-v2 on port 80) gets 20 percent of the calls from web to helper – except /delay, which stays with the old version. The retries from task 7 stay in effect, default-deny stays. Prove the distribution.

Task 11: The Entrance (7 %)

A Cilium Gateway pruefung accepts HTTP on port 80 for the host pruefung.lab.internal and forwards to web. From cp1, /api/notes returns 200 via the Gateway’s NodePort on every node. A Pod without a proxy that calls web directly still gets 403 – opening up the Pod network does not count.

Task 12: Only example.com (6 %)

Outside the cluster, Pods in pruefung can only reach https://example.com. /api/upstream via web still returns 200, https://linkerd.io and http://example.com fail from last. This must also hold for fresh connections from notes-api.

Task 13: A helper from west (8 %)

On west, a helper runs in the mesh in the namespace pruefung (go-httpbin 2.26.0 with USE_REAL_HOSTNAME=true, Service port 80). In ost, it appears as helper-west, and last gets a Pod name from west under http://helper-west/hostname. Prove at the gateway in west with which identity last arrives. At the start, no link is active (Day 13, Cleanup).

Task 14: The Statistics Stop Counting (7 %)

Since task 6, statistik only reports errors. Repair the service so that it may count with its own identity. Default-deny stays, and postgres still accepts neither plaintext nor foreign identities.

Task 15: Keeping an Eye on west (4 %)

Read only. Write to 15.txt the chart and revision of the release linkerd-control-plane on west (line 1), the effective values controllerReplicas and webhookFailurePolicy (lines 2 and 3) and the number of ready replicas of linkerd-destination (line 4).

Task 16: A Proxy on a Budget (5 %)

The proxy of web gets 50m CPU and 32 MiB memory as a request and 128 MiB as a limit – by annotation only, the app itself stays without resources. Show the resources of the proxy container and the QoS class of the Pod before and after.

Stop the clock.

Solutions

All commands on cp1 in the folder ~/linkerd/pruefung.

Solution 1

linkerd version --short | tail -1 > 01.txt
kubectl get crd -o name | grep -c linkerd >> 01.txt
linkerd check 2>&1 | tail -1 >> 01.txt
kubectl get ns -l linkerd.io/extension \
  -o jsonpath='{.items[*].metadata.labels.linkerd\.io/extension}{"\n"}' >> 01.txt
cat 01.txt

Expected:

edge-26.11.2
11
Status check results are √
multicluster viz

--short shows the CLI first, then the control plane – tail -1 takes the second. Eleven CRDs: the ten from Day 1 plus links.multicluster.linkerd.io. linkerd check recognizes extensions by the label linkerd.io/extension on their namespace (Day 15).

Solution 2

kubectl annotate namespace pruefung linkerd.io/inject=enabled
for a in web notes-api helper last postgres; do
  kubectl -n pruefung create serviceaccount $a
done
for d in web notes-api helper last; do
  kubectl -n pruefung set serviceaccount deploy/$d $d
done
kubectl -n pruefung set serviceaccount statefulset/postgres postgres
kubectl -n pruefung annotate service postgres config.linkerd.io/opaque-ports=5432
kubectl -n pruefung rollout restart deploy/statistik
kubectl -n pruefung rollout status statefulset/postgres --timeout=180s
kubectl -n pruefung rollout status deploy --timeout=240s
kubectl -n pruefung get pods
linkerd check --proxy -n pruefung 2>&1 | grep -E '‼|×|opaque|Status'

Expected (abridged):

NAME                         READY   STATUS    RESTARTS   AGE
helper-79d57fbbbb-bg874      2/2     Running   0          24s
last-78895758f9-wvbz6        2/2     Running   0          24s
notes-api-6ff48b7779-86v22   2/2     Running   0          24s
postgres-0                   2/2     Running   0          21s
statistik-64cf4b6f5b-fvwxw   2/2     Running   0          22s
web-7d94ccc84d-tp7wk         2/2     Running   0          25s
‼ issuer cert is valid for at least 60 days
√ opaque ports are properly annotated
Status check results are √

set serviceaccount changes the Pod template and restarts the four Deployments and Postgres anyway – with a proxy, because the namespace is already annotated. Only statistik needs a restart of its own (Day 2). You know the issuer warning from Day 5; it is not a finding.

Solution 3

linkerd diagnostics proxy-metrics -n pruefung po/postgres-0 \
  | grep '^tcp_open_total{direction="inbound",peer="src"' \
  | grep 'target_port="5432"' > pg-tcp.txt
grep -o 'client_id="[^"]*' pg-tcp.txt | cut -d'"' -f2 > 03.txt
grep port_skipped pg-tcp.txt | grep -oE '[0-9]+$' >> 03.txt
cat 03.txt
linkerd viz edges -n pruefung deploy

Expected (number varies):

notes-api.pruefung.serviceaccount.identity.linkerd.cluster.local
11
SRC    DST         SRC_NS     DST_NS     SECURED
last   web         pruefung   pruefung   √
web    helper      pruefung   pruefung   √
web    notes-api   pruefung   pruefung   √

The plaintext connections carry no_tls_reason="port_skipped" – someone is talking to Postgres past the proxy (Day 3, Drill 7). Who that is, task 14 clarifies. edges shows every HTTP edge with √; Postgres is missing because it is a StatefulSet.

Solution 4

linkerd viz stat -n pruefung deploy/helper --from deploy/web
linkerd viz stat -n pruefung deploy/helper --from deploy/web -o json \
  | jq -r '.[0].success' > 04.txt
kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090 \
  >/dev/null 2>&1 &
sleep 4
Q='sum(rate(response_total{namespace="pruefung", deployment="web",
 direction="outbound", dst_deployment="helper", classification="success"}[5m]))
 / sum(rate(response_total{namespace="pruefung", deployment="web",
 direction="outbound", dst_deployment="helper"}[5m]))'
curl -s localhost:9090/api/v1/query --data-urlencode "query=$Q" \
  | jq -r '.data.result[0].value[1]' >> 04.txt
kill %1
cat 04.txt

Expected (numbers vary):

NAME     MESHED   SUCCESS      RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99   TCP_CONN
helper      1/1    71.43%   0.7rps           8ms          36ms          39ms          1
0.7142857142857143
0.6820478364789672

Both around 0.7 – the baseline of /flaky. --from deploy/web leaves out the readiness probes that embellish the inbound rate of helper (Day 4, Drill 4). viz calculates over one minute, the PromQL query over five.

Solution 5

kubectl -n linkerd get configmap linkerd-identity-trust-roots \
  -o jsonpath='{.data.ca-bundle\.crt}' | openssl x509 -noout -enddate > 05.txt
kubectl -n linkerd get certificate linkerd-identity-issuer \
  -o jsonpath='{.status.notAfter} {.status.renewalTime}{"\n"}' >> 05.txt
kubectl -n linkerd get configmap linkerd-identity-trust-roots \
  -o jsonpath='{.data.ca-bundle\.crt}' | step certificate fingerprint - >> 05.txt
step certificate fingerprint ~/linkerd/tag05/ca.crt >> 05.txt
cat 05.txt

Expected (dates depend on your run):

notAfter=Nov 16 15:51:50 2036 GMT
2026-11-21T16:02:00Z 2026-11-20T15:02:00Z
8b44a80f0940a58823716aecd2aeac2895e43d94bd968826ec821389f65bab81
8b44a80f0940a58823716aecd2aeac2895e43d94bd968826ec821389f65bab81

Ten years for the anchor, 48 hours for the issuer, renewal 25 hours before expiry (Day 5). Identical fingerprints: the cluster trusts exactly your ca.crt.

Solution 6

The rules from Day 6 fit one to one. proxy-admin comes along before the default policy is switched – otherwise minutes of metrics are missing:

for f in servers authz admin; do
  sed 's/namespace: notes/namespace: pruefung/' ~/linkerd/tag06/$f.yaml; echo ---
done > policy.yaml
grep -c '^kind:' policy.yaml
kubectl apply -f policy.yaml
kubectl annotate namespace pruefung config.linkerd.io/default-inbound-policy=deny
kubectl -n pruefung rollout restart deploy
kubectl -n pruefung rollout restart statefulset/postgres
kubectl -n pruefung rollout status statefulset/postgres --timeout=180s
kubectl -n pruefung rollout status deploy --timeout=240s
kubectl -n pruefung run fremd --image=docker.io/curlimages/curl:8.23.0 \
  --command -- sleep 3600
kubectl -n pruefung wait --for=condition=Ready pod/fremd --timeout=90s
kubectl -n pruefung exec fremd -c fremd -- curl -s -o /dev/null -w '%{http_code}\n' \
  http://helper/get

Expected (abridged):

14
server.policy.linkerd.io/web-http created
...
authorizationpolicy.policy.linkerd.io/admin-von-prometheus created
namespace/pruefung annotated
...
pod/fremd condition met
403

One minute later, the cross-check:

kubectl -n pruefung logs deploy/last -c last --since=40s | awk '{print $2, $3}' \
  | sort | uniq -c
kubectl -n pruefung get server -o custom-columns=NAME:.metadata.name,\
POLICY:.spec.accessPolicy
kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090 \
  >/dev/null 2>&1 &
sleep 4
curl -s localhost:9090/api/v1/query \
  --data-urlencode 'query=up{job="monitoring/linkerd-proxy", namespace="pruefung"}' \
  | jq -r '.data.result[] | .metric.pod + " " + .value[1]'
kill %1

Expected (abridged, numbers vary):

     53 /api/notes 200
     20 /flaky 200
      7 /flaky 500
NAME             POLICY
helper-http      deny
notes-api-http   deny
postgres         deny
proxy-admin      deny
web-http         deny
statistik-597fcff454-2qpqf 1
helper-5798bffc7f-7bt64 1
...
postgres-0 1
fremd 1

The annotation on the namespace only takes effect in new Pods, hence the restart (Day 6, Drill 6). fremd has a proxy and the identity default.pruefung – no rule, so 403. A look at kubectl -n pruefung logs deploy/statistik already shows the next error; that is task 14.

Solution 7

The three routes from Day 7, rewritten for pruefung. Without the catch-all route helper-rest, all other paths would run into a 404 (Day 7, Drill 3):

for f in route-retry route-rest route-delay; do
  sed 's/namespace: tag07/namespace: pruefung/' ~/linkerd/tag07/$f.yaml; echo ---
done > routes.yaml
grep -A1 'value: /unstable' routes.yaml
kubectl apply -f routes.yaml
sleep 5
kubectl -n pruefung get httproute \
  -o custom-columns=NAME:.metadata.name,STATUS:.status.parents[0].conditions[0].reason
kubectl -n pruefung exec deploy/last -c last -- \
  curl -s -o /dev/null -w '%{http_code} %{time_total}s\n' http://web/slow
kubectl -n pruefung exec deploy/last -c last -- \
  curl -s -o /dev/null -w '%{http_code}\n' http://web/

Expected:

    - path: {type: PathPrefix, value: /unstable}
      method: GET
httproute.gateway.networking.k8s.io/helper-unstable created
httproute.gateway.networking.k8s.io/helper-rest created
httproute.gateway.networking.k8s.io/helper-delay created
NAME              STATUS
helper-delay      Accepted
helper-rest       Accepted
helper-unstable   Accepted
504 1.044076s
200

After one minute:

kubectl -n pruefung logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
  | sort | uniq -c
kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090 \
  >/dev/null 2>&1 &
sleep 4
linkerd viz stat-outbound -n pruefung deploy/web \
  --prometheusURL=http://localhost:9090 | grep -A1 helper-unstable
kill %1

Expected (abridged, numbers vary):

     70 /api/notes 200
     36 /flaky 200
ROUTE            TYPE       BACKEND    SUCCESS   RPS  ...  TIMEOUTS  RETRIES
helper-unstable  HTTPRoute             100.00%  0.63  ...     0.00%   28.30%
└────────────────────────►  helper:80   71.70%  0.88  ...     0.00%

method: GET in the rule excludes POST, which then runs via helper-rest without a retry (Day 7, Drill 6). Alongside the routes, Cilium creates the EndpointSlice cilium-gateway-helper; it disappears with the namespace during cleanup.

Solution 8

kubectl apply -f kaputt.yaml
kubectl -n pruefung rollout status deploy/notes-api-kaputt --timeout=120s
sleep 60
kubectl -n pruefung logs deploy/last -c last --since=50s | awk '{print $2, $3}' \
  | sort | uniq -c
linkerd viz stat -n pruefung po -l app=notes-api --from deploy/web

Expected (abridged, numbers vary):

     22 /api/notes 200
     50 /api/notes 500
     34 /flaky 200
      2 /flaky 500
NAME                                 STATUS   MESHED   SUCCESS      RPS  ...
notes-api-795f4f4468-pdvd6          Running      1/1   100.00%   0.5rps  ...
notes-api-kaputt-6d4cf657d6-h22mh   Running      1/1     0.00%   1.0rps  ...

The broken Pod even gets more traffic: it answers faster, and the load balancer chooses by latency. It carries the labels and the ServiceAccount of notes-api, so the server and the rule from task 6 let it in. Failure accrual on the Service:

kubectl -n pruefung annotate service notes-api \
  balancer.linkerd.io/failure-accrual=consecutive \
  balancer.linkerd.io/failure-accrual-consecutive-max-failures=3 \
  balancer.linkerd.io/failure-accrual-consecutive-min-penalty=30s \
  balancer.linkerd.io/failure-accrual-consecutive-max-penalty=2m
sleep 75
kubectl -n pruefung logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
  | sort | uniq -c
linkerd viz stat -n pruefung po -l app=notes-api --from deploy/web

Expected (abridged, numbers vary):

service/notes-api annotated
     77 /api/notes 200
      1 /api/notes 500
     37 /flaky 200
      2 /flaky 500
NAME                                 STATUS   MESHED   SUCCESS      RPS  ...
notes-api-795f4f4468-pdvd6          Running      1/1   100.00%   1.4rps  ...
notes-api-kaputt-6d4cf657d6-h22mh   Running      1/1     0.00%   0.0rps  ...

The single 500 is a probe request after the penalty has expired (Day 8, Drill 4). No restart, no change to web: every client proxy reads the annotation on the Service.

Solution 9

kubectl -n pruefung create serviceaccount bericht
kubectl -n pruefung create deployment bericht \
  --image=docker.io/curlimages/curl:8.23.0 -- sleep infinity
kubectl -n pruefung set serviceaccount deploy/bericht bericht
kubectl -n pruefung rollout status deploy/bericht --timeout=120s
kubectl -n pruefung exec deploy/bericht -c curl -- sh -c 'for i in $(seq 20); do
  curl -s -o /dev/null -w "%{http_code}\n" http://helper/get; done' | sort | uniq -c

Expected (abridged):

deployment "bericht" successfully rolled out
     20 403

First the permission (default-deny), then the brake. A rate limit hangs on a Server – here on the existing helper-http:

cat > bericht.yaml <<'EOF'
apiVersion: policy.linkerd.io/v1alpha1
kind: MeshTLSAuthentication
metadata: {name: bericht, namespace: pruefung}
spec:
  identityRefs: [{kind: ServiceAccount, name: bericht}]
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: helper-von-bericht, namespace: pruefung}
spec:
  targetRef: {group: policy.linkerd.io, kind: Server, name: helper-http}
  requiredAuthenticationRefs:
  - {group: policy.linkerd.io, kind: MeshTLSAuthentication, name: bericht}
---
apiVersion: policy.linkerd.io/v1alpha1
kind: HTTPLocalRateLimitPolicy
metadata: {name: helper-rl, namespace: pruefung}
spec:
  targetRef: {group: policy.linkerd.io, kind: Server, name: helper-http}
  total:
    requestsPerSecond: 50
  identity:
    requestsPerSecond: 2
  overrides:
  - requestsPerSecond: 20
    clientRefs:
    - {kind: ServiceAccount, name: web}
EOF
kubectl apply -f bericht.yaml
sleep 10
kubectl -n pruefung exec deploy/bericht -c curl -- sh -c 'for i in $(seq 40); do
  curl -s -o /dev/null -w "%{http_code}\n" http://helper/get; done' | sort | uniq -c
kubectl -n pruefung get httplocalratelimitpolicy
sleep 30
kubectl -n pruefung logs deploy/last -c last --since=40s | awk '{print $2, $3}' \
  | sort | uniq -c
linkerd diagnostics proxy-metrics -n pruefung deploy/helper \
  | grep '^inbound_http_local_ratelimit' \
  | grep -oE 'ratelimit_scope="[^"]*"|client_id="[a-z]+|\} [0-9]+' | paste - - -

Expected (numbers vary):

      9 200
     31 429
NAME        TARGET_KIND   TARGET_NAME   TOTAL_RPS   IDENTITY_RPS
helper-rl   Server        helper-http   50          2
     53 /api/notes 200
      1 /api/notes 500
     27 /flaky 200
ratelimit_scope="identity"  client_id="bericht  } 31

All 429s fall on bericht in the scope identity. total is set high on purpose: rejected requests also consume the shared bucket and would otherwise hit web (Day 8, Drill 6). The single 500 on /api/notes is again the probe request from task 8.

Solution 10

helper-v2 has a label of its own – so the Service helper does not select it, and the server helper-http does not cover it. Under default-deny, it needs its own server with a rule, otherwise web would get 403 there:

cat > helper-v2.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: helper-v2, namespace: pruefung}
spec:
  replicas: 1
  selector: {matchLabels: {app: helper-v2}}
  template:
    metadata: {labels: {app: helper-v2}}
    spec:
      serviceAccountName: helper
      containers:
      - name: helper
        image: ghcr.io/mccutchen/go-httpbin:2.26.0
        ports: [{containerPort: 8080, name: http}]
        readinessProbe:
          httpGet: {path: /status/200, port: http}
---
apiVersion: v1
kind: Service
metadata: {name: helper-v2, namespace: pruefung}
spec:
  selector: {app: helper-v2}
  ports: [{port: 80, targetPort: http, name: http}]
---
apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata: {name: helper-v2-http, namespace: pruefung}
spec:
  podSelector: {matchLabels: {app: helper-v2}}
  port: http
  proxyProtocol: HTTP/1
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: helper-v2-von-web, namespace: pruefung}
spec:
  targetRef: {group: policy.linkerd.io, kind: Server, name: helper-v2-http}
  requiredAuthenticationRefs:
  - {group: policy.linkerd.io, kind: MeshTLSAuthentication, name: web}
EOF
kubectl apply -f helper-v2.yaml
kubectl -n pruefung rollout status deploy/helper-v2 --timeout=120s

The weights go into the existing routes helper-unstable and helper-rest. A second route just for the canary would compete with them for the same paths:

B='[{"name":"helper","port":80,"weight":80},{"name":"helper-v2","port":80,"weight":20}]'
for r in helper-unstable helper-rest; do
  kubectl -n pruefung patch httproute $r --type=json \
    -p "[{\"op\":\"add\",\"path\":\"/spec/rules/0/backendRefs\",\"value\":$B}]"
done
sleep 70
kubectl -n pruefung logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
  | sort | uniq -c
linkerd viz stat -n pruefung deploy --from deploy/web

Expected (abridged, numbers vary):

httproute.gateway.networking.k8s.io/helper-unstable patched
httproute.gateway.networking.k8s.io/helper-rest patched
     77 /api/notes 200
      1 /api/notes 500
     39 /flaky 200
NAME               MESHED   SUCCESS      RPS   LATENCY_P50   LATENCY_P95   ...
helper                1/1    71.43%   0.8rps          15ms         170ms   ...
helper-v2             1/1    53.33%   0.2rps          25ms         185ms   ...
notes-api             1/1   100.00%   1.4rps          12ms          43ms   ...
notes-api-kaputt      1/1     0.00%   0.0rps           0ms           0ms   ...

0.8 to 0.2 – exactly 80/20, and thanks to retries /flaky stays at almost 100 percent. Under the route helper-unstable, linkerd viz stat-outbound now shows two backends, helper:80 and helper-v2:80; right after a restart of web, the shares there fluctuate strongly, so wait a minute. helper-delay stays unchanged, just as the task requires for /delay.

Solution 11

cat > gateway.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata: {name: pruefung, namespace: pruefung}
spec:
  gatewayClassName: cilium
  listeners:
  - {name: http, protocol: HTTP, port: 80}
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: web, namespace: pruefung}
spec:
  parentRefs:
  - {name: pruefung}
  hostnames: [pruefung.lab.internal]
  rules:
  - backendRefs:
    - {name: web, port: 80}
EOF
kubectl apply -f gateway.yaml
sleep 15
NP=$(kubectl -n pruefung get svc cilium-gateway-pruefung \
  -o jsonpath='{.spec.ports[?(@.port==80)].nodePort}')
echo $NP
curl -s -o /dev/null -w '%{http_code}\n' -H 'Host: pruefung.lab.internal' \
  http://172.16.0.126:$NP/api/notes
kubectl get ciliumnodes \
  -o jsonpath='{range .items[*]}{.metadata.name} {.spec.ingress.ipv4}{"\n"}{end}'

Expected:

gateway.gateway.networking.k8s.io/pruefung created
httproute.gateway.networking.k8s.io/web created
30555
403
cp1 10.0.0.235
worker1 10.0.1.169
worker2 10.0.2.57

403: the server web-http only lets last in, and Cilium’s Envoy arrives without an identity from the nodes’ ingress IPs (Day 10, Drill 4). Exactly these three addresses are what you allow (different IPs on your setup):

cat > ingress-authz.yaml <<'EOF'
apiVersion: policy.linkerd.io/v1alpha1
kind: NetworkAuthentication
metadata: {name: cilium-ingress, namespace: pruefung}
spec:
  networks:
  - cidr: 10.0.0.235/32
  - cidr: 10.0.1.169/32
  - cidr: 10.0.2.57/32
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: web-von-gateway, namespace: pruefung}
spec:
  targetRef: {group: policy.linkerd.io, kind: Server, name: web-http}
  requiredAuthenticationRefs:
  - {group: policy.linkerd.io, kind: NetworkAuthentication, name: cilium-ingress}
EOF
kubectl apply -f ingress-authz.yaml
sleep 5
for h in 172.16.0.126 172.16.0.228 172.16.0.83; do
  curl -s -o /dev/null -w "$h %{http_code}\n" -H 'Host: pruefung.lab.internal' \
    http://$h:$NP/api/notes
done
kubectl -n pruefung run ohne --image=docker.io/curlimages/curl:8.23.0 \
  --restart=Never --annotations=linkerd.io/inject=disabled --command -- sleep 3600
kubectl -n pruefung wait --for=condition=Ready pod/ohne --timeout=60s
kubectl -n pruefung exec ohne -- curl -s -o /dev/null -w '%{http_code}\n' \
  http://web/api/notes

Expected:

networkauthentication.policy.linkerd.io/cilium-ingress created
authorizationpolicy.policy.linkerd.io/web-von-gateway created
172.16.0.126 200
172.16.0.228 200
172.16.0.83 200
pod/ohne created
pod/ohne condition met
403

Every node accepts the NodePort and forwards from its own ingress IP – hence all three. ohne has a Pod IP from the Pod network and stays outside.

Solution 12

kubectl -n pruefung exec deploy/last -c last -- curl -s http://web/api/upstream; echo
cat > egress.yaml <<'EOF'
apiVersion: policy.linkerd.io/v1alpha1
kind: EgressNetwork
metadata: {name: internet, namespace: pruefung}
spec:
  trafficPolicy: Deny
---
apiVersion: gateway.networking.k8s.io/v1
kind: TLSRoute
metadata: {name: example-com, namespace: pruefung}
spec:
  parentRefs:
  - {group: policy.linkerd.io, kind: EgressNetwork, name: internet, port: 443}
  hostnames: [example.com]
  rules:
  - backendRefs:
    - {group: policy.linkerd.io, kind: EgressNetwork, name: internet}
EOF
kubectl apply -f egress.yaml
sleep 5
kubectl -n pruefung exec deploy/last -c last -- \
  curl -s -o /dev/null -w '%{http_code}\n' https://linkerd.io
kubectl -n pruefung exec deploy/last -c last -- \
  curl -s -o /dev/null -w '%{http_code}\n' https://example.com
kubectl -n pruefung exec deploy/last -c last -- \
  curl -s -o /dev/null -w '%{http_code}\n' http://example.com

Expected:

{"status":200,"url":"https://example.com"}
egressnetwork.policy.linkerd.io/internet created
tlsroute.gateway.networking.k8s.io/example-com created
000
command terminated with exit code 35
200
403

The proxy answers HTTP with 403 and aborts HTTPS during the handshake (exit code 35). Now the fresh connections: notes-api keeps its connection to example.com open via keep-alive, and a policy only affects new connections (Day 11, Drill 3):

kubectl -n pruefung rollout restart deploy/notes-api
kubectl -n pruefung rollout status deploy/notes-api
kubectl -n pruefung exec deploy/last -c last -- curl -s http://web/api/upstream; echo

Expected:

deployment.apps/notes-api restarted
deployment "notes-api" successfully rolled out
{"status":200,"url":"https://example.com"}

With a new Pod and a new connection, notes-api still gets through – the proof that the TLSRoute takes effect and not just an old connection. The cross-check without the route would be connection reset by peer.

Solution 13

During cleanup, Day 13 deleted the link and the controller controller-west. The values of the extension on ost still contain it, though – applying them again only creates the missing parts:

linkerd --context ost multicluster install -f ~/linkerd/tag12/mc-ost-gw.yaml \
  > mc-ost.yaml
kubectl --context ost apply -f mc-ost.yaml | grep -v unchanged
linkerd --context west multicluster link-gen --cluster-name west > link-west.yaml
grep -E '^kind:|gatewayAddress|gatewayPort' link-west.yaml
kubectl --context ost apply -f link-west.yaml
kubectl --context ost -n linkerd-multicluster rollout status deploy/controller-west \
  --timeout=120s

Expected (abridged):

deployment.apps/controller-west created
service/probe-west created
serviceaccount/controller-west created
...
kind: Secret
kind: Secret
kind: Link
  gatewayAddress: 172.16.0.182
  gatewayPort: "4143"
secret/cluster-credentials-west created
secret/cluster-credentials-west created
link.multicluster.linkerd.io/west created
deployment "controller-west" successfully rolled out

link-gen also runs unchanged against the Helm-managed west (Day 15). Running kubectl diff -f mc-ost.yaml beforehand shows: only the objects for the controller are added. Now the service in west – the namespace pruefung already exists in ost:

kubectl --context west create namespace pruefung
kubectl --context west annotate namespace pruefung linkerd.io/inject=enabled
kubectl --context west -n pruefung create deployment helper \
  --image=ghcr.io/mccutchen/go-httpbin:2.26.0 --port=8080
kubectl --context west -n pruefung set env deploy/helper USE_REAL_HOSTNAME=true
kubectl --context west -n pruefung expose deploy helper --port=80 --target-port=8080
kubectl --context west -n pruefung rollout status deploy/helper --timeout=120s
kubectl --context west -n pruefung label service helper mirror.linkerd.io/exported=true
sleep 60
kubectl --context ost -n pruefung get svc helper-west
linkerd --context ost multicluster gateways
kubectl --context ost -n pruefung exec deploy/last -c last -- \
  curl -s http://helper-west/hostname

Expected (abridged):

service/helper labeled
NAME          TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
helper-west   ClusterIP   10.103.87.18   <none>        80/TCP    40s
CLUSTER  ALIVE    NUM_SVC      LATENCY
west     True           1         60ms
{
  "hostname": "helper-74b8b7dccf-dbxlc"
}

Warning: The new controller-west first waits around 30 seconds in the log at Attempting to acquire leader lease – the lease of its predecessor from Day 12 is still in linkerd-multicluster. Only then does it mirror. Whoever checks right away does not find helper-west yet.

Who arrives in west?

linkerd --context west diagnostics proxy-metrics -n linkerd-multicluster \
  deploy/linkerd-gateway \
  | grep '^tcp_open_total{direction="inbound",peer="src"' \
  | grep -oE '(target_port|client_id|no_tls_reason)="[^"]*"|[0-9]+$' | paste - - -
linkerd --context ost multicluster check 2>&1 | grep -A1 'trust anchors'

Expected (abridged):

target_port="4191"  no_tls_reason="no_tls_from_remote"                       888
target_port="4191"  client_id="controller-west.linkerd-multicluster.servic…"  1
target_port="4143"  client_id="last.pruefung.serviceaccount.identity.linke…"  1
√ clusters share trust anchors
    * west

Port 4143 is the tunnel: last from ost with its full identity. The check compares the anchor from linkerd-config of both clusters – green, because values-west.yaml from Day 14 brings it along as identityTrustAnchorsPEM.

Warning: If you left out the line identityTrustAnchorsPEM on Day 14, the check reports × clusters share trust anchors here with Problematic clusters: * west. With identity.externalCA=true, the linkerd-config of west then contains identityTrustAnchorsPEM: "", and the anchor lives only in the ConfigMap. The fingerprint comparison from Day 12, Drill 4 still shows 8b44a80f…bab81 twice, and the mTLS call above works – a false alarm. The fix (add the line, helm upgrade with the values files) is not required in the exam.

Solution 14

First the symptom, then Postgres’s view:

kubectl -n pruefung logs deploy/statistik -c statistik --tail=3 | cut -c1-80
kubectl -n pruefung logs postgres-0 -c linkerd-proxy --since=1m \
  | grep 'Connection denied' | tail -1 \
  | grep -oE 'server.name=[^ ]*|tls=[^ ]*|client=[^ ]*'
kubectl -n pruefung get pod -l app=statistik -o wide | awk '{print $1, $6}'
kubectl -n pruefung get deploy statistik \
  -o jsonpath='{.spec.template.metadata.annotations}{"\n"}' | cut -c1-80

Expected (abridged):

21:39:34 psql: error: connection to server at "postgres" (10.100.217.87), port 543
    This probably means the server terminated abnormally
    before or while processing the request.
server.name=postgres
tls=None(PortSkipped)
client=10.0.2.173:56718
NAME IP
statistik-597fcff454-2qpqf 10.0.2.173
{"config.linkerd.io/skip-outbound-ports":"5432","kubectl.kubernetes.io/restartedAt

PortSkipped from the IP of the statistik Pod: the proxy of statistik skips port 5432, and psql talks plaintext past the mesh (Day 16, Drill 5). Before task 6, this was accepted – those were the plaintext connections from task 3. Error one:

kubectl -n pruefung patch deploy statistik --type=json -p '[{"op":"remove",
  "path":"/spec/template/metadata/annotations/config.linkerd.io~1skip-outbound-ports"}]'
kubectl -n pruefung rollout status deploy/statistik --timeout=120s
sleep 25
kubectl -n pruefung logs deploy/statistik -c statistik --tail=1
kubectl -n pruefung logs postgres-0 -c linkerd-proxy --since=20s \
  | grep 'Connection denied' | tail -1 | grep -oE 'server.name=[^ ]*|Name\("[^"]*'

Expected (abridged):

deployment "statistik" successfully rolled out
    before or while processing the request.
server.name=postgres
Name("statistik.pruefung.serviceaccount.identity.linkerd.cluster.local

Now statistik arrives with mTLS and its own identity – and is still rejected. Error two: the rule postgres-von-notes-api only knows notes-api. A separate rule for statistik, without touching the existing one:

cat > statistik-authz.yaml <<'EOF'
apiVersion: policy.linkerd.io/v1alpha1
kind: MeshTLSAuthentication
metadata: {name: statistik, namespace: pruefung}
spec:
  identityRefs: [{kind: ServiceAccount, name: statistik}]
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: postgres-von-statistik, namespace: pruefung}
spec:
  targetRef: {group: policy.linkerd.io, kind: Server, name: postgres}
  requiredAuthenticationRefs:
  - {group: policy.linkerd.io, kind: MeshTLSAuthentication, name: statistik}
EOF
kubectl apply -f statistik-authz.yaml
sleep 25
kubectl -n pruefung logs deploy/statistik -c statistik --tail=2
linkerd diagnostics proxy-metrics -n pruefung po/postgres-0 \
  | grep '^inbound_tcp_authz_allow_total' \
  | grep -oE '(authz_name|client_id)="[^"]*"|\} [0-9]+' | paste - - -

Expected (abridged):

meshtlsauthentication.policy.linkerd.io/statistik created
authorizationpolicy.policy.linkerd.io/postgres-von-statistik created
21:40:46 0
21:40:56 0
authz_name="postgres-von-statistik"  client_id="statistik.pruefung.serviceac…"  } 3
authz_name="postgres-von-notes-api"  client_id="notes-api.pruefung.serviceac…"  } 8

Whoever only removes the annotation or only creates the rule gets no points – only both together bring statistik back. Opening Postgres in pruefung to the Pod network via NetworkAuthentication would be the forbidden shortcut.

Solution 15

helm --kube-context west list -n linkerd -o json | jq -r \
  '.[] | select(.name=="linkerd-control-plane") | .chart + " " + .revision' > 15.txt
helm --kube-context west get values linkerd-control-plane -n linkerd --all \
  | grep -E '^(controllerReplicas|webhookFailurePolicy):' >> 15.txt
kubectl --context west -n linkerd get deploy linkerd-destination \
  -o jsonpath='{.status.readyReplicas}{"\n"}' >> 15.txt
cat 15.txt

Expected:

linkerd-control-plane-2026.11.2 5
controllerReplicas: 2
webhookFailurePolicy: Fail
2

Revision 5 after installation, anti-affinity fix, upgrade, rollback and another upgrade (Days 14 and 15). Without --all, webhookFailurePolicy would only appear because it is set in values-ha.yaml – --all shows the effective value, no matter where it comes from.

Solution 16

kubectl -n pruefung get pod -l app=web -o jsonpath='{.items[0].status.qosClass}{"\n"}'
kubectl -n pruefung patch deploy web -p '{"spec":{"template":{"metadata":{
  "annotations":{"config.linkerd.io/proxy-cpu-request":"50m",
  "config.linkerd.io/proxy-memory-request":"32Mi",
  "config.linkerd.io/proxy-memory-limit":"128Mi"}}}}}'
kubectl -n pruefung rollout status deploy/web --timeout=120s
kubectl -n pruefung get pod -l app=web -o \
  jsonpath='{.items[0].spec.initContainers[?(@.name=="linkerd-proxy")].resources}'
echo
kubectl -n pruefung get pod -l app=web -o jsonpath='{.items[0].status.qosClass}{"\n"}'

Expected:

BestEffort
deployment.apps/web patched
deployment "web" successfully rolled out
{"limits":{"memory":"128Mi"},"requests":{"cpu":"50m","memory":"32Mi"}}
Burstable

Without the annotation, the proxy has no resources, and the whole Pod is BestEffort – the first to lose out on full workers (Day 17). The proxy is a native sidecar, which is why it is listed under initContainers. You deliberately did not set a CPU limit: it would also fix the number of the proxy’s worker threads (Day 17, Drill 8).

Scoring

Add up the weights of the tasks that are solved completely and provably – there are no partial points. Whoever bypasses a requirement gets zero for the task: turning off default-deny or accessPolicy: all-unauthenticated, opening the Pod network for web or postgres, deleting the broken Pod, trafficPolicy: Allow, linkerd upgrade or helm upgrade.

Result Assessment
85–100 % confident in operations – from the first proxy to the second cluster
70–84 % the craft is solid; repeat weak spots specifically
below 70 % repeat the days for the tasks you missed and start again
Tasks Area Days
1–3 installation, injection, mTLS (18 %) 1–3
4–5 observing and certificates (11 %) 4, 5
6, 14 authorization and troubleshooting (15 %) 6, 16
7–10 retries, circuit breaking, rate limit, canary (26 %) 7–9
11–12 ingress and egress (13 %) 10, 11
13, 15 multicluster and Helm (12 %) 12–15
16 proxy resources (5 %) 17

Not covered: rotating the trust anchor (Day 5), federated services and cluster failure (Day 13), upgrade and rollback (Day 15) – they change the whole cluster and do not fit into an exam next to running services. Go through these days once more to finish.

Cleanup

First the export and the link, then the namespaces. Since Linkerd 2.18 there is no multicluster unlink; you remove the parts one by one as on Day 13:

kubectl --context west -n pruefung label service helper mirror.linkerd.io/exported-
kubectl --context ost get svc -A -l mirror.linkerd.io/mirrored-service=true
kubectl --context ost -n linkerd-multicluster delete link west
kubectl --context ost delete -l mirror.linkerd.io/cluster-name=west -f mc-ost.yaml
kubectl --context ost -n linkerd-multicluster delete secret cluster-credentials-west
kubectl --context ost -n linkerd delete secret cluster-credentials-west
rm link-west.yaml

Expected (abridged):

service/helper unlabeled
No resources found
link.multicluster.linkerd.io "west" deleted from linkerd-multicluster namespace
deployment.apps "controller-west" deleted from linkerd-multicluster namespace
service "probe-west" deleted from linkerd-multicluster namespace
...
secret "cluster-credentials-west" deleted from linkerd namespace

link-west.yaml contains credentials for the API of west – it does not belong in a working folder. Then the namespaces:

kubectl get endpointslices -A | grep cilium-gateway
kubectl --context west delete namespace pruefung
kubectl delete namespace pruefung --timeout=180s
kubectl get endpointslices -A | grep cilium-gateway | grep -c pruefung

Expected (abridged):

pruefung   cilium-gateway-helper     IPv4   9999   192.192.192.192   13m
pruefung   cilium-gateway-pruefung   IPv4   9999   192.192.192.192   6m20s
namespace "pruefung" deleted
namespace "pruefung" deleted
0

Together with the namespace, routes, Gateway, policies, EgressNetwork, PVC and both of Cilium’s EndpointSlices disappear – including cilium-gateway-helper, which would have been left behind when deleting a single route (Day 9, Drill 7). The files in ~/linkerd/pruefung remain as templates.

Checkpoint

kubectl get ns | grep -c pruefung
kubectl --context west get ns | grep -c pruefung
kubectl get pv | grep -c pruefung
kubectl get links -A

Expected: 0 three times, then No resources found – no link active, as after Day 13.

linkerd --context ost multicluster check 2>&1 | tail -1
linkerd --context ost multicluster gateways

Expected:

Status check results are √
CLUSTER  ALIVE    NUM_SVC      LATENCY

Without a link, the check no longer verifies trust anchors – not even the false alarm from Solution 13, if you had it.

kubectl -n notes get pods
kubectl -n notes logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
  | sort | uniq -c
linkerd check 2>&1 | tail -1

Expected (numbers vary): all eight Pods 2/2 Running, then

     85 /api/notes 200
     31 /flaky 200
     12 /flaky 500
Status check results are √

notes did not notice anything of the entire exam. With that, you have completed the drills.