Appendix B: Final Exam
Goal: You solve 16 practical tasks from across the book in 120 minutes and grade yourself. · Duration: 120 min · Prerequisite: Days 0–18
Rules of Play
- Time: 120 minutes, set a timer. Each task has a weight in percent, 100 % in total.
- There is no certificate. Your own yardstick: from 70 %, the craft is solid.
- Allowed:
linkerd.io/2-edge,gateway-api.sigs.k8s.io,--help,kubectl explainand your own files under~/linkerd/tagNNas templates. Not allowed: this book during the exam. - Everything runs on
cp1in the namespacepruefung, files in~/linkerd/pruefung. Tasks 13 and 15 also addresswest(--context west). - You do not touch
notes. No cluster-wide changes: nolinkerd upgrade, nohelm upgrade, no reinstalling or removing an extension. The only exception is the link in task 13 – it disappears again during cleanup. - Order: tasks 1 to 6 in sequence, task 4 before task 7, task 7 before task 10. From task 8 on, the default-deny from task 6 applies. Otherwise the order is up to you.
- The solutions are at the end. Only read them once the time is up.
Preparation
A copy of the app from the Day 0 manifests, one replica per service,
still without the mesh. In addition,
notes-api with a target on the internet (Day 11) and a
service statistik that a colleague built. Every ten
seconds, it counts the notes directly in Postgres:
mkdir -p ~/linkerd/pruefung && cd ~/linkerd/pruefung
kubectl create namespace pruefung
kubectl -n pruefung create secret generic notes-db \
--from-literal=password=lab-db-123 \
--from-literal=url='postgres://notes:lab-db-123@postgres:5432/notes?sslmode=disable'
kubectl -n pruefung create secret generic notes-api-key --from-literal=key=lab-key-123
for f in postgres notes-api helper web last; do
sed 's/namespace: notes/namespace: pruefung/; s/\.notes\.svc/.pruefung.svc/
s/replicas: 2/replicas: 1/' ~/linkerd/tag00/$f.yaml; echo ---
done > app.yaml
kubectl -n pruefung create serviceaccount statistik
cat > statistik.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: statistik, namespace: pruefung}
spec:
replicas: 1
selector: {matchLabels: {app: statistik}}
template:
metadata:
labels: {app: statistik}
annotations: {config.linkerd.io/skip-outbound-ports: "5432"}
spec:
serviceAccountName: statistik
containers:
- name: statistik
image: docker.io/library/postgres:18.7-alpine
env:
- {name: PGCONNECT_TIMEOUT, value: "3"}
- name: PGPASSWORD
valueFrom: {secretKeyRef: {name: notes-db, key: password}}
command: [sh, -c]
args:
- |
while true; do
n=$(psql -h postgres -U notes -tAc 'select count(*) from notes' 2>&1)
echo "$(date +%T) $n"
sleep 10
done
EOF
kubectl apply -f app.yaml -f statistik.yaml
kubectl -n pruefung set env deploy/notes-api NOTES_UPSTREAM=https://example.com
kubectl -n pruefung rollout status statefulset/postgres --timeout=180s
kubectl -n pruefung rollout status deploy --timeout=240sFor task 8, a broken notes-api is ready – only as a
file, not yet applied. It answers the readiness probe, and everything
else with 500:
cat > kaputt.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata: {name: kaputt-conf, namespace: pruefung}
data:
default.conf: |
server {
listen 8080;
location = /healthz { return 200 "ok\n"; }
location / { return 500 "kaputt\n"; }
}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: notes-api-kaputt, namespace: pruefung}
spec:
replicas: 1
selector: {matchLabels: {app: notes-api, variante: kaputt}}
template:
metadata: {labels: {app: notes-api, variante: kaputt}}
spec:
serviceAccountName: notes-api
containers:
- name: notes-api
image: docker.io/nginxinc/nginx-unprivileged:1.31.7-alpine
ports: [{containerPort: 8080, name: http}]
readinessProbe:
httpGet: {path: /healthz, port: http}
volumeMounts: [{name: conf, mountPath: /etc/nginx/conf.d}]
volumes: [{name: conf, configMap: {name: kaputt-conf}}]
EOFAfter one minute, the baseline:
kubectl -n pruefung logs deploy/last --since=35s | awk '{print $2, $3}' | sort | uniq -c
kubectl -n pruefung logs deploy/statistik --tail=2Expected (numbers vary):
58 /api/notes 200
20 /flaky 200
9 /flaky 500
21:23:34 0
21:23:44 0
The 0 is the number of notes in the new database. Start
the clock.
Task 1: Inventory (5 %)
Write four lines to ~/linkerd/pruefung/01.txt: the
version of the control plane, the number of Linkerd CRDs in the cluster,
the last line of linkerd check and the names of the
installed extensions (one line, separated by spaces).
Task 2: Into the Mesh (7 %)
Bring pruefung into the mesh. Each of the five app
services gets its own ServiceAccount with its name;
statistik already has one. Afterwards, all Pods run with a
proxy (2/2), and
linkerd check --proxy -n pruefung reports no error and no
warning about opaque ports.
Task 3: Who Talks to Postgres? (6 %)
Without tcpdump: write to 03.txt the
identity with which notes-api arrives at
postgres-0 (line 1), and the number of plaintext
connections that postgres-0 has accepted on port 5432 so
far (line 2). Also show with one command that all HTTP edges in
pruefung run over mTLS.
Task 4: Golden Metrics (6 %)
Solve before task 7. Write to 04.txt the success rate of
helper as seen from web, as a decimal number –
once according to linkerd viz (line 1), once via PromQL
over five minutes directly from Prometheus (line 2).
Task 5: The Auditors Ask About Certificates (5 %)
Read only. Write to 05.txt: when the trust anchor in the
cluster expires, when the issuer expires and when it is renewed, and two
fingerprints proving that exactly the anchor from
~/linkerd/tag05/ca.crt is distributed in the cluster.
Task 6: Default-Deny (8 %)
In pruefung, only those who must may talk:
last → web → notes-api → postgres and
web → helper. New ports without a rule are blocked. Probes
and the Prometheus scrapes (up = 1 for every proxy in
pruefung) keep working, and the load generator shows the
same rates as before. Use a Pod fremd (with a proxy,
ServiceAccount default) to prove that it gets 403 from
helper.
Task 7: Retries and Timeouts (7 %)
/flaky succeeds at least 95 percent of the time: the
client proxy of web retries GET requests to
/unstable on 5xx at most twice, other methods never. All
other paths of helper stay reachable. /slow
aborts with 504 after one second.
Task 8: A Pod That Lies (6 %)
Start the broken Pod with kubectl apply -f kaputt.yaml.
It is ready, but only returns 500. Without deleting it, scaling it or
changing its labels: every client should avoid it after three
consecutive failures for at least 30 seconds, at most two minutes.
Afterwards, /api/notes is back above 99 percent. Prove per
Pod who still gets traffic from web.
Task 9: A Brake for the Report (7 %)
Create a Deployment bericht
(curlimages/curl:8.23.0, sleep infinity, its
own ServiceAccount bericht). bericht may call
helper directly, but at most 2 requests per second and Pod.
web may send up to 20, and each helper Pod
accepts at most 50 in total. Proof: 40 fast calls from
bericht end mostly with 429, and the load generator does
not see a single one.
Task 10: Canary (6 %)
A new version helper-v2 (go-httpbin 2.26.0, label
app: helper-v2, ServiceAccount helper, Service
helper-v2 on port 80) gets 20 percent of the calls from
web to helper – except /delay,
which stays with the old version. The retries from task 7 stay in
effect, default-deny stays. Prove the distribution.
Task 11: The Entrance (7 %)
A Cilium Gateway pruefung accepts HTTP on port 80 for
the host pruefung.lab.internal and forwards to
web. From cp1, /api/notes returns
200 via the Gateway’s NodePort on every node. A Pod
without a proxy that calls web directly still gets 403 –
opening up the Pod network does not count.
Task 12: Only example.com (6 %)
Outside the cluster, Pods in pruefung can only reach
https://example.com. /api/upstream via
web still returns 200, https://linkerd.io and
http://example.com fail from last. This must
also hold for fresh connections from notes-api.
Task 13: A helper from west (8 %)
On west, a helper runs in the mesh in the
namespace pruefung (go-httpbin 2.26.0 with
USE_REAL_HOSTNAME=true, Service port 80). In
ost, it appears as helper-west, and
last gets a Pod name from west under
http://helper-west/hostname. Prove at the gateway in
west with which identity last arrives. At the
start, no link is active (Day 13, Cleanup).
Task 14: The Statistics Stop Counting (7 %)
Since task 6, statistik only reports errors. Repair the
service so that it may count with its own identity. Default-deny stays,
and postgres still accepts neither plaintext nor foreign
identities.
Task 15: Keeping an Eye on west (4 %)
Read only. Write to 15.txt the chart and revision of the
release linkerd-control-plane on west (line
1), the effective values controllerReplicas and
webhookFailurePolicy (lines 2 and 3) and the number of
ready replicas of linkerd-destination (line 4).
Task 16: A Proxy on a Budget (5 %)
The proxy of web gets 50m CPU and 32 MiB memory as a
request and 128 MiB as a limit – by annotation only, the app itself
stays without resources. Show the resources of the proxy container and
the QoS class of the Pod before and after.
Stop the clock.
Solutions
All commands on cp1 in the folder
~/linkerd/pruefung.
Solution 1
linkerd version --short | tail -1 > 01.txt
kubectl get crd -o name | grep -c linkerd >> 01.txt
linkerd check 2>&1 | tail -1 >> 01.txt
kubectl get ns -l linkerd.io/extension \
-o jsonpath='{.items[*].metadata.labels.linkerd\.io/extension}{"\n"}' >> 01.txt
cat 01.txtExpected:
edge-26.11.2
11
Status check results are √
multicluster viz
--short shows the CLI first, then the control plane –
tail -1 takes the second. Eleven CRDs: the ten from Day 1
plus links.multicluster.linkerd.io.
linkerd check recognizes extensions by the label
linkerd.io/extension on their namespace (Day 15).
Solution 2
kubectl annotate namespace pruefung linkerd.io/inject=enabled
for a in web notes-api helper last postgres; do
kubectl -n pruefung create serviceaccount $a
done
for d in web notes-api helper last; do
kubectl -n pruefung set serviceaccount deploy/$d $d
done
kubectl -n pruefung set serviceaccount statefulset/postgres postgres
kubectl -n pruefung annotate service postgres config.linkerd.io/opaque-ports=5432
kubectl -n pruefung rollout restart deploy/statistik
kubectl -n pruefung rollout status statefulset/postgres --timeout=180s
kubectl -n pruefung rollout status deploy --timeout=240s
kubectl -n pruefung get pods
linkerd check --proxy -n pruefung 2>&1 | grep -E '‼|×|opaque|Status'Expected (abridged):
NAME READY STATUS RESTARTS AGE
helper-79d57fbbbb-bg874 2/2 Running 0 24s
last-78895758f9-wvbz6 2/2 Running 0 24s
notes-api-6ff48b7779-86v22 2/2 Running 0 24s
postgres-0 2/2 Running 0 21s
statistik-64cf4b6f5b-fvwxw 2/2 Running 0 22s
web-7d94ccc84d-tp7wk 2/2 Running 0 25s
‼ issuer cert is valid for at least 60 days
√ opaque ports are properly annotated
Status check results are √
set serviceaccount changes the Pod template and restarts
the four Deployments and Postgres anyway – with a proxy, because the
namespace is already annotated. Only statistik needs a
restart of its own (Day 2). You know the issuer warning from Day 5; it
is not a finding.
Solution 3
linkerd diagnostics proxy-metrics -n pruefung po/postgres-0 \
| grep '^tcp_open_total{direction="inbound",peer="src"' \
| grep 'target_port="5432"' > pg-tcp.txt
grep -o 'client_id="[^"]*' pg-tcp.txt | cut -d'"' -f2 > 03.txt
grep port_skipped pg-tcp.txt | grep -oE '[0-9]+$' >> 03.txt
cat 03.txt
linkerd viz edges -n pruefung deployExpected (number varies):
notes-api.pruefung.serviceaccount.identity.linkerd.cluster.local
11
SRC DST SRC_NS DST_NS SECURED
last web pruefung pruefung √
web helper pruefung pruefung √
web notes-api pruefung pruefung √
The plaintext connections carry
no_tls_reason="port_skipped" – someone is talking to
Postgres past the proxy (Day 3, Drill 7). Who that is, task 14
clarifies. edges shows every HTTP edge with √;
Postgres is missing because it is a StatefulSet.
Solution 4
linkerd viz stat -n pruefung deploy/helper --from deploy/web
linkerd viz stat -n pruefung deploy/helper --from deploy/web -o json \
| jq -r '.[0].success' > 04.txt
kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090 \
>/dev/null 2>&1 &
sleep 4
Q='sum(rate(response_total{namespace="pruefung", deployment="web",
direction="outbound", dst_deployment="helper", classification="success"}[5m]))
/ sum(rate(response_total{namespace="pruefung", deployment="web",
direction="outbound", dst_deployment="helper"}[5m]))'
curl -s localhost:9090/api/v1/query --data-urlencode "query=$Q" \
| jq -r '.data.result[0].value[1]' >> 04.txt
kill %1
cat 04.txtExpected (numbers vary):
NAME MESHED SUCCESS RPS LATENCY_P50 LATENCY_P95 LATENCY_P99 TCP_CONN
helper 1/1 71.43% 0.7rps 8ms 36ms 39ms 1
0.7142857142857143
0.6820478364789672
Both around 0.7 – the baseline of /flaky.
--from deploy/web leaves out the readiness probes that
embellish the inbound rate of helper (Day 4, Drill 4).
viz calculates over one minute, the PromQL query over
five.
Solution 5
kubectl -n linkerd get configmap linkerd-identity-trust-roots \
-o jsonpath='{.data.ca-bundle\.crt}' | openssl x509 -noout -enddate > 05.txt
kubectl -n linkerd get certificate linkerd-identity-issuer \
-o jsonpath='{.status.notAfter} {.status.renewalTime}{"\n"}' >> 05.txt
kubectl -n linkerd get configmap linkerd-identity-trust-roots \
-o jsonpath='{.data.ca-bundle\.crt}' | step certificate fingerprint - >> 05.txt
step certificate fingerprint ~/linkerd/tag05/ca.crt >> 05.txt
cat 05.txtExpected (dates depend on your run):
notAfter=Nov 16 15:51:50 2036 GMT
2026-11-21T16:02:00Z 2026-11-20T15:02:00Z
8b44a80f0940a58823716aecd2aeac2895e43d94bd968826ec821389f65bab81
8b44a80f0940a58823716aecd2aeac2895e43d94bd968826ec821389f65bab81
Ten years for the anchor, 48 hours for the issuer, renewal 25 hours
before expiry (Day 5). Identical fingerprints: the cluster trusts
exactly your ca.crt.
Solution 6
The rules from Day 6 fit one to one. proxy-admin comes
along before the default policy is switched – otherwise
minutes of metrics are missing:
for f in servers authz admin; do
sed 's/namespace: notes/namespace: pruefung/' ~/linkerd/tag06/$f.yaml; echo ---
done > policy.yaml
grep -c '^kind:' policy.yaml
kubectl apply -f policy.yaml
kubectl annotate namespace pruefung config.linkerd.io/default-inbound-policy=deny
kubectl -n pruefung rollout restart deploy
kubectl -n pruefung rollout restart statefulset/postgres
kubectl -n pruefung rollout status statefulset/postgres --timeout=180s
kubectl -n pruefung rollout status deploy --timeout=240s
kubectl -n pruefung run fremd --image=docker.io/curlimages/curl:8.23.0 \
--command -- sleep 3600
kubectl -n pruefung wait --for=condition=Ready pod/fremd --timeout=90s
kubectl -n pruefung exec fremd -c fremd -- curl -s -o /dev/null -w '%{http_code}\n' \
http://helper/getExpected (abridged):
14
server.policy.linkerd.io/web-http created
...
authorizationpolicy.policy.linkerd.io/admin-von-prometheus created
namespace/pruefung annotated
...
pod/fremd condition met
403
One minute later, the cross-check:
kubectl -n pruefung logs deploy/last -c last --since=40s | awk '{print $2, $3}' \
| sort | uniq -c
kubectl -n pruefung get server -o custom-columns=NAME:.metadata.name,\
POLICY:.spec.accessPolicy
kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090 \
>/dev/null 2>&1 &
sleep 4
curl -s localhost:9090/api/v1/query \
--data-urlencode 'query=up{job="monitoring/linkerd-proxy", namespace="pruefung"}' \
| jq -r '.data.result[] | .metric.pod + " " + .value[1]'
kill %1Expected (abridged, numbers vary):
53 /api/notes 200
20 /flaky 200
7 /flaky 500
NAME POLICY
helper-http deny
notes-api-http deny
postgres deny
proxy-admin deny
web-http deny
statistik-597fcff454-2qpqf 1
helper-5798bffc7f-7bt64 1
...
postgres-0 1
fremd 1
The annotation on the namespace only takes effect in new Pods, hence
the restart (Day 6, Drill 6). fremd has a proxy and the
identity default.pruefung – no rule, so 403. A look at
kubectl -n pruefung logs deploy/statistik already shows the
next error; that is task 14.
Solution 7
The three routes from Day 7, rewritten for pruefung.
Without the catch-all route helper-rest, all other paths
would run into a 404 (Day 7, Drill 3):
for f in route-retry route-rest route-delay; do
sed 's/namespace: tag07/namespace: pruefung/' ~/linkerd/tag07/$f.yaml; echo ---
done > routes.yaml
grep -A1 'value: /unstable' routes.yaml
kubectl apply -f routes.yaml
sleep 5
kubectl -n pruefung get httproute \
-o custom-columns=NAME:.metadata.name,STATUS:.status.parents[0].conditions[0].reason
kubectl -n pruefung exec deploy/last -c last -- \
curl -s -o /dev/null -w '%{http_code} %{time_total}s\n' http://web/slow
kubectl -n pruefung exec deploy/last -c last -- \
curl -s -o /dev/null -w '%{http_code}\n' http://web/Expected:
- path: {type: PathPrefix, value: /unstable}
method: GET
httproute.gateway.networking.k8s.io/helper-unstable created
httproute.gateway.networking.k8s.io/helper-rest created
httproute.gateway.networking.k8s.io/helper-delay created
NAME STATUS
helper-delay Accepted
helper-rest Accepted
helper-unstable Accepted
504 1.044076s
200
After one minute:
kubectl -n pruefung logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
| sort | uniq -c
kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090 \
>/dev/null 2>&1 &
sleep 4
linkerd viz stat-outbound -n pruefung deploy/web \
--prometheusURL=http://localhost:9090 | grep -A1 helper-unstable
kill %1Expected (abridged, numbers vary):
70 /api/notes 200
36 /flaky 200
ROUTE TYPE BACKEND SUCCESS RPS ... TIMEOUTS RETRIES
helper-unstable HTTPRoute 100.00% 0.63 ... 0.00% 28.30%
└────────────────────────► helper:80 71.70% 0.88 ... 0.00%
method: GET in the rule excludes POST,
which then runs via helper-rest without a retry (Day 7,
Drill 6). Alongside the routes, Cilium creates the EndpointSlice
cilium-gateway-helper; it disappears with the namespace
during cleanup.
Solution 8
kubectl apply -f kaputt.yaml
kubectl -n pruefung rollout status deploy/notes-api-kaputt --timeout=120s
sleep 60
kubectl -n pruefung logs deploy/last -c last --since=50s | awk '{print $2, $3}' \
| sort | uniq -c
linkerd viz stat -n pruefung po -l app=notes-api --from deploy/webExpected (abridged, numbers vary):
22 /api/notes 200
50 /api/notes 500
34 /flaky 200
2 /flaky 500
NAME STATUS MESHED SUCCESS RPS ...
notes-api-795f4f4468-pdvd6 Running 1/1 100.00% 0.5rps ...
notes-api-kaputt-6d4cf657d6-h22mh Running 1/1 0.00% 1.0rps ...
The broken Pod even gets more traffic: it answers
faster, and the load balancer chooses by latency. It carries the labels
and the ServiceAccount of notes-api, so the server and the
rule from task 6 let it in. Failure accrual on the Service:
kubectl -n pruefung annotate service notes-api \
balancer.linkerd.io/failure-accrual=consecutive \
balancer.linkerd.io/failure-accrual-consecutive-max-failures=3 \
balancer.linkerd.io/failure-accrual-consecutive-min-penalty=30s \
balancer.linkerd.io/failure-accrual-consecutive-max-penalty=2m
sleep 75
kubectl -n pruefung logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
| sort | uniq -c
linkerd viz stat -n pruefung po -l app=notes-api --from deploy/webExpected (abridged, numbers vary):
service/notes-api annotated
77 /api/notes 200
1 /api/notes 500
37 /flaky 200
2 /flaky 500
NAME STATUS MESHED SUCCESS RPS ...
notes-api-795f4f4468-pdvd6 Running 1/1 100.00% 1.4rps ...
notes-api-kaputt-6d4cf657d6-h22mh Running 1/1 0.00% 0.0rps ...
The single 500 is a probe request after the penalty has
expired (Day 8, Drill 4). No restart, no change to web:
every client proxy reads the annotation on the Service.
Solution 9
kubectl -n pruefung create serviceaccount bericht
kubectl -n pruefung create deployment bericht \
--image=docker.io/curlimages/curl:8.23.0 -- sleep infinity
kubectl -n pruefung set serviceaccount deploy/bericht bericht
kubectl -n pruefung rollout status deploy/bericht --timeout=120s
kubectl -n pruefung exec deploy/bericht -c curl -- sh -c 'for i in $(seq 20); do
curl -s -o /dev/null -w "%{http_code}\n" http://helper/get; done' | sort | uniq -cExpected (abridged):
deployment "bericht" successfully rolled out
20 403
First the permission (default-deny), then the brake. A rate limit
hangs on a Server – here on the existing
helper-http:
cat > bericht.yaml <<'EOF'
apiVersion: policy.linkerd.io/v1alpha1
kind: MeshTLSAuthentication
metadata: {name: bericht, namespace: pruefung}
spec:
identityRefs: [{kind: ServiceAccount, name: bericht}]
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: helper-von-bericht, namespace: pruefung}
spec:
targetRef: {group: policy.linkerd.io, kind: Server, name: helper-http}
requiredAuthenticationRefs:
- {group: policy.linkerd.io, kind: MeshTLSAuthentication, name: bericht}
---
apiVersion: policy.linkerd.io/v1alpha1
kind: HTTPLocalRateLimitPolicy
metadata: {name: helper-rl, namespace: pruefung}
spec:
targetRef: {group: policy.linkerd.io, kind: Server, name: helper-http}
total:
requestsPerSecond: 50
identity:
requestsPerSecond: 2
overrides:
- requestsPerSecond: 20
clientRefs:
- {kind: ServiceAccount, name: web}
EOF
kubectl apply -f bericht.yaml
sleep 10
kubectl -n pruefung exec deploy/bericht -c curl -- sh -c 'for i in $(seq 40); do
curl -s -o /dev/null -w "%{http_code}\n" http://helper/get; done' | sort | uniq -c
kubectl -n pruefung get httplocalratelimitpolicy
sleep 30
kubectl -n pruefung logs deploy/last -c last --since=40s | awk '{print $2, $3}' \
| sort | uniq -c
linkerd diagnostics proxy-metrics -n pruefung deploy/helper \
| grep '^inbound_http_local_ratelimit' \
| grep -oE 'ratelimit_scope="[^"]*"|client_id="[a-z]+|\} [0-9]+' | paste - - -Expected (numbers vary):
9 200
31 429
NAME TARGET_KIND TARGET_NAME TOTAL_RPS IDENTITY_RPS
helper-rl Server helper-http 50 2
53 /api/notes 200
1 /api/notes 500
27 /flaky 200
ratelimit_scope="identity" client_id="bericht } 31
All 429s fall on bericht in the scope
identity. total is set high on purpose:
rejected requests also consume the shared bucket and would otherwise hit
web (Day 8, Drill 6). The single 500 on
/api/notes is again the probe request from task 8.
Solution 10
helper-v2 has a label of its own – so the Service
helper does not select it, and the server
helper-http does not cover it. Under default-deny, it needs
its own server with a rule, otherwise web would get 403
there:
cat > helper-v2.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: helper-v2, namespace: pruefung}
spec:
replicas: 1
selector: {matchLabels: {app: helper-v2}}
template:
metadata: {labels: {app: helper-v2}}
spec:
serviceAccountName: helper
containers:
- name: helper
image: ghcr.io/mccutchen/go-httpbin:2.26.0
ports: [{containerPort: 8080, name: http}]
readinessProbe:
httpGet: {path: /status/200, port: http}
---
apiVersion: v1
kind: Service
metadata: {name: helper-v2, namespace: pruefung}
spec:
selector: {app: helper-v2}
ports: [{port: 80, targetPort: http, name: http}]
---
apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata: {name: helper-v2-http, namespace: pruefung}
spec:
podSelector: {matchLabels: {app: helper-v2}}
port: http
proxyProtocol: HTTP/1
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: helper-v2-von-web, namespace: pruefung}
spec:
targetRef: {group: policy.linkerd.io, kind: Server, name: helper-v2-http}
requiredAuthenticationRefs:
- {group: policy.linkerd.io, kind: MeshTLSAuthentication, name: web}
EOF
kubectl apply -f helper-v2.yaml
kubectl -n pruefung rollout status deploy/helper-v2 --timeout=120sThe weights go into the existing routes
helper-unstable and helper-rest. A second
route just for the canary would compete with them for the same
paths:
B='[{"name":"helper","port":80,"weight":80},{"name":"helper-v2","port":80,"weight":20}]'
for r in helper-unstable helper-rest; do
kubectl -n pruefung patch httproute $r --type=json \
-p "[{\"op\":\"add\",\"path\":\"/spec/rules/0/backendRefs\",\"value\":$B}]"
done
sleep 70
kubectl -n pruefung logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
| sort | uniq -c
linkerd viz stat -n pruefung deploy --from deploy/webExpected (abridged, numbers vary):
httproute.gateway.networking.k8s.io/helper-unstable patched
httproute.gateway.networking.k8s.io/helper-rest patched
77 /api/notes 200
1 /api/notes 500
39 /flaky 200
NAME MESHED SUCCESS RPS LATENCY_P50 LATENCY_P95 ...
helper 1/1 71.43% 0.8rps 15ms 170ms ...
helper-v2 1/1 53.33% 0.2rps 25ms 185ms ...
notes-api 1/1 100.00% 1.4rps 12ms 43ms ...
notes-api-kaputt 1/1 0.00% 0.0rps 0ms 0ms ...
0.8 to 0.2 – exactly 80/20, and thanks to retries /flaky
stays at almost 100 percent. Under the route
helper-unstable, linkerd viz stat-outbound now
shows two backends, helper:80 and
helper-v2:80; right after a restart of web,
the shares there fluctuate strongly, so wait a minute.
helper-delay stays unchanged, just as the task requires for
/delay.
Solution 11
cat > gateway.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata: {name: pruefung, namespace: pruefung}
spec:
gatewayClassName: cilium
listeners:
- {name: http, protocol: HTTP, port: 80}
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: web, namespace: pruefung}
spec:
parentRefs:
- {name: pruefung}
hostnames: [pruefung.lab.internal]
rules:
- backendRefs:
- {name: web, port: 80}
EOF
kubectl apply -f gateway.yaml
sleep 15
NP=$(kubectl -n pruefung get svc cilium-gateway-pruefung \
-o jsonpath='{.spec.ports[?(@.port==80)].nodePort}')
echo $NP
curl -s -o /dev/null -w '%{http_code}\n' -H 'Host: pruefung.lab.internal' \
http://172.16.0.126:$NP/api/notes
kubectl get ciliumnodes \
-o jsonpath='{range .items[*]}{.metadata.name} {.spec.ingress.ipv4}{"\n"}{end}'Expected:
gateway.gateway.networking.k8s.io/pruefung created
httproute.gateway.networking.k8s.io/web created
30555
403
cp1 10.0.0.235
worker1 10.0.1.169
worker2 10.0.2.57
403: the server web-http only lets last in,
and Cilium’s Envoy arrives without an identity from the nodes’ ingress
IPs (Day 10, Drill 4). Exactly these three addresses are what you allow
(different IPs on your setup):
cat > ingress-authz.yaml <<'EOF'
apiVersion: policy.linkerd.io/v1alpha1
kind: NetworkAuthentication
metadata: {name: cilium-ingress, namespace: pruefung}
spec:
networks:
- cidr: 10.0.0.235/32
- cidr: 10.0.1.169/32
- cidr: 10.0.2.57/32
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: web-von-gateway, namespace: pruefung}
spec:
targetRef: {group: policy.linkerd.io, kind: Server, name: web-http}
requiredAuthenticationRefs:
- {group: policy.linkerd.io, kind: NetworkAuthentication, name: cilium-ingress}
EOF
kubectl apply -f ingress-authz.yaml
sleep 5
for h in 172.16.0.126 172.16.0.228 172.16.0.83; do
curl -s -o /dev/null -w "$h %{http_code}\n" -H 'Host: pruefung.lab.internal' \
http://$h:$NP/api/notes
done
kubectl -n pruefung run ohne --image=docker.io/curlimages/curl:8.23.0 \
--restart=Never --annotations=linkerd.io/inject=disabled --command -- sleep 3600
kubectl -n pruefung wait --for=condition=Ready pod/ohne --timeout=60s
kubectl -n pruefung exec ohne -- curl -s -o /dev/null -w '%{http_code}\n' \
http://web/api/notesExpected:
networkauthentication.policy.linkerd.io/cilium-ingress created
authorizationpolicy.policy.linkerd.io/web-von-gateway created
172.16.0.126 200
172.16.0.228 200
172.16.0.83 200
pod/ohne created
pod/ohne condition met
403
Every node accepts the NodePort and forwards from its own ingress IP
– hence all three. ohne has a Pod IP from the Pod network
and stays outside.
Solution 12
kubectl -n pruefung exec deploy/last -c last -- curl -s http://web/api/upstream; echo
cat > egress.yaml <<'EOF'
apiVersion: policy.linkerd.io/v1alpha1
kind: EgressNetwork
metadata: {name: internet, namespace: pruefung}
spec:
trafficPolicy: Deny
---
apiVersion: gateway.networking.k8s.io/v1
kind: TLSRoute
metadata: {name: example-com, namespace: pruefung}
spec:
parentRefs:
- {group: policy.linkerd.io, kind: EgressNetwork, name: internet, port: 443}
hostnames: [example.com]
rules:
- backendRefs:
- {group: policy.linkerd.io, kind: EgressNetwork, name: internet}
EOF
kubectl apply -f egress.yaml
sleep 5
kubectl -n pruefung exec deploy/last -c last -- \
curl -s -o /dev/null -w '%{http_code}\n' https://linkerd.io
kubectl -n pruefung exec deploy/last -c last -- \
curl -s -o /dev/null -w '%{http_code}\n' https://example.com
kubectl -n pruefung exec deploy/last -c last -- \
curl -s -o /dev/null -w '%{http_code}\n' http://example.comExpected:
{"status":200,"url":"https://example.com"}
egressnetwork.policy.linkerd.io/internet created
tlsroute.gateway.networking.k8s.io/example-com created
000
command terminated with exit code 35
200
403
The proxy answers HTTP with 403 and aborts HTTPS during the handshake
(exit code 35). Now the fresh connections: notes-api keeps
its connection to example.com open via keep-alive, and a
policy only affects new connections (Day 11, Drill 3):
kubectl -n pruefung rollout restart deploy/notes-api
kubectl -n pruefung rollout status deploy/notes-api
kubectl -n pruefung exec deploy/last -c last -- curl -s http://web/api/upstream; echoExpected:
deployment.apps/notes-api restarted
deployment "notes-api" successfully rolled out
{"status":200,"url":"https://example.com"}
With a new Pod and a new connection, notes-api still
gets through – the proof that the TLSRoute takes effect and not just an
old connection. The cross-check without the route would be
connection reset by peer.
Solution 13
During cleanup, Day 13 deleted the link and the
controller controller-west. The values of the extension on
ost still contain it, though – applying them again only
creates the missing parts:
linkerd --context ost multicluster install -f ~/linkerd/tag12/mc-ost-gw.yaml \
> mc-ost.yaml
kubectl --context ost apply -f mc-ost.yaml | grep -v unchanged
linkerd --context west multicluster link-gen --cluster-name west > link-west.yaml
grep -E '^kind:|gatewayAddress|gatewayPort' link-west.yaml
kubectl --context ost apply -f link-west.yaml
kubectl --context ost -n linkerd-multicluster rollout status deploy/controller-west \
--timeout=120sExpected (abridged):
deployment.apps/controller-west created
service/probe-west created
serviceaccount/controller-west created
...
kind: Secret
kind: Secret
kind: Link
gatewayAddress: 172.16.0.182
gatewayPort: "4143"
secret/cluster-credentials-west created
secret/cluster-credentials-west created
link.multicluster.linkerd.io/west created
deployment "controller-west" successfully rolled out
link-gen also runs unchanged against the Helm-managed
west (Day 15). Running
kubectl diff -f mc-ost.yaml beforehand shows: only the
objects for the controller are added. Now the service in
west – the namespace pruefung already exists
in ost:
kubectl --context west create namespace pruefung
kubectl --context west annotate namespace pruefung linkerd.io/inject=enabled
kubectl --context west -n pruefung create deployment helper \
--image=ghcr.io/mccutchen/go-httpbin:2.26.0 --port=8080
kubectl --context west -n pruefung set env deploy/helper USE_REAL_HOSTNAME=true
kubectl --context west -n pruefung expose deploy helper --port=80 --target-port=8080
kubectl --context west -n pruefung rollout status deploy/helper --timeout=120s
kubectl --context west -n pruefung label service helper mirror.linkerd.io/exported=true
sleep 60
kubectl --context ost -n pruefung get svc helper-west
linkerd --context ost multicluster gateways
kubectl --context ost -n pruefung exec deploy/last -c last -- \
curl -s http://helper-west/hostnameExpected (abridged):
service/helper labeled
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
helper-west ClusterIP 10.103.87.18 <none> 80/TCP 40s
CLUSTER ALIVE NUM_SVC LATENCY
west True 1 60ms
{
"hostname": "helper-74b8b7dccf-dbxlc"
}
Warning: The new
controller-westfirst waits around 30 seconds in the log atAttempting to acquire leader lease– the lease of its predecessor from Day 12 is still inlinkerd-multicluster. Only then does it mirror. Whoever checks right away does not findhelper-westyet.
Who arrives in west?
linkerd --context west diagnostics proxy-metrics -n linkerd-multicluster \
deploy/linkerd-gateway \
| grep '^tcp_open_total{direction="inbound",peer="src"' \
| grep -oE '(target_port|client_id|no_tls_reason)="[^"]*"|[0-9]+$' | paste - - -
linkerd --context ost multicluster check 2>&1 | grep -A1 'trust anchors'Expected (abridged):
target_port="4191" no_tls_reason="no_tls_from_remote" 888
target_port="4191" client_id="controller-west.linkerd-multicluster.servic…" 1
target_port="4143" client_id="last.pruefung.serviceaccount.identity.linke…" 1
√ clusters share trust anchors
* west
Port 4143 is the tunnel: last from ost with
its full identity. The check compares the anchor from
linkerd-config of both clusters – green, because
values-west.yaml from Day 14 brings it along as
identityTrustAnchorsPEM.
Warning: If you left out the line
identityTrustAnchorsPEMon Day 14, the check reports× clusters share trust anchorshere withProblematic clusters: * west. Withidentity.externalCA=true, thelinkerd-configofwestthen containsidentityTrustAnchorsPEM: "", and the anchor lives only in the ConfigMap. The fingerprint comparison from Day 12, Drill 4 still shows8b44a80f…bab81twice, and the mTLS call above works – a false alarm. The fix (add the line,helm upgradewith the values files) is not required in the exam.
Solution 14
First the symptom, then Postgres’s view:
kubectl -n pruefung logs deploy/statistik -c statistik --tail=3 | cut -c1-80
kubectl -n pruefung logs postgres-0 -c linkerd-proxy --since=1m \
| grep 'Connection denied' | tail -1 \
| grep -oE 'server.name=[^ ]*|tls=[^ ]*|client=[^ ]*'
kubectl -n pruefung get pod -l app=statistik -o wide | awk '{print $1, $6}'
kubectl -n pruefung get deploy statistik \
-o jsonpath='{.spec.template.metadata.annotations}{"\n"}' | cut -c1-80Expected (abridged):
21:39:34 psql: error: connection to server at "postgres" (10.100.217.87), port 543
This probably means the server terminated abnormally
before or while processing the request.
server.name=postgres
tls=None(PortSkipped)
client=10.0.2.173:56718
NAME IP
statistik-597fcff454-2qpqf 10.0.2.173
{"config.linkerd.io/skip-outbound-ports":"5432","kubectl.kubernetes.io/restartedAt
PortSkipped from the IP of the statistik
Pod: the proxy of statistik skips port 5432, and
psql talks plaintext past the mesh (Day 16, Drill 5).
Before task 6, this was accepted – those were the plaintext connections
from task 3. Error one:
kubectl -n pruefung patch deploy statistik --type=json -p '[{"op":"remove",
"path":"/spec/template/metadata/annotations/config.linkerd.io~1skip-outbound-ports"}]'
kubectl -n pruefung rollout status deploy/statistik --timeout=120s
sleep 25
kubectl -n pruefung logs deploy/statistik -c statistik --tail=1
kubectl -n pruefung logs postgres-0 -c linkerd-proxy --since=20s \
| grep 'Connection denied' | tail -1 | grep -oE 'server.name=[^ ]*|Name\("[^"]*'Expected (abridged):
deployment "statistik" successfully rolled out
before or while processing the request.
server.name=postgres
Name("statistik.pruefung.serviceaccount.identity.linkerd.cluster.local
Now statistik arrives with mTLS and its own identity –
and is still rejected. Error two: the rule
postgres-von-notes-api only knows notes-api. A
separate rule for statistik, without touching the existing
one:
cat > statistik-authz.yaml <<'EOF'
apiVersion: policy.linkerd.io/v1alpha1
kind: MeshTLSAuthentication
metadata: {name: statistik, namespace: pruefung}
spec:
identityRefs: [{kind: ServiceAccount, name: statistik}]
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: postgres-von-statistik, namespace: pruefung}
spec:
targetRef: {group: policy.linkerd.io, kind: Server, name: postgres}
requiredAuthenticationRefs:
- {group: policy.linkerd.io, kind: MeshTLSAuthentication, name: statistik}
EOF
kubectl apply -f statistik-authz.yaml
sleep 25
kubectl -n pruefung logs deploy/statistik -c statistik --tail=2
linkerd diagnostics proxy-metrics -n pruefung po/postgres-0 \
| grep '^inbound_tcp_authz_allow_total' \
| grep -oE '(authz_name|client_id)="[^"]*"|\} [0-9]+' | paste - - -Expected (abridged):
meshtlsauthentication.policy.linkerd.io/statistik created
authorizationpolicy.policy.linkerd.io/postgres-von-statistik created
21:40:46 0
21:40:56 0
authz_name="postgres-von-statistik" client_id="statistik.pruefung.serviceac…" } 3
authz_name="postgres-von-notes-api" client_id="notes-api.pruefung.serviceac…" } 8
Whoever only removes the annotation or only creates the rule gets no
points – only both together bring statistik back. Opening
Postgres in pruefung to the Pod network via
NetworkAuthentication would be the forbidden shortcut.
Solution 15
helm --kube-context west list -n linkerd -o json | jq -r \
'.[] | select(.name=="linkerd-control-plane") | .chart + " " + .revision' > 15.txt
helm --kube-context west get values linkerd-control-plane -n linkerd --all \
| grep -E '^(controllerReplicas|webhookFailurePolicy):' >> 15.txt
kubectl --context west -n linkerd get deploy linkerd-destination \
-o jsonpath='{.status.readyReplicas}{"\n"}' >> 15.txt
cat 15.txtExpected:
linkerd-control-plane-2026.11.2 5
controllerReplicas: 2
webhookFailurePolicy: Fail
2
Revision 5 after installation, anti-affinity fix, upgrade, rollback
and another upgrade (Days 14 and 15). Without --all,
webhookFailurePolicy would only appear because it is set in
values-ha.yaml – --all shows the effective
value, no matter where it comes from.
Solution 16
kubectl -n pruefung get pod -l app=web -o jsonpath='{.items[0].status.qosClass}{"\n"}'
kubectl -n pruefung patch deploy web -p '{"spec":{"template":{"metadata":{
"annotations":{"config.linkerd.io/proxy-cpu-request":"50m",
"config.linkerd.io/proxy-memory-request":"32Mi",
"config.linkerd.io/proxy-memory-limit":"128Mi"}}}}}'
kubectl -n pruefung rollout status deploy/web --timeout=120s
kubectl -n pruefung get pod -l app=web -o \
jsonpath='{.items[0].spec.initContainers[?(@.name=="linkerd-proxy")].resources}'
echo
kubectl -n pruefung get pod -l app=web -o jsonpath='{.items[0].status.qosClass}{"\n"}'Expected:
BestEffort
deployment.apps/web patched
deployment "web" successfully rolled out
{"limits":{"memory":"128Mi"},"requests":{"cpu":"50m","memory":"32Mi"}}
Burstable
Without the annotation, the proxy has no resources, and the whole Pod
is BestEffort – the first to lose out on full workers (Day
17). The proxy is a native sidecar, which is why it is listed under
initContainers. You deliberately did not set a CPU limit:
it would also fix the number of the proxy’s worker threads (Day 17,
Drill 8).
Scoring
Add up the weights of the tasks that are solved completely and
provably – there are no partial points. Whoever bypasses a requirement
gets zero for the task: turning off default-deny or
accessPolicy: all-unauthenticated, opening the Pod network
for web or postgres, deleting the broken Pod,
trafficPolicy: Allow, linkerd upgrade or
helm upgrade.
| Result | Assessment |
|---|---|
| 85–100 % | confident in operations – from the first proxy to the second cluster |
| 70–84 % | the craft is solid; repeat weak spots specifically |
| below 70 % | repeat the days for the tasks you missed and start again |
| Tasks | Area | Days |
|---|---|---|
| 1–3 | installation, injection, mTLS (18 %) | 1–3 |
| 4–5 | observing and certificates (11 %) | 4, 5 |
| 6, 14 | authorization and troubleshooting (15 %) | 6, 16 |
| 7–10 | retries, circuit breaking, rate limit, canary (26 %) | 7–9 |
| 11–12 | ingress and egress (13 %) | 10, 11 |
| 13, 15 | multicluster and Helm (12 %) | 12–15 |
| 16 | proxy resources (5 %) | 17 |
Not covered: rotating the trust anchor (Day 5), federated services and cluster failure (Day 13), upgrade and rollback (Day 15) – they change the whole cluster and do not fit into an exam next to running services. Go through these days once more to finish.
Cleanup
First the export and the link, then the namespaces. Since Linkerd
2.18 there is no multicluster unlink; you remove the parts
one by one as on Day 13:
kubectl --context west -n pruefung label service helper mirror.linkerd.io/exported-
kubectl --context ost get svc -A -l mirror.linkerd.io/mirrored-service=true
kubectl --context ost -n linkerd-multicluster delete link west
kubectl --context ost delete -l mirror.linkerd.io/cluster-name=west -f mc-ost.yaml
kubectl --context ost -n linkerd-multicluster delete secret cluster-credentials-west
kubectl --context ost -n linkerd delete secret cluster-credentials-west
rm link-west.yamlExpected (abridged):
service/helper unlabeled
No resources found
link.multicluster.linkerd.io "west" deleted from linkerd-multicluster namespace
deployment.apps "controller-west" deleted from linkerd-multicluster namespace
service "probe-west" deleted from linkerd-multicluster namespace
...
secret "cluster-credentials-west" deleted from linkerd namespace
link-west.yaml contains credentials for the API of
west – it does not belong in a working folder. Then the
namespaces:
kubectl get endpointslices -A | grep cilium-gateway
kubectl --context west delete namespace pruefung
kubectl delete namespace pruefung --timeout=180s
kubectl get endpointslices -A | grep cilium-gateway | grep -c pruefungExpected (abridged):
pruefung cilium-gateway-helper IPv4 9999 192.192.192.192 13m
pruefung cilium-gateway-pruefung IPv4 9999 192.192.192.192 6m20s
namespace "pruefung" deleted
namespace "pruefung" deleted
0
Together with the namespace, routes, Gateway, policies,
EgressNetwork, PVC and both of Cilium’s EndpointSlices disappear –
including cilium-gateway-helper, which would have been left
behind when deleting a single route (Day 9, Drill 7). The files in
~/linkerd/pruefung remain as templates.
Checkpoint
kubectl get ns | grep -c pruefung
kubectl --context west get ns | grep -c pruefung
kubectl get pv | grep -c pruefung
kubectl get links -AExpected: 0 three times, then
No resources found – no link active, as after Day 13.
linkerd --context ost multicluster check 2>&1 | tail -1
linkerd --context ost multicluster gatewaysExpected:
Status check results are √
CLUSTER ALIVE NUM_SVC LATENCY
Without a link, the check no longer verifies trust anchors – not even the false alarm from Solution 13, if you had it.
kubectl -n notes get pods
kubectl -n notes logs deploy/last -c last --since=55s | awk '{print $2, $3}' \
| sort | uniq -c
linkerd check 2>&1 | tail -1Expected (numbers vary): all eight Pods
2/2 Running, then
85 /api/notes 200
31 /flaky 200
12 /flaky 500
Status check results are √
notes did not notice anything of the entire exam. With
that, you have completed the drills.