Day 19: Network Metrics
Goal: You enable Hubble and Cilium metrics, collect them with Prometheus, find drops, DNS errors and HTTP 5xx via PromQL and alert on too many drops. · Duration: approx. 75 min · Prerequisite: Day 12
Briefing
- CKNE: Observability – “Analyzing Network Health Using Metrics”.
- Three sources: Cilium agent (:9962, datapath, drops, BPF maps), operator (:9963, IPAM), Hubble (:9965, derived from flows: DNS, drops, TCP, HTTP).
- Hubble metrics for DNS and HTTP need L7 visibility – the traffic has
to pass through the DNS proxy or Envoy, i.e. a CNP with a
dnsorhttprule. - A slim kube-prometheus-stack: Prometheus, operator, Grafana,
kube-state-metrics – all on
cp1(11 GB), the workers (3 GB) stay free.
cilium-agent :9962 ─┐
hubble :9965 ─┼─ ServiceMonitor ─> Prometheus (cp1) ─> Grafana (cp1)
operator :9963 ─┘ └─ PrometheusRule ─> ALERTS
Drill 1: A slim kube-prometheus-stack
mkdir -p ~/ckne/tag19 && cd ~/ckne/tag19
helm repo add prometheus-community \
https://prometheus-community.github.io/helm-charts
helm repo update prometheus-community
cat > kps.yaml <<'EOF'
alertmanager:
enabled: false
defaultRules:
create: false
nodeExporter:
enabled: false
kubeProxy:
enabled: false
kubeEtcd:
enabled: false
kubeControllerManager:
enabled: false
kubeScheduler:
enabled: false
prometheusOperator:
nodeSelector: {kubernetes.io/hostname: cp1}
tolerations: [{key: node-role.kubernetes.io/control-plane, effect: NoSchedule}]
kube-state-metrics:
nodeSelector: {kubernetes.io/hostname: cp1}
tolerations: [{key: node-role.kubernetes.io/control-plane, effect: NoSchedule}]
grafana:
adminPassword: ckne-lab
nodeSelector: {kubernetes.io/hostname: cp1}
tolerations: [{key: node-role.kubernetes.io/control-plane, effect: NoSchedule}]
prometheus:
prometheusSpec:
retention: 12h
serviceMonitorSelectorNilUsesHelmValues: false
podMonitorSelectorNilUsesHelmValues: false
ruleSelectorNilUsesHelmValues: false
resources:
requests: {cpu: 200m, memory: 512Mi}
limits: {memory: 1536Mi}
nodeSelector: {kubernetes.io/hostname: cp1}
tolerations: [{key: node-role.kubernetes.io/control-plane, effect: NoSchedule}]
EOF
helm install kps prometheus-community/kube-prometheus-stack --version 93.4.1 \
-n tag19-mon --create-namespace -f kps.yaml
k -n tag19-mon get pods -o wideExpected (abridged, after 1–2 min):
STATUS: deployed
NAME READY STATUS ... NODE
kps-grafana-579658887c-vhfwh 3/3 Running ... cp1
kps-kube-prometheus-stack-operator-7459df4df-td655 1/1 Running ... cp1
kps-kube-state-metrics-cdf6565bc-h5924 1/1 Running ... cp1
prometheus-kps-kube-prometheus-stack-prometheus-0 2/2 Running ... cp1
Without …NilUsesHelmValues: false, Prometheus only reads
ServiceMonitors and rules with the label release: kps –
Cilium’s and your own would be ignored. The disabled targets
(kube-proxy, etcd, scheduler) either don’t exist here or only listen on
localhost; as down they would just make noise.
Save your progress? Sign up for free
Drill 2: Enable Cilium and Hubble metrics
The httpV2 option contains commas – hence a values file instead of
--set. The backslash at the end of the line continues the
string in YAML without a space:
cat > cilium-metrics.yaml <<'EOF'
prometheus:
enabled: true
serviceMonitor:
enabled: true
operator:
prometheus:
enabled: true
serviceMonitor:
enabled: true
dashboards:
enabled: true
dashboards:
enabled: true
hubble:
metrics:
enabled:
- dns
- drop
- tcp
- flow
- port-distribution
- "httpV2:labelsContext=source_namespace,source_workload,\
destination_namespace,destination_workload,traffic_direction"
serviceMonitor:
enabled: true
dashboards:
enabled: true
EOF
helm get values cilium -n kube-system -o yaml > values-vorher.yaml
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 \
--reuse-values -f cilium-metrics.yaml
k -n kube-system rollout restart ds/cilium deploy/cilium-operator
k -n kube-system rollout status ds/cilium
cilium status --wait
cilium config view | grep -E 'hubble-metrics |prometheus-serve-addr'
k -n kube-system get servicemonitors
k get cm -A -l grafana_dashboard=1 | grep kube-systemExpected (abridged):
hubble-metrics dns drop tcp flow port-distribution httpV2:labelsCont...
operator-prometheus-serve-addr :9963
prometheus-serve-addr :9962
NAME AGE
cilium-agent 60s
cilium-operator 60s
hubble 60s
kube-system cilium-dashboard 1 62s
kube-system cilium-operator-dashboard 1 62s
kube-system hubble-dashboard 1 62s
kube-system hubble-dns-namespace 1 62s
kube-system hubble-l7-http-metrics-by-workload 1 62s
kube-system hubble-network-overview-namespace 1 62s
--reuse-values -f merges the file into the existing
values. The Cilium chart only creates the ServiceMonitors if the
Prometheus Operator CRDs already exist – hence Drill 1 first. The
dashboards end up as ConfigMaps with the label
grafana_dashboard=1; Grafana’s sidecar collects them from
all namespaces.
Save your progress? Sign up for free
Drill 3: Metrics at the source
The agent runs in the host network – the ports are open directly on every node:
curl -s localhost:9962/metrics | grep -c '^cilium_'
curl -s localhost:9962/metrics | grep '^cilium_drop_count_total'
curl -s localhost:9965/metrics | grep '^hubble_drop_total'
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg metrics list \
-p drop_countExpected (abridged):
1367
cilium_drop_count_total{direction="EGRESS",reason="Unsupported L3 protocol"} 56
cilium_drop_count_total{direction="INGRESS",reason="VLAN traffic disallowed by VLAN
filter"} 45331
hubble_drop_total{protocol="IGMP",reason="VLAN_FILTERED"} 8
hubble_drop_total{protocol="UDP",reason="VLAN_FILTERED"} 573
Metric Labels Value
cilium_drop_count_total direction=EGRESS reason=Policy denied 157.000000
cilium_drop_count_total direction=INGRESS reason=VLAN traffic disall... 43640.000000
The biggest item is an old acquaintance from Day 0: the home network
router sends VLAN-tagged broadcasts, and Cilium drops them at
eth0. Not a cluster fault, but this is exactly what a
“noisy” drop reason looks like. The agent counters come from an eBPF map
and survive agent restarts; Hubble only starts counting at startup.
Save your progress? Sign up for free
Drill 4: Load with errors and L7 visibility
A client calls a good and a broken URL in a loop and looks up a name
that doesn’t exist. With /echo?code=503, agnhost returns
any status code:
k create ns tag19
k config set-context --current --namespace=tag19
k create deployment echo --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 \
--port=8080 -- /agnhost netexec --http-port=8080
k expose deployment echo --port=80 --target-port=8080
cat > client.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: client
labels:
app: client
spec:
containers:
- name: netshoot
image: nicolaka/netshoot:v0.16
command: ["sh", "-c"]
args:
- |
while true; do
curl -s -o /dev/null echo/hostname
curl -s -o /dev/null "echo/echo?msg=kaputt&code=503"
nslookup gibtsnicht.tag19 >/dev/null
sleep 2
done
EOF
k apply -f client.yaml
cat > l7.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: l7-sicht
spec:
endpointSelector:
matchLabels:
app: client
egress:
- toEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: kube-system
k8s-app: kube-dns
toPorts:
- ports:
- port: "53"
protocol: ANY
rules:
dns:
- matchPattern: "*"
- toEndpoints:
- matchLabels:
app: echo
toPorts:
- ports:
- port: "8080"
protocol: TCP
rules:
http:
- {}
EOF
k apply -f l7.yaml
k wait --for=condition=Ready pod --allExpected: pod/client condition met,
pod/echo-… condition met.
http: [{}] allows every HTTP request but routes it
through Envoy – only then does Hubble know method, status and duration.
The same applies to dns and the DNS proxy. Warning: the
policy turns on default deny in egress for client.
Save your progress? Sign up for free
Drill 5: PromQL for drops, DNS and HTTP
promtool lives in the Prometheus container – no
port-forward needed. The small shell function pq saves
typing:
pq() {
k -n tag19-mon exec prometheus-kps-kube-prometheus-stack-prometheus-0 \
-c prometheus -- promtool query instant http://localhost:9090 "$1"
}
pq 'count by (job) (up)'
pq 'sum by (reason, node) (rate(hubble_drop_total[5m]))'
pq 'sum by (rcode) (rate(hubble_dns_responses_total[1m]))'
pq 'sum by (status)
(rate(hubble_http_requests_total{destination_namespace="tag19"}[1m]))'Expected (abridged):
{job="cilium-agent"} => 3 @[1795046465.991]
{job="hubble-metrics"} => 3 @[1795046465.991]
{job="cilium-operator"} => 1 @[1795046465.991]
{job="coredns"} => 2 @[1795046465.991]
{node="worker1", reason="UNSUPPORTED_L3_PROTOCOL"} => 0.0588 @[1795046540.075]
{node="worker1", reason="VLAN_FILTERED"} => 1.8449 @[1795046540.075]
{node="cp1", reason="VLAN_FILTERED"} => 1.8351 @[1795046540.075]
{node="worker2", reason="VLAN_FILTERED"} => 1.8493 @[1795046540.075]
{rcode="Non-Existent Domain"} => 3.7845 @[1795046521.537]
{rcode="No Error"} => 1.1644 @[1795046521.537]
{status="200"} => 0.2952 @[1795046522.103]
{status="503"} => 0.2952 @[1795046522.103]
VLAN drops spread evenly across all nodes at the same rate – that
points to a source outside the cluster.
UNSUPPORTED_L3_PROTOCOL are frames without IPv4/IPv6 (such
as LLDP), also harmless. For latency and TCP health:
pq 'histogram_quantile(0.95, sum by (le, destination_workload)
(rate(hubble_http_request_duration_seconds_bucket{
destination_namespace="tag19"}[5m])))'
pq 'sum by (flag) (rate(hubble_tcp_flags_total[5m]))'
pq 'topk(3, sum by (port, protocol) (rate(hubble_port_distribution_total[5m])))'Expected (abridged):
{destination_workload="echo"} => 0.21724817180106548 @[1795046541.286]
{flag="FIN"} => 2.605055091869563 @[1795046542.539]
{flag="RST"} => 0.2063515935481269 @[1795046542.539]
{flag="SYN"} => 1.7882409839229911 @[1795046542.539]
{flag="SYN-ACK"} => 1.4431221986062863 @[1795046542.539]
{port="53", protocol="UDP"} => 4.389030689206084 @[1795046541.845]
{port="8080", protocol="TCP"} => 3.6105215395468577 @[1795046541.845]
{port="10250", protocol="TCP"} => 1.411318715734712 @[1795046541.845]
histogram_quantile interpolates within the bucket
boundaries – with few requests the p95 is rough. Many SYN
without matching SYN-ACK or a high share of
RST point to rejected connections.
Save your progress? Sign up for free
Drill 6: Grafana dashboards
k -n tag19-mon port-forward --address 0.0.0.0 svc/kps-grafana 3000:80 &
curl -s -u admin:ckne-lab 'localhost:3000/api/search?type=dash-db' \
| jq -r '.[].title' | grep -iE 'cilium|hubble'Expected:
Cilium Metrics
Cilium Operator
Hubble / DNS Overview (Namespace)
Hubble / Network Overview (Namespace)
Hubble L7 HTTP Metrics by Workload
Hubble Metrics and Monitoring
Open http://172.16.0.126:3000 (admin / ckne-lab) in the
browser: “Hubble L7 HTTP Metrics by Workload” shows echo
with 50 % 5xx, “Hubble / DNS Overview” the NXDOMAIN flood. Stop with
kill %1. Without Helm, you import the same dashboards as
JSON under Dashboards → New → Import.
Save your progress? Sign up for free
Drill 7: Alert on drops
cat > alarm.yaml <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: tag19-drops
spec:
groups:
- name: tag19-netz
rules:
- alert: PolicyDropsHoch
expr: sum by (node) (rate(hubble_drop_total{reason="POLICY_DENIED"}[1m])) > 0.2
for: 1m
labels:
severity: warning
annotations:
summary: "Policy-Drops auf {{ $labels.node }}: {{ $value | humanize }}/s"
EOF
k apply -f alarm.yaml
k run stoerer -l app=client --image=nicolaka/netshoot:v0.16 --command -- \
sh -c 'while true; do curl -s -m1 http://1.1.1.1; done'
pq 'sum by (node) (rate(hubble_drop_total{reason="POLICY_DENIED"}[1m]))'
pq 'ALERTS'Repeat pq 'ALERTS' after one minute.
Expected (abridged):
{node="worker2"} => 1.741585 @[1795046665.746]
ALERTS{alertname="PolicyDropsHoch", alertstate="pending", node="worker2", ...} => 1
ALERTS{alertname="PolicyDropsHoch", alertstate="firing", node="worker2", ...} => 1
stoerer carries app=client and thus
inherits the egress policy: 1.1.1.1 is not allowed, every
SYN is dropped. The alert deliberately filters on
POLICY_DENIED – a threshold on all drops would fire
constantly here because of the VLAN frames. pending becomes
firing after for: 1m; without Alertmanager it
stays a state in Prometheus.
Save your progress? Sign up for free
Exercise: Speed Drills
- Calculate the share of 5xx responses among all HTTP responses in
tag19. Target time: 60 s - Show all drop reasons except VLAN with a rate greater than 0. Target time: 45 s
- Which Hubble metrics are enabled? Target time: 20 s
Solution
pq 'sum(rate(hubble_http_requests_total{
destination_namespace="tag19",status=~"5.."}[5m]))
/ sum(rate(hubble_http_requests_total{destination_namespace="tag19"}[5m]))'
pq 'sum by (reason) (rate(hubble_drop_total{reason!="VLAN_FILTERED"}[5m])) > 0'
cilium config view | grep 'hubble-metrics 'Expected (abridged):
{} => 0.5 @[1795046744.4]
{reason="UNSUPPORTED_L3_PROTOCOL"} => 0.16801044313429858 @[1795046745.059]
{reason="POLICY_DENIED"} => 1.2121667697916683 @[1795046745.059]
hubble-metrics dns drop tcp flow port-distribution httpV2:labelsContext=...
Save your progress? Sign up for free
Checkpoint
In a new shell, first recreate the pq function from
Drill 5.
pq 'count by (job) (up{job=~"cilium.*|hubble.*"})'Expected: cilium-agent 3,
hubble-metrics 3, cilium-operator 1.
pq 'sum by (status)
(rate(hubble_http_requests_total{destination_namespace="tag19"}[1m]))'Expected: two lines, status="200" and
status="503", roughly equal.
k get prometheusrule tag19-dropsExpected: the rule exists; pq 'ALERTS'
shows PolicyDropsHoch firing as long as
stoerer is running.
helm list -n tag19-monExpected: release kps
deployed. The Prometheus stack in tag19-mon
and the Cilium metric values (prometheus,
operator.prometheus, hubble.metrics,
dashboards) stay in place after the cleanup for the
practice exam (Appendix B, Task 17) – everything runs on
cp1 and uses about 800 MiB there.
Challenge
The NXDOMAIN rate is a multiple of the number of
nslookup calls. Use Hubble to find out why, and name two
remedies.
Hint: Look at the individual DNS requests from
client and its /etc/resolv.conf.
Solution
hubble observe -P --port-forward-port 0 --pod tag19/client --protocol dns --last 12
k exec client -- cat /etc/resolv.confExpected (abridged):
... dns-request proxy FORWARDED (DNS Query gibtsnicht.tag19.tag19.svc.cluster.local. A)
... dns-request proxy FORWARDED (DNS Query gibtsnicht.tag19.svc.cluster.local. A)
... dns-request proxy FORWARDED (DNS Query gibtsnicht.tag19.cluster.local. A)
... dns-request proxy FORWARDED (DNS Query gibtsnicht.tag19.localdomain. A)
... dns-request proxy FORWARDED (DNS Query gibtsnicht.tag19. A)
search tag19.svc.cluster.local svc.cluster.local cluster.local localdomain
nameserver 10.96.0.10
options ndots:5
gibtsnicht.tag19 has fewer than five dots
(ndots:5), so the resolver appends all four search domains
one after another and finally queries the name itself – five NXDOMAIN
per call. Remedies: fully qualified names with a trailing dot
(gibtsnicht.tag19.) or dnsConfig.options with
a smaller ndots in the Pod (Day 8).
Save your progress? Sign up for free
Cleanup
k delete namespace tag19
k config set-context --current --namespace=defaultThe Prometheus stack and metric values stay for Appendix B (see
Checkpoint). Full teardown afterwards, in the directory
~/ckne/tag19:
helm uninstall kps -n tag19-mon
k delete namespace tag19-mon
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 --reset-values \
-f values-vorher.yaml
k -n kube-system rollout restart ds/cilium deploy/cilium-operatorExam Tip
- No metric there? Check the chain:
cilium config view | grep hubble-metrics→curl localhost:9965/metricson the node →up{job="hubble-metrics"}in Prometheus. DNS/HTTP metrics are missing without an L7 policy. - Drop reasons are terse in
hubble_drop_total(POLICY_DENIED) and spelled out incilium_drop_count_total(Policy denied). - Docs: docs.cilium.io “Monitoring & Metrics” (section “Hubble Exported Metrics”, context options) and “Running Prometheus & Grafana”.
Enterprise Note
- Limit cardinality:
labelsContextwith Pod names or IPs multiplies time series; workload and namespace are almost always enough. - Prometheus with persistence and remote write (Thanos, Mimir) for long retention; alerts via Alertmanager to the on-call team.
- Tune alert thresholds to known background noise (here VLAN), otherwise the team becomes desensitized.
hubble.metrics.dynamic.enabledlets you change metrics without an agent restart.- Protect the Hubble metrics port with TLS/mTLS
(
hubble.metrics.tls) when Prometheus scrapes from outside the cluster.