Skip to content
Hands-on IT IT Drills

Day 19: Network Metrics

Goal: You enable Hubble and Cilium metrics, collect them with Prometheus, find drops, DNS errors and HTTP 5xx via PromQL and alert on too many drops. · Duration: approx. 75 min · Prerequisite: Day 12

Briefing

  • CKNE: Observability – “Analyzing Network Health Using Metrics”.
  • Three sources: Cilium agent (:9962, datapath, drops, BPF maps), operator (:9963, IPAM), Hubble (:9965, derived from flows: DNS, drops, TCP, HTTP).
  • Hubble metrics for DNS and HTTP need L7 visibility – the traffic has to pass through the DNS proxy or Envoy, i.e. a CNP with a dns or http rule.
  • A slim kube-prometheus-stack: Prometheus, operator, Grafana, kube-state-metrics – all on cp1 (11 GB), the workers (3 GB) stay free.
 cilium-agent :9962 ─┐
 hubble       :9965 ─┼─ ServiceMonitor ─> Prometheus (cp1) ─> Grafana (cp1)
 operator     :9963 ─┘                        └─ PrometheusRule ─> ALERTS

Drill 1: A slim kube-prometheus-stack

mkdir -p ~/ckne/tag19 && cd ~/ckne/tag19
helm repo add prometheus-community \
  https://prometheus-community.github.io/helm-charts
helm repo update prometheus-community
cat > kps.yaml <<'EOF'
alertmanager:
  enabled: false
defaultRules:
  create: false
nodeExporter:
  enabled: false
kubeProxy:
  enabled: false
kubeEtcd:
  enabled: false
kubeControllerManager:
  enabled: false
kubeScheduler:
  enabled: false
prometheusOperator:
  nodeSelector: {kubernetes.io/hostname: cp1}
  tolerations: [{key: node-role.kubernetes.io/control-plane, effect: NoSchedule}]
kube-state-metrics:
  nodeSelector: {kubernetes.io/hostname: cp1}
  tolerations: [{key: node-role.kubernetes.io/control-plane, effect: NoSchedule}]
grafana:
  adminPassword: ckne-lab
  nodeSelector: {kubernetes.io/hostname: cp1}
  tolerations: [{key: node-role.kubernetes.io/control-plane, effect: NoSchedule}]
prometheus:
  prometheusSpec:
    retention: 12h
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false
    ruleSelectorNilUsesHelmValues: false
    resources:
      requests: {cpu: 200m, memory: 512Mi}
      limits: {memory: 1536Mi}
    nodeSelector: {kubernetes.io/hostname: cp1}
    tolerations: [{key: node-role.kubernetes.io/control-plane, effect: NoSchedule}]
EOF
helm install kps prometheus-community/kube-prometheus-stack --version 93.4.1 \
  -n tag19-mon --create-namespace -f kps.yaml
k -n tag19-mon get pods -o wide

Expected (abridged, after 1–2 min):

STATUS: deployed
NAME                                                 READY   STATUS    ...   NODE
kps-grafana-579658887c-vhfwh                         3/3     Running   ...   cp1
kps-kube-prometheus-stack-operator-7459df4df-td655   1/1     Running   ...   cp1
kps-kube-state-metrics-cdf6565bc-h5924               1/1     Running   ...   cp1
prometheus-kps-kube-prometheus-stack-prometheus-0    2/2     Running   ...   cp1

Without …NilUsesHelmValues: false, Prometheus only reads ServiceMonitors and rules with the label release: kps – Cilium’s and your own would be ignored. The disabled targets (kube-proxy, etcd, scheduler) either don’t exist here or only listen on localhost; as down they would just make noise.

Save your progress? Sign up for free

Drill 2: Enable Cilium and Hubble metrics

The httpV2 option contains commas – hence a values file instead of --set. The backslash at the end of the line continues the string in YAML without a space:

cat > cilium-metrics.yaml <<'EOF'
prometheus:
  enabled: true
  serviceMonitor:
    enabled: true
operator:
  prometheus:
    enabled: true
    serviceMonitor:
      enabled: true
  dashboards:
    enabled: true
dashboards:
  enabled: true
hubble:
  metrics:
    enabled:
    - dns
    - drop
    - tcp
    - flow
    - port-distribution
    - "httpV2:labelsContext=source_namespace,source_workload,\
      destination_namespace,destination_workload,traffic_direction"
    serviceMonitor:
      enabled: true
    dashboards:
      enabled: true
EOF
helm get values cilium -n kube-system -o yaml > values-vorher.yaml
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 \
  --reuse-values -f cilium-metrics.yaml
k -n kube-system rollout restart ds/cilium deploy/cilium-operator
k -n kube-system rollout status ds/cilium
cilium status --wait
cilium config view | grep -E 'hubble-metrics |prometheus-serve-addr'
k -n kube-system get servicemonitors
k get cm -A -l grafana_dashboard=1 | grep kube-system

Expected (abridged):

hubble-metrics            dns drop tcp flow port-distribution httpV2:labelsCont...
operator-prometheus-serve-addr                    :9963
prometheus-serve-addr                             :9962
NAME              AGE
cilium-agent      60s
cilium-operator   60s
hubble            60s
kube-system   cilium-dashboard                     1      62s
kube-system   cilium-operator-dashboard            1      62s
kube-system   hubble-dashboard                     1      62s
kube-system   hubble-dns-namespace                 1      62s
kube-system   hubble-l7-http-metrics-by-workload   1      62s
kube-system   hubble-network-overview-namespace    1      62s

--reuse-values -f merges the file into the existing values. The Cilium chart only creates the ServiceMonitors if the Prometheus Operator CRDs already exist – hence Drill 1 first. The dashboards end up as ConfigMaps with the label grafana_dashboard=1; Grafana’s sidecar collects them from all namespaces.

Save your progress? Sign up for free

Drill 3: Metrics at the source

The agent runs in the host network – the ports are open directly on every node:

curl -s localhost:9962/metrics | grep -c '^cilium_'
curl -s localhost:9962/metrics | grep '^cilium_drop_count_total'
curl -s localhost:9965/metrics | grep '^hubble_drop_total'
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg metrics list \
  -p drop_count

Expected (abridged):

1367
cilium_drop_count_total{direction="EGRESS",reason="Unsupported L3 protocol"} 56
cilium_drop_count_total{direction="INGRESS",reason="VLAN traffic disallowed by VLAN
filter"} 45331
hubble_drop_total{protocol="IGMP",reason="VLAN_FILTERED"} 8
hubble_drop_total{protocol="UDP",reason="VLAN_FILTERED"} 573
Metric                    Labels                                              Value
cilium_drop_count_total   direction=EGRESS reason=Policy denied               157.000000
cilium_drop_count_total   direction=INGRESS reason=VLAN traffic disall... 43640.000000

The biggest item is an old acquaintance from Day 0: the home network router sends VLAN-tagged broadcasts, and Cilium drops them at eth0. Not a cluster fault, but this is exactly what a “noisy” drop reason looks like. The agent counters come from an eBPF map and survive agent restarts; Hubble only starts counting at startup.

Save your progress? Sign up for free

Drill 4: Load with errors and L7 visibility

A client calls a good and a broken URL in a loop and looks up a name that doesn’t exist. With /echo?code=503, agnhost returns any status code:

k create ns tag19
k config set-context --current --namespace=tag19
k create deployment echo --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 \
  --port=8080 -- /agnhost netexec --http-port=8080
k expose deployment echo --port=80 --target-port=8080
cat > client.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: client
  labels:
    app: client
spec:
  containers:
  - name: netshoot
    image: nicolaka/netshoot:v0.16
    command: ["sh", "-c"]
    args:
    - |
      while true; do
        curl -s -o /dev/null echo/hostname
        curl -s -o /dev/null "echo/echo?msg=kaputt&code=503"
        nslookup gibtsnicht.tag19 >/dev/null
        sleep 2
      done
EOF
k apply -f client.yaml
cat > l7.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: l7-sicht
spec:
  endpointSelector:
    matchLabels:
      app: client
  egress:
  - toEndpoints:
    - matchLabels:
        k8s:io.kubernetes.pod.namespace: kube-system
        k8s-app: kube-dns
    toPorts:
    - ports:
      - port: "53"
        protocol: ANY
      rules:
        dns:
        - matchPattern: "*"
  - toEndpoints:
    - matchLabels:
        app: echo
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      rules:
        http:
        - {}
EOF
k apply -f l7.yaml
k wait --for=condition=Ready pod --all

Expected: pod/client condition met, pod/echo-… condition met.

http: [{}] allows every HTTP request but routes it through Envoy – only then does Hubble know method, status and duration. The same applies to dns and the DNS proxy. Warning: the policy turns on default deny in egress for client.

Save your progress? Sign up for free

Drill 5: PromQL for drops, DNS and HTTP

promtool lives in the Prometheus container – no port-forward needed. The small shell function pq saves typing:

pq() {
  k -n tag19-mon exec prometheus-kps-kube-prometheus-stack-prometheus-0 \
    -c prometheus -- promtool query instant http://localhost:9090 "$1"
}
pq 'count by (job) (up)'
pq 'sum by (reason, node) (rate(hubble_drop_total[5m]))'
pq 'sum by (rcode) (rate(hubble_dns_responses_total[1m]))'
pq 'sum by (status)
  (rate(hubble_http_requests_total{destination_namespace="tag19"}[1m]))'

Expected (abridged):

{job="cilium-agent"} => 3 @[1795046465.991]
{job="hubble-metrics"} => 3 @[1795046465.991]
{job="cilium-operator"} => 1 @[1795046465.991]
{job="coredns"} => 2 @[1795046465.991]
{node="worker1", reason="UNSUPPORTED_L3_PROTOCOL"} => 0.0588 @[1795046540.075]
{node="worker1", reason="VLAN_FILTERED"} => 1.8449 @[1795046540.075]
{node="cp1", reason="VLAN_FILTERED"} => 1.8351 @[1795046540.075]
{node="worker2", reason="VLAN_FILTERED"} => 1.8493 @[1795046540.075]
{rcode="Non-Existent Domain"} => 3.7845 @[1795046521.537]
{rcode="No Error"} => 1.1644 @[1795046521.537]
{status="200"} => 0.2952 @[1795046522.103]
{status="503"} => 0.2952 @[1795046522.103]

VLAN drops spread evenly across all nodes at the same rate – that points to a source outside the cluster. UNSUPPORTED_L3_PROTOCOL are frames without IPv4/IPv6 (such as LLDP), also harmless. For latency and TCP health:

pq 'histogram_quantile(0.95, sum by (le, destination_workload)
  (rate(hubble_http_request_duration_seconds_bucket{
    destination_namespace="tag19"}[5m])))'
pq 'sum by (flag) (rate(hubble_tcp_flags_total[5m]))'
pq 'topk(3, sum by (port, protocol) (rate(hubble_port_distribution_total[5m])))'

Expected (abridged):

{destination_workload="echo"} => 0.21724817180106548 @[1795046541.286]
{flag="FIN"} => 2.605055091869563 @[1795046542.539]
{flag="RST"} => 0.2063515935481269 @[1795046542.539]
{flag="SYN"} => 1.7882409839229911 @[1795046542.539]
{flag="SYN-ACK"} => 1.4431221986062863 @[1795046542.539]
{port="53", protocol="UDP"} => 4.389030689206084 @[1795046541.845]
{port="8080", protocol="TCP"} => 3.6105215395468577 @[1795046541.845]
{port="10250", protocol="TCP"} => 1.411318715734712 @[1795046541.845]

histogram_quantile interpolates within the bucket boundaries – with few requests the p95 is rough. Many SYN without matching SYN-ACK or a high share of RST point to rejected connections.

Save your progress? Sign up for free

Drill 6: Grafana dashboards

k -n tag19-mon port-forward --address 0.0.0.0 svc/kps-grafana 3000:80 &
curl -s -u admin:ckne-lab 'localhost:3000/api/search?type=dash-db' \
  | jq -r '.[].title' | grep -iE 'cilium|hubble'

Expected:

Cilium Metrics
Cilium Operator
Hubble / DNS Overview (Namespace)
Hubble / Network Overview (Namespace)
Hubble L7 HTTP Metrics by Workload
Hubble Metrics and Monitoring

Open http://172.16.0.126:3000 (admin / ckne-lab) in the browser: “Hubble L7 HTTP Metrics by Workload” shows echo with 50 % 5xx, “Hubble / DNS Overview” the NXDOMAIN flood. Stop with kill %1. Without Helm, you import the same dashboards as JSON under Dashboards → New → Import.

Save your progress? Sign up for free

Drill 7: Alert on drops

cat > alarm.yaml <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: tag19-drops
spec:
  groups:
  - name: tag19-netz
    rules:
    - alert: PolicyDropsHoch
      expr: sum by (node) (rate(hubble_drop_total{reason="POLICY_DENIED"}[1m])) > 0.2
      for: 1m
      labels:
        severity: warning
      annotations:
        summary: "Policy-Drops auf {{ $labels.node }}: {{ $value | humanize }}/s"
EOF
k apply -f alarm.yaml
k run stoerer -l app=client --image=nicolaka/netshoot:v0.16 --command -- \
  sh -c 'while true; do curl -s -m1 http://1.1.1.1; done'
pq 'sum by (node) (rate(hubble_drop_total{reason="POLICY_DENIED"}[1m]))'
pq 'ALERTS'

Repeat pq 'ALERTS' after one minute.

Expected (abridged):

{node="worker2"} => 1.741585 @[1795046665.746]
ALERTS{alertname="PolicyDropsHoch", alertstate="pending", node="worker2", ...} => 1
ALERTS{alertname="PolicyDropsHoch", alertstate="firing", node="worker2", ...} => 1

stoerer carries app=client and thus inherits the egress policy: 1.1.1.1 is not allowed, every SYN is dropped. The alert deliberately filters on POLICY_DENIED – a threshold on all drops would fire constantly here because of the VLAN frames. pending becomes firing after for: 1m; without Alertmanager it stays a state in Prometheus.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Calculate the share of 5xx responses among all HTTP responses in tag19. Target time: 60 s
  2. Show all drop reasons except VLAN with a rate greater than 0. Target time: 45 s
  3. Which Hubble metrics are enabled? Target time: 20 s

Solution

pq 'sum(rate(hubble_http_requests_total{
    destination_namespace="tag19",status=~"5.."}[5m]))
  / sum(rate(hubble_http_requests_total{destination_namespace="tag19"}[5m]))'
pq 'sum by (reason) (rate(hubble_drop_total{reason!="VLAN_FILTERED"}[5m])) > 0'
cilium config view | grep 'hubble-metrics '

Expected (abridged):

{} => 0.5 @[1795046744.4]
{reason="UNSUPPORTED_L3_PROTOCOL"} => 0.16801044313429858 @[1795046745.059]
{reason="POLICY_DENIED"} => 1.2121667697916683 @[1795046745.059]
hubble-metrics            dns drop tcp flow port-distribution httpV2:labelsContext=...

Save your progress? Sign up for free

Checkpoint

In a new shell, first recreate the pq function from Drill 5.

pq 'count by (job) (up{job=~"cilium.*|hubble.*"})'

Expected: cilium-agent 3, hubble-metrics 3, cilium-operator 1.

pq 'sum by (status)
  (rate(hubble_http_requests_total{destination_namespace="tag19"}[1m]))'

Expected: two lines, status="200" and status="503", roughly equal.

k get prometheusrule tag19-drops

Expected: the rule exists; pq 'ALERTS' shows PolicyDropsHoch firing as long as stoerer is running.

helm list -n tag19-mon

Expected: release kps deployed. The Prometheus stack in tag19-mon and the Cilium metric values (prometheus, operator.prometheus, hubble.metrics, dashboards) stay in place after the cleanup for the practice exam (Appendix B, Task 17) – everything runs on cp1 and uses about 800 MiB there.

Challenge

The NXDOMAIN rate is a multiple of the number of nslookup calls. Use Hubble to find out why, and name two remedies.

Hint: Look at the individual DNS requests from client and its /etc/resolv.conf.

Solution

hubble observe -P --port-forward-port 0 --pod tag19/client --protocol dns --last 12
k exec client -- cat /etc/resolv.conf

Expected (abridged):

... dns-request proxy FORWARDED (DNS Query gibtsnicht.tag19.tag19.svc.cluster.local. A)
... dns-request proxy FORWARDED (DNS Query gibtsnicht.tag19.svc.cluster.local. A)
... dns-request proxy FORWARDED (DNS Query gibtsnicht.tag19.cluster.local. A)
... dns-request proxy FORWARDED (DNS Query gibtsnicht.tag19.localdomain. A)
... dns-request proxy FORWARDED (DNS Query gibtsnicht.tag19. A)
search tag19.svc.cluster.local svc.cluster.local cluster.local localdomain
nameserver 10.96.0.10
options ndots:5

gibtsnicht.tag19 has fewer than five dots (ndots:5), so the resolver appends all four search domains one after another and finally queries the name itself – five NXDOMAIN per call. Remedies: fully qualified names with a trailing dot (gibtsnicht.tag19.) or dnsConfig.options with a smaller ndots in the Pod (Day 8).

Save your progress? Sign up for free

Cleanup

k delete namespace tag19
k config set-context --current --namespace=default

The Prometheus stack and metric values stay for Appendix B (see Checkpoint). Full teardown afterwards, in the directory ~/ckne/tag19:

helm uninstall kps -n tag19-mon
k delete namespace tag19-mon
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 --reset-values \
  -f values-vorher.yaml
k -n kube-system rollout restart ds/cilium deploy/cilium-operator

Exam Tip

  • No metric there? Check the chain: cilium config view | grep hubble-metrics → curl localhost:9965/metrics on the node → up{job="hubble-metrics"} in Prometheus. DNS/HTTP metrics are missing without an L7 policy.
  • Drop reasons are terse in hubble_drop_total (POLICY_DENIED) and spelled out in cilium_drop_count_total (Policy denied).
  • Docs: docs.cilium.io “Monitoring & Metrics” (section “Hubble Exported Metrics”, context options) and “Running Prometheus & Grafana”.

Enterprise Note

  • Limit cardinality: labelsContext with Pod names or IPs multiplies time series; workload and namespace are almost always enough.
  • Prometheus with persistence and remote write (Thanos, Mimir) for long retention; alerts via Alertmanager to the on-call team.
  • Tune alert thresholds to known background noise (here VLAN), otherwise the team becomes desensitized.
  • hubble.metrics.dynamic.enabled lets you change metrics without an agent restart.
  • Protect the Hubble metrics port with TLS/mTLS (hubble.metrics.tls) when Prometheus scrapes from outside the cluster.