Skip to content
Hands-on IT IT Drills

Day 5: Grafana

Goal: You find your way around Grafana, use Explore and the bundled dashboards, and build a first dashboard of your own for notes-api. · Duration: approx. 60 min · Prerequisite: Day 4

Briefing

  • Grafana draws what others store. It has no data of its own but queries data sources: Prometheus for metrics, from Day 7 Loki for logs, from Day 10 Tempo for traces.
  • Explore is the place to try things out: one query, one graph, nothing saved.
  • A dashboard consists of panels. Every panel has one or more queries and a visualization – time series, single stat, table.
  • kube-prometheus-stack brings Grafana fully wired up: data sources and a good two dozen dashboards for Kubernetes are already there.
 Browser ──► :30300 Grafana ──► data source Prometheus ──► kps-prometheus:9090
                         └────► data source Alertmanager ─► kps-alertmanager:9093

Drill 1: Logging In

Grafana answers whether it’s healthy even without logging in:

curl -s localhost:30300/api/health

Expected:

{
  "database": "ok",
  "version": "13.3.0",
  "commit": "6f8ce3c693b493043ebcd4787566d2b6b118b0f0"
}

Open http://<VM-IP>:30300 in the browser and log in: user admin, password lab-grafana-pw-123 – the password from kps.yaml. Grafana doesn’t ask for a new password because the chart set it.

Save your progress? Sign up for free

Drill 2: The Data Sources

Everything the UI shows is also available through the HTTP API. With user and password via -u:

curl -s -u admin:lab-grafana-pw-123 localhost:30300/api/datasources \
  | jq -c '.[] | [.name, .uid, .url]'

Expected:

["Alertmanager","alertmanager","http://kps-alertmanager.monitoring:9093/"]
["Prometheus","prometheus","http://kps-prometheus.monitoring:9090/"]
  • Grafana talks to Prometheus through the Service in the cluster, not through the NodePort.
  • The uid is the fixed name of a data source. Dashboards refer to it – that becomes important on Day 6.
  • A helper container in the Grafana pod created the data sources. It reads ConfigMaps with a specific label and passes them on to Grafana. That’s why the pod has three containers.

In the UI you find them under Connections → Data sources.

Save your progress? Sign up for free

Drill 3: Explore

Click Explore in the menu on the left. At the top, the data source is Prometheus.

  1. Switch the editor on the right from Builder to Code.
  2. Enter the recording rule from Day 3: sum by (method) (job:notes_http_requests:rate5m)
  3. Click Run query (or press Shift+Enter).

Expected: two lines, GET just above 4 and POST at 1, below them a table with the same values.

  • At the top right you set the time range. Choose Last 30 minutes: the dips and spikes from Day 4 are visible – errors, pods at zero, delay.
  • Under Options → Legend, {{method}} turns {method="GET"} into a plain GET.
  • Explore doesn’t remember anything. What you want to keep goes into a dashboard.

Save your progress? Sign up for free

Drill 4: Bundled Dashboards

curl -s -u admin:lab-grafana-pw-123 \
  'localhost:30300/api/search?type=dash-db&query=Kubernetes' | jq -r '.[].title'

Expected (shortened):

Kubernetes / API server
Kubernetes / Compute Resources /  Multi-Cluster
Kubernetes / Compute Resources / Cluster
Kubernetes / Compute Resources / Namespace (Pods)
...
Kubernetes / Persistent Volumes

In the UI, open Dashboards and there Kubernetes / Compute Resources / Namespace (Pods). At the top, choose the value app for namespace.

Expected: panels for CPU and memory, split by pod. postgres-0 needs the most CPU, the two notes-api pods share the rest.

Compare with what Kubernetes itself reports:

kubectl -n app top pods

Expected (the numbers differ):

NAME                         CPU(cores)   MEMORY(bytes)
last-6fdd4b976d-47f97        14m          3Mi
notes-api-57f4f47769-frk2p   13m          7Mi
notes-api-57f4f47769-s66pc   11m          7Mi
postgres-0                   33m          64Mi

The same orders of magnitude. kubectl top asks the metrics-server of k3s, the dashboard asks Prometheus for the cAdvisor metrics. Both come from the kubelet.

Save your progress? Sign up for free

Drill 5: A Dashboard of Your Own

  1. Dashboards → New → New dashboard, then Add visualization.
  2. Choose the data source Prometheus.
  3. Switch the query editor to Code and enter: sum by (method) (job:notes_http_requests:rate5m)
  4. Set Legend to Custom and enter {{method}}.
  5. In the options pane on the right: Title Anfragen pro Sekunde, under Standard options → Unit requests/sec (rps).
  6. At the top, Save dashboard, title notes-api, save.

A second panel for the response time:

  1. Add → Visualization.
  2. Query: job:notes_http_request_duration_seconds:p95_5m, legend p95.
  3. Title Antwortzeit p95, Unit seconds (s).
  4. Under Thresholds, change the red value from 80 to 0.25 and choose As lines for Show thresholds.
  5. Save.

Expected: two panels. The line in the second is at a few milliseconds; the red line at 250 ms is the same limit as in the NotesApiSlow alert.

Save your progress? Sign up for free

Drill 6: The Dashboard via the API

curl -s -u admin:lab-grafana-pw-123 \
  'localhost:30300/api/search?type=dash-db&query=notes-api' | jq '.[0] | {title, uid}'

Expected (the uid differs):

{
  "title": "notes-api",
  "uid": "al6g5q"
}

The uid is the fixed name of the dashboard; it also appears in its address in the browser: http://<VM-IP>:30300/d/al6g5q/notes-api.

You see the content in the UI: in the dashboard at the top, Export → Export as JSON. A dashboard is nothing more than a JSON document with panels, queries and settings. If you click it together, it only lives in Grafana’s database. Day 6 shows how it gets into Git.

Save your progress? Sign up for free

Checkpoint

curl -s -u admin:lab-grafana-pw-123 \
  'localhost:30300/api/search?type=dash-db&query=notes-api' | jq -r '.[].title'

Expected:

notes-api
curl -s -u admin:lab-grafana-pw-123 \
  'localhost:30300/api/search?type=dash-db' | jq length

Expected:

26

25 bundled ones and your own.

Challenge

Add a panel Fehlerquote to the dashboard that shows the share of 5xx responses in percent. Trigger a fault with NOTES_ERROR_RATE=0.1 and watch.

Hint: The recording rule has the label code. Percent (0.0-1.0) fits as the unit.

Solution

Query in the new panel:

sum(job:notes_http_requests:rate5m{code=~"5.."})
  / sum(job:notes_http_requests:rate5m)

Unit Percent (0.0-1.0), title Fehlerquote, save. Then:

kubectl -n app set env deploy/notes-api NOTES_ERROR_RATE=0.1

Expected: within five minutes the line rises to about 9 percent – not 10, because /healthz and /metrics never fail and count in the denominator. Without a fault, the panel shows No data: if there is no 5xx response, there is no time series with code=~"5.." either.

Take it back:

kubectl -n app set env deploy/notes-api NOTES_ERROR_RATE-

Save your progress? Sign up for free

Enterprise Note

  • Login via SSO: In production, nobody logs in as admin. Grafana speaks OIDC, SAML and LDAP and maps groups to roles: Viewer, Editor, Admin.
  • Folders and permissions: Dashboards live in folders, permissions are attached to the folder. Every team gets its own.
  • “No data” is a statement: An empty panel can mean “no errors” or “no data”. With or vector(0) at the end of the query, the panel shows 0 instead of nothing – but also when notes-api isn’t being scraped at all.