Day 6: Dashboards as Code
Goal: A dashboard as a JSON file, put into the cluster as a ConfigMap with Kustomize and loaded by Grafana automatically – with a variable and without clicking. · Duration: approx. 60 min · Prerequisite: Day 5
Briefing
- A clicked-together dashboard lives in Grafana’s database. Nobody sees who changed what, and after a reinstall it’s gone.
- In the Grafana pod runs a sidecar that looks for
ConfigMaps with the label
grafana_dashboard: "1"in all namespaces. When it finds one, it stores its content as a file, and Grafana loads the dashboard. - That way the dashboard lives next to the application in Git, goes through reviews and is rolled out with it. The bundled dashboards come the same way.
- Variables make a dashboard reusable: a drop-down list at the top sets a value that every query uses.
dashboards/notes-api.json ─► kubectl apply -k ─► ConfigMap (label grafana_dashboard)
│ sidecar in the Grafana pod
▼
Grafana loads the dashboard
Drill 1: The Dashboard as a File
mkdir -p ~/obs-lab/dashboards
cd ~/obs-lab/dashboards
cat > notes-api.json <<'EOF'
{
"uid": "notes-api-code",
"title": "notes-api Überblick",
"tags": ["notes-api"],
"time": {"from": "now-30m", "to": "now"},
"refresh": "30s",
"schemaVersion": 41,
"templating": {
"list": [
{
"name": "pod",
"label": "Pod",
"type": "query",
"datasource": {"type": "prometheus", "uid": "prometheus"},
"query": {"query": "label_values(up{job=\"notes-api\"}, pod)", "refId": "pod"},
"refresh": 2,
"multi": true,
"includeAll": true,
"current": {"text": "All", "value": "$__all"}
}
]
},
"panels": [
{
"id": 1, "type": "stat", "title": "Pods bereit",
"gridPos": {"x": 0, "y": 0, "w": 4, "h": 6},
"datasource": {"type": "prometheus", "uid": "prometheus"},
"targets": [{"refId": "A", "expr": "count(up{job=\"notes-api\"} == 1)"}]
},
{
"id": 2, "type": "timeseries", "title": "Anfragen pro Sekunde",
"gridPos": {"x": 4, "y": 0, "w": 10, "h": 6},
"datasource": {"type": "prometheus", "uid": "prometheus"},
"fieldConfig": {"defaults": {"unit": "reqps"}, "overrides": []},
"targets": [{"refId": "A", "legendFormat": "{{method}}",
"expr": "sum by (method) (job:notes_http_requests:rate5m)"}]
},
{
"id": 3, "type": "timeseries", "title": "Antwortzeit p95",
"gridPos": {"x": 14, "y": 0, "w": 10, "h": 6},
"datasource": {"type": "prometheus", "uid": "prometheus"},
"fieldConfig": {"defaults": {"unit": "s"}, "overrides": []},
"targets": [{"refId": "A", "legendFormat": "p95",
"expr": "job:notes_http_request_duration_seconds:p95_5m"}]
},
{
"id": 4, "type": "timeseries", "title": "Notizen pro Sekunde je Pod",
"gridPos": {"x": 0, "y": 6, "w": 24, "h": 7},
"datasource": {"type": "prometheus", "uid": "prometheus"},
"targets": [{"refId": "A", "legendFormat": "{{pod}}",
"expr": "sum by (pod) (rate(notes_created_total{pod=~\"$pod\"}[5m]))"}]
}
]
}
EOF
jq -r '.panels[].title' notes-api.jsonExpected:
Pods bereit
Anfragen pro Sekunde
Antwortzeit p95
Notizen pro Sekunde je Pod
jqreads the file. If it isn’t valid JSON, it reports the error with line and column – before Grafana silently fails on it.uidis the fixed name. It keeps the dashboard’s address the same no matter how often you reload it. The title differs from the one on Day 5, because Grafana rejects two dashboards with the same title in the same folder.- The data source is set as
uid: prometheus– the fixed name from Day 5, Drill 2. gridPossets position and size: 24 columns wide,hin grid rows.- The variable
podgets its values from Prometheus withlabel_values(...).$podin the fourth panel becomes a regular expression likenotes-api-57f4f47769-5x8dq|notes-api-57f4f47769-qj4nb. Hence=~.
Save your progress? Sign up for free
Drill 2: Kustomize Builds the ConfigMap
cat > kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: app
configMapGenerator:
- name: notes-api-dashboard
files:
- notes-api.json
options:
labels:
grafana_dashboard: "1"
disableNameSuffixHash: true
EOF
kubectl apply -k .Expected:
configmap/notes-api-dashboard created
- Kustomize is built into
kubectl(-k).configMapGeneratorcreates a ConfigMap from the JSON file. You don’t write any JSON into YAML. - The label
grafana_dashboard: "1"is the signal for the sidecar. disableNameSuffixHash: truekeeps the name fixed. Otherwise Kustomize appends a hash of the content, and every change would result in a new ConfigMap.- The ConfigMap lives in
app, with the application. The sidecar searches all namespaces.
What landed in the cluster:
kubectl -n app get configmap -l grafana_dashboard=1Expected:
NAME DATA AGE
notes-api-dashboard 1 71s
Save your progress? Sign up for free
Drill 3: Grafana Has Loaded It
The sidecar reports in its log:
kubectl -n monitoring logs deploy/kps-grafana -c grafana-sc-dashboard | grep notes-apiExpected:
{"time": "...", "level": "INFO", "msg": "Writing /tmp/dashboards/notes-api.json (as...
The sidecar writes the file and then asks Grafana to reread its dashboards. Check via the API:
curl -s -u admin:lab-grafana-pw-123 localhost:30300/api/dashboards/uid/notes-api-code \
| jq '{title: .dashboard.title, provisioned: .meta.provisioned}'Expected:
{
"title": "notes-api Überblick",
"provisioned": true
}
provisioned: true means: the dashboard belongs to the
file, not to the database. Open it in the browser at
http://<VM-IP>:30300/d/notes-api-code. If you change
something there and want to save, Grafana refuses and offers to copy the
JSON. The truth lives in Git.
At the top is the drop-down list Pod. Choose a
single pod: the bottom panel only shows that one. The top panels stay
the same because their queries don’t use $pod.
Save your progress? Sign up for free
Drill 4: Change and Roll Out Again
The “Pods bereit” panel should turn red when fewer than two pods are
ready. With jq you add the thresholds without editing the
file by hand:
jq '.panels[0].fieldConfig = {"defaults": {"thresholds": {"mode": "absolute",
"steps": [{"color": "red", "value": null}, {"color": "green", "value": 2}]}},
"overrides": []}' notes-api.json > neu.json
mv neu.json notes-api.json
kubectl apply -k .Expected:
configmap/notes-api-dashboard configured
- The first step (
value: null) applies from minus infinity: red. From 2 on it turns green. > notes-api.jsonwould empty the file beforejqreads it. Hence the detour vianeu.json.
After about a minute:
curl -s -u admin:lab-grafana-pw-123 localhost:30300/api/dashboards/uid/notes-api-code \
| jq -c '.dashboard.panels[0].fieldConfig.defaults.thresholds.steps'Expected:
[{"color":"red","value":null},{"color":"green","value":2}]
Try it out:
kubectl -n app scale deploy/notes-api --replicas 1Expected: after a scrape, the panel shows 1 on a red
background. Back:
kubectl -n app scale deploy/notes-api --replicas 2Save your progress? Sign up for free
Checkpoint
curl -s -u admin:lab-grafana-pw-123 'localhost:30300/api/search?tag=notes-api' \
| jq -r '.[].title'Expected:
notes-api Überblick
tags from the file become a search term.
kubectl get configmap -A -l grafana_dashboard=1 --no-headers | wc -lExpected:
26
25 from the chart, one from you. Only the clicked-together dashboard from Day 5 lives in the database alone.
Challenge
Move the panel Fehlerquote from Day 5 into the file:
at the bottom, full width, unit percentunit.
Hint: Don’t export the panel – write it following
the pattern of the others. jq '.panels += [{...}]' appends
an element. With --arg q '…' you pass the query in as the
variable $q and save yourself escaping the quotes.
Solution
jq --arg q 'sum(job:notes_http_requests:rate5m{code=~"5.."})
/ sum(job:notes_http_requests:rate5m)' \
'.panels += [{"id": 5, "type": "timeseries", "title": "Fehlerquote",
"gridPos": {"x": 0, "y": 13, "w": 24, "h": 6},
"datasource": {"type": "prometheus", "uid": "prometheus"},
"fieldConfig": {"defaults": {"unit": "percentunit"}, "overrides": []},
"targets": [{"refId": "A", "expr": $q}]}]' notes-api.json > neu.json
mv neu.json notes-api.json
kubectl apply -k .After a minute:
curl -s -u admin:lab-grafana-pw-123 localhost:30300/api/dashboards/uid/notes-api-code \
| jq '.dashboard.panels | length'Expected:
5
Save your progress? Sign up for free
Enterprise Note
- Programming dashboards: With many similar dashboards, JSON gets tedious. Grafonnet (Jsonnet) and the Grafana Foundation SDK (Go, Python, TypeScript) generate the JSON from code with loops and functions.
- Folders: With
grafana.sidecar.dashboards.folderAnnotationin the chart values, an annotation on the ConfigMap determines the folder. That way every team gets its own. - Grafana Operator: If you run several Grafana
instances, you describe dashboards, data sources and folders as their
own resources (
GrafanaDashboard). The sidecar is the lean variant for one instance.