Skip to content
Hands-on IT IT Drills

Day 6: Dashboards as Code

Goal: A dashboard as a JSON file, put into the cluster as a ConfigMap with Kustomize and loaded by Grafana automatically – with a variable and without clicking. · Duration: approx. 60 min · Prerequisite: Day 5

Briefing

  • A clicked-together dashboard lives in Grafana’s database. Nobody sees who changed what, and after a reinstall it’s gone.
  • In the Grafana pod runs a sidecar that looks for ConfigMaps with the label grafana_dashboard: "1" in all namespaces. When it finds one, it stores its content as a file, and Grafana loads the dashboard.
  • That way the dashboard lives next to the application in Git, goes through reviews and is rolled out with it. The bundled dashboards come the same way.
  • Variables make a dashboard reusable: a drop-down list at the top sets a value that every query uses.
 dashboards/notes-api.json ─► kubectl apply -k ─► ConfigMap (label grafana_dashboard)
                                                       │ sidecar in the Grafana pod
                                                       ▼
                                             Grafana loads the dashboard

Drill 1: The Dashboard as a File

mkdir -p ~/obs-lab/dashboards
cd ~/obs-lab/dashboards
cat > notes-api.json <<'EOF'
{
  "uid": "notes-api-code",
  "title": "notes-api Überblick",
  "tags": ["notes-api"],
  "time": {"from": "now-30m", "to": "now"},
  "refresh": "30s",
  "schemaVersion": 41,
  "templating": {
    "list": [
      {
        "name": "pod",
        "label": "Pod",
        "type": "query",
        "datasource": {"type": "prometheus", "uid": "prometheus"},
        "query": {"query": "label_values(up{job=\"notes-api\"}, pod)", "refId": "pod"},
        "refresh": 2,
        "multi": true,
        "includeAll": true,
        "current": {"text": "All", "value": "$__all"}
      }
    ]
  },
  "panels": [
    {
      "id": 1, "type": "stat", "title": "Pods bereit",
      "gridPos": {"x": 0, "y": 0, "w": 4, "h": 6},
      "datasource": {"type": "prometheus", "uid": "prometheus"},
      "targets": [{"refId": "A", "expr": "count(up{job=\"notes-api\"} == 1)"}]
    },
    {
      "id": 2, "type": "timeseries", "title": "Anfragen pro Sekunde",
      "gridPos": {"x": 4, "y": 0, "w": 10, "h": 6},
      "datasource": {"type": "prometheus", "uid": "prometheus"},
      "fieldConfig": {"defaults": {"unit": "reqps"}, "overrides": []},
      "targets": [{"refId": "A", "legendFormat": "{{method}}",
                   "expr": "sum by (method) (job:notes_http_requests:rate5m)"}]
    },
    {
      "id": 3, "type": "timeseries", "title": "Antwortzeit p95",
      "gridPos": {"x": 14, "y": 0, "w": 10, "h": 6},
      "datasource": {"type": "prometheus", "uid": "prometheus"},
      "fieldConfig": {"defaults": {"unit": "s"}, "overrides": []},
      "targets": [{"refId": "A", "legendFormat": "p95",
                   "expr": "job:notes_http_request_duration_seconds:p95_5m"}]
    },
    {
      "id": 4, "type": "timeseries", "title": "Notizen pro Sekunde je Pod",
      "gridPos": {"x": 0, "y": 6, "w": 24, "h": 7},
      "datasource": {"type": "prometheus", "uid": "prometheus"},
      "targets": [{"refId": "A", "legendFormat": "{{pod}}",
        "expr": "sum by (pod) (rate(notes_created_total{pod=~\"$pod\"}[5m]))"}]
    }
  ]
}
EOF
jq -r '.panels[].title' notes-api.json

Expected:

Pods bereit
Anfragen pro Sekunde
Antwortzeit p95
Notizen pro Sekunde je Pod
  • jq reads the file. If it isn’t valid JSON, it reports the error with line and column – before Grafana silently fails on it.
  • uid is the fixed name. It keeps the dashboard’s address the same no matter how often you reload it. The title differs from the one on Day 5, because Grafana rejects two dashboards with the same title in the same folder.
  • The data source is set as uid: prometheus – the fixed name from Day 5, Drill 2.
  • gridPos sets position and size: 24 columns wide, h in grid rows.
  • The variable pod gets its values from Prometheus with label_values(...). $pod in the fourth panel becomes a regular expression like notes-api-57f4f47769-5x8dq|notes-api-57f4f47769-qj4nb. Hence =~.

Save your progress? Sign up for free

Drill 2: Kustomize Builds the ConfigMap

cat > kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: app
configMapGenerator:
  - name: notes-api-dashboard
    files:
      - notes-api.json
    options:
      labels:
        grafana_dashboard: "1"
      disableNameSuffixHash: true
EOF
kubectl apply -k .

Expected:

configmap/notes-api-dashboard created
  • Kustomize is built into kubectl (-k). configMapGenerator creates a ConfigMap from the JSON file. You don’t write any JSON into YAML.
  • The label grafana_dashboard: "1" is the signal for the sidecar.
  • disableNameSuffixHash: true keeps the name fixed. Otherwise Kustomize appends a hash of the content, and every change would result in a new ConfigMap.
  • The ConfigMap lives in app, with the application. The sidecar searches all namespaces.

What landed in the cluster:

kubectl -n app get configmap -l grafana_dashboard=1

Expected:

NAME                  DATA   AGE
notes-api-dashboard   1      71s

Save your progress? Sign up for free

Drill 3: Grafana Has Loaded It

The sidecar reports in its log:

kubectl -n monitoring logs deploy/kps-grafana -c grafana-sc-dashboard | grep notes-api

Expected:

{"time": "...", "level": "INFO", "msg": "Writing /tmp/dashboards/notes-api.json (as...

The sidecar writes the file and then asks Grafana to reread its dashboards. Check via the API:

curl -s -u admin:lab-grafana-pw-123 localhost:30300/api/dashboards/uid/notes-api-code \
  | jq '{title: .dashboard.title, provisioned: .meta.provisioned}'

Expected:

{
  "title": "notes-api Überblick",
  "provisioned": true
}

provisioned: true means: the dashboard belongs to the file, not to the database. Open it in the browser at http://<VM-IP>:30300/d/notes-api-code. If you change something there and want to save, Grafana refuses and offers to copy the JSON. The truth lives in Git.

At the top is the drop-down list Pod. Choose a single pod: the bottom panel only shows that one. The top panels stay the same because their queries don’t use $pod.

Save your progress? Sign up for free

Drill 4: Change and Roll Out Again

The “Pods bereit” panel should turn red when fewer than two pods are ready. With jq you add the thresholds without editing the file by hand:

jq '.panels[0].fieldConfig = {"defaults": {"thresholds": {"mode": "absolute",
  "steps": [{"color": "red", "value": null}, {"color": "green", "value": 2}]}},
  "overrides": []}' notes-api.json > neu.json
mv neu.json notes-api.json
kubectl apply -k .

Expected:

configmap/notes-api-dashboard configured
  • The first step (value: null) applies from minus infinity: red. From 2 on it turns green.
  • > notes-api.json would empty the file before jq reads it. Hence the detour via neu.json.

After about a minute:

curl -s -u admin:lab-grafana-pw-123 localhost:30300/api/dashboards/uid/notes-api-code \
  | jq -c '.dashboard.panels[0].fieldConfig.defaults.thresholds.steps'

Expected:

[{"color":"red","value":null},{"color":"green","value":2}]

Try it out:

kubectl -n app scale deploy/notes-api --replicas 1

Expected: after a scrape, the panel shows 1 on a red background. Back:

kubectl -n app scale deploy/notes-api --replicas 2

Save your progress? Sign up for free

Checkpoint

curl -s -u admin:lab-grafana-pw-123 'localhost:30300/api/search?tag=notes-api' \
  | jq -r '.[].title'

Expected:

notes-api Überblick

tags from the file become a search term.

kubectl get configmap -A -l grafana_dashboard=1 --no-headers | wc -l

Expected:

26

25 from the chart, one from you. Only the clicked-together dashboard from Day 5 lives in the database alone.

Challenge

Move the panel Fehlerquote from Day 5 into the file: at the bottom, full width, unit percentunit.

Hint: Don’t export the panel – write it following the pattern of the others. jq '.panels += [{...}]' appends an element. With --arg q '…' you pass the query in as the variable $q and save yourself escaping the quotes.

Solution

jq --arg q 'sum(job:notes_http_requests:rate5m{code=~"5.."})
  / sum(job:notes_http_requests:rate5m)' \
  '.panels += [{"id": 5, "type": "timeseries", "title": "Fehlerquote",
  "gridPos": {"x": 0, "y": 13, "w": 24, "h": 6},
  "datasource": {"type": "prometheus", "uid": "prometheus"},
  "fieldConfig": {"defaults": {"unit": "percentunit"}, "overrides": []},
  "targets": [{"refId": "A", "expr": $q}]}]' notes-api.json > neu.json
mv neu.json notes-api.json
kubectl apply -k .

After a minute:

curl -s -u admin:lab-grafana-pw-123 localhost:30300/api/dashboards/uid/notes-api-code \
  | jq '.dashboard.panels | length'

Expected:

5

Save your progress? Sign up for free

Enterprise Note

  • Programming dashboards: With many similar dashboards, JSON gets tedious. Grafonnet (Jsonnet) and the Grafana Foundation SDK (Go, Python, TypeScript) generate the JSON from code with loops and functions.
  • Folders: With grafana.sidecar.dashboards.folderAnnotation in the chart values, an annotation on the ConfigMap determines the folder. That way every team gets its own.
  • Grafana Operator: If you run several Grafana instances, you describe dashboards, data sources and folders as their own resources (GrafanaDashboard). The sidecar is the lean variant for one instance.