Skip to content
Hands-on IT IT Drills

Day 14: Running the Stack

Goal: You know what Prometheus stores and what it costs, cut off expensive metrics, observe the stack itself and run through an upgrade including a rollback. · Duration: approx. 75 min · Prerequisite: Day 13

Briefing

  • An observability stack is an application itself – with storage that fills up, queries that get too expensive and upgrades that can go wrong.
  • The biggest cost driver in Prometheus is the number of time series in the head, the part of the data that is still in memory. Each time series costs a few kilobytes there.
  • What isn’t needed shouldn’t be stored in the first place. Relabeling discards metrics at scrape time, before they take up space.
  • Meta-monitoring means: Prometheus, Loki and the Collector report their own metrics. Whoever observes must be observed too.
 Scrape ──► metricRelabelings ──► Head (RAM) ──► blocks (disk) ──► retention
             (discard)              │
                             prometheus_tsdb_*

Drill 1: What Prometheus Stores

curl -s localhost:30090/api/v1/status/tsdb | jq '.data.headStats'

Expected (the numbers differ):

{
  "numSeries": 128726,
  "numLabelPairs": 6410,
  "chunkCount": 253003,
  "minTime": 1793871784853,
  "maxTime": 1793874279677
}

Almost 130,000 time series. Which metrics bring the most?

curl -s localhost:30090/api/v1/status/tsdb \
  | jq -r '.data.seriesCountByMetricName[:5][] | "\(.value)\t\(.name)"'

Expected (roughly):

13702   apiserver_request_duration_seconds_bucket
10608   etcd_request_duration_seconds_bucket
9984    apiserver_request_body_size_bytes_bucket
8850    apiserver_request_sli_duration_seconds_bucket
7560    apiserver_watch_events_dispatch_duration_seconds_bucket
  • The front-runners are histograms of the API server: many buckets times many resource types times many verbs. notes-api is far behind.
  • /api/v1/status/tsdb returns the same data as the page Status → TSDB Status in the UI.

Save your progress? Sign up for free

Drill 2: What notes-api Costs

promtool query instant http://localhost:30090 'count({job="notes-api"})'
promtool query instant http://localhost:30090 \
  'count by (__name__) ({job="notes-api", __name__=~"go_.*"})' | wc -l

Expected (roughly):

{} => 212 @[...]
31

212 time series for two pods, plus 31 different Go runtime metrics. Does anyone need go_gc_duration_seconds in this lab? No. A metricRelabelings entry in the ServiceMonitor discards them at scrape time – here is the whole file from Day 2:

cd ~/obs-lab
cat > notes-servicemonitor.yaml <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: notes-api
  namespace: app
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: notes-api
  endpoints:
    - port: http
      path: /metrics
      interval: 15s
      metricRelabelings:
        - sourceLabels: [__name__]
          regex: go_(gc|memstats|sched|cpu)_.*
          action: drop
EOF
kubectl apply -f notes-servicemonitor.yaml

Expected:

servicemonitor.monitoring.coreos.com/notes-api configured
  • metricRelabelings applies after the scrape, before storage. sourceLabels: [__name__] checks the metric name against regex.
  • go_goroutines and go_info stay – they don’t match the pattern and are useful.
  • There is also relabelings (without “metric”): it applies before the scrape to the targets, for example to exclude whole pods.

After a minute, Prometheus only writes the remaining time series. The old ones drop out of the query after five minutes without a new value:

promtool query instant http://localhost:30090 'count({job="notes-api"})'

Expected (after about six minutes, roughly):

{} => 148 @[...]

Almost a third saved – without a line of code.

Save your progress? Sign up for free

Drill 3: The Stack Observes Itself

How many samples does Prometheus write per second?

promtool query instant http://localhost:30090 \
  'sum(rate(prometheus_tsdb_head_samples_appended_total[5m]))'

Expected (roughly):

{} => 4230.422222222222 @[...]

A good 4000 samples per second. sum combines the time series for simple values and histograms, which Prometheus counts separately.

And who delivers the most per scrape?

promtool query instant http://localhost:30090 \
  'topk(3, sum by (job) (scrape_samples_post_metric_relabeling))'

Expected (roughly):

{job="kubelet"} => 63867 @[...]
{job="apiserver"} => 46654 @[...]
{job="kps-grafana"} => 4796 @[...]
  • Prometheus creates scrape_samples_post_metric_relabeling itself for every target – after the relabeling from Drill 2. Without _post_metric_relabeling you see the number before.
  • Kubelet and API server are the most expensive customers: cAdvisor delivers dozens of values per container, the API server histograms per resource and verb. In many clusters, metrics nobody queries are discarded there.

How much space does the data take on disk?

promtool query instant http://localhost:30090 \
  'prometheus_tsdb_head_chunks_storage_size_bytes / 1024 / 1024'
promtool query instant http://localhost:30090 \
  'prometheus_tsdb_storage_blocks_bytes / 1024 / 1024'

Expected (shortened, roughly):

{..., job="kps-prometheus", ...} => 7.999890327453613 @[...]
{..., job="kps-prometheus", ...} => 0 @[...]
  • About 8 MB are on disk as head chunks. Every two hours, Prometheus writes a block from them; the second number only grows then.
  • Since the restart on Day 11, no block has been completed yet. With retention: 3d, storage grows up to a limit and then stays there.

Save your progress? Sign up for free

Drill 4: A Safety Net Against Cardinality

What if an application suddenly delivers millions of time series? A ServiceMonitor can set an upper limit per scrape. Try it with one that’s too small:

kubectl -n app patch servicemonitor notes-api --type merge \
  -p '{"spec":{"sampleLimit":50}}'

Expected:

servicemonitor.monitoring.coreos.com/notes-api patched

After a minute:

promtool query instant http://localhost:30090 'up{job="notes-api"}'

Expected (shortened):

up{..., pod="notes-api-76d9df4449-8rszn", ...} => 0 @[...]
up{..., pod="notes-api-76d9df4449-zgs5d", ...} => 0 @[...]
  • notes-api delivers more than 50 samples. Prometheus discards the whole scrape and sets up to 0 – better no data than storage that overflows.
  • The Target health page names the reason: “sample limit exceeded”. scrape_samples_scraped shows how many samples there were.
  • NotesApiDown from Day 4 speaks up shortly – rightly so: to Prometheus, notes-api is invisible.

patch was only for the experiment. The file is the truth – but apply doesn’t help here:

kubectl apply -f notes-servicemonitor.yaml
kubectl -n app get servicemonitor notes-api -o jsonpath='{.spec.sampleLimit}{"\n"}'

Expected:

servicemonitor.monitoring.coreos.com/notes-api unchanged
50

apply only removes fields it once set itself. It remembers its last state in the annotation kubectl.kubernetes.io/last-applied-configuration. sampleLimit came from patch – apply knows nothing about it. replace replaces the whole object with the file:

kubectl replace -f notes-servicemonitor.yaml

Expected:

servicemonitor.monitoring.coreos.com/notes-api replaced

Save your progress? Sign up for free

Drill 5: Upgrade and Rollback

What has Helm installed, and in which revision?

helm list -n monitoring -o json | jq -r '.[] | [.name, .revision, .chart] | @tsv'

Expected:

alloy   2   alloy-1.15.0
kps 4   kube-prometheus-stack-92.18.0
loki    2   loki-20.4.1
otel    4   opentelemetry-collector-0.178.2
tempo   1   tempo-3.2.1

Every change to the values was a revision. helm history shows them:

helm history alloy -n monitoring

Expected:

REVISION  UPDATED                   STATUS      CHART         APP VERSION  DESCRIPTION
1         Thu Nov  5 09:07:47 2026  superseded  alloy-1.15.0  v1.21.3      Install co...
2         Thu Nov  5 10:20:49 2026  deployed    alloy-1.15.0  v1.21.3      Upgrade co...

Run through a rollback – revision 1 is Alloy without the events from Day 13:

helm rollback alloy 1 -n monitoring --wait
helm history alloy -n monitoring --max 1

Expected:

Rollback was a success! Happy Helming!
REVISION  UPDATED                   STATUS    CHART         APP VERSION  DESCRIPTION
3         Thu Nov  5 10:33:53 2026  deployed  alloy-1.15.0  v1.21.3      Rollback to 1
  • A rollback is a new revision, not a deletion of history.
  • Now Alloy no longer collects events. But the state in alloy.yaml is still the one with events – file and cluster have drifted apart.

Back to the state of the file:

helm upgrade alloy grafana/alloy --version 1.15.0 -n monitoring -f alloy.yaml --wait

Expected (shortened):

Release "alloy" has been upgraded. Happy Helming!
...
REVISION: 4

Which newer versions are available?

helm search repo prometheus-community/kube-prometheus-stack --versions | head -4

Expected (the versions differ):

NAME                                        CHART VERSION  APP VERSION  DESCRIPTION
prometheus-community/kube-prometheus-stack  92.18.0        v0.94.2      kube-prom...
prometheus-community/kube-prometheus-stack  92.17.1        v0.94.2      kube-prom...
prometheus-community/kube-prometheus-stack  92.17.0        v0.94.2      kube-prom...

Warning: With kube-prometheus-stack, helm upgrade does not update the CRDs. Before jumping to a new major version, the chart’s instructions list the commands you use to apply the CRDs first. If you skip that, you end up with a new operator and old resource types.

Save your progress? Sign up for free

Checkpoint

promtool query instant http://localhost:30090 'count(up{job="notes-api"} == 1)'

Expected:

{} => 2 @[...]

notes-api is being scraped again.

kubectl -n app get servicemonitor notes-api -o jsonpath='{.spec.sampleLimit}{"\n"}'

Expected: an empty line. No limit anymore.

helm history alloy -n monitoring --max 1 -o json | jq -r '.[0].description'

Expected:

Upgrade complete

Challenge

Set a sensible sampleLimit for notes-api: generous enough for everyday use, small enough to catch an explosion. How many samples does a pod deliver right now?

Hint: scrape_samples_scraped{job="notes-api"}. A buffer of about five times that is a good start.

Solution

promtool query instant http://localhost:30090 'scrape_samples_scraped{job="notes-api"}'

Expected (shortened, roughly):

scrape_samples_scraped{..., pod="notes-api-76d9df4449-8rszn", ...} => 99 @[...]
scrape_samples_scraped{..., pod="notes-api-76d9df4449-zgs5d", ...} => 99 @[...]

scrape_samples_scraped counts before relabeling – the limit applies after. About 75 samples remain; a limit of 500 leaves room. Into the file, directly under spec:

cat > notes-servicemonitor.yaml <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: notes-api
  namespace: app
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: notes-api
  sampleLimit: 500
  endpoints:
    - port: http
      path: /metrics
      interval: 15s
      metricRelabelings:
        - sourceLabels: [__name__]
          regex: go_(gc|memstats|sched|cpu)_.*
          action: drop
EOF
kubectl apply -f notes-servicemonitor.yaml

Expected:

servicemonitor.monitoring.coreos.com/notes-api configured

Save your progress? Sign up for free

Enterprise Note

  • Long-term storage: For months instead of days, Prometheus sends its data via remote_write to Thanos, Mimir or VictoriaMetrics. Prometheus itself then keeps only hours.
  • Meta-monitoring from the outside: A Prometheus can’t report that it’s dead. A second, small Prometheus – or an external service with the Watchdog from Day 4 – monitors the first one.
  • Rehearse upgrades: Values files in Git, a test cluster with the same configuration, helm diff upgrade (plugin) before every step. And always read the notes for the new major version.