Day 14: Running the Stack
Goal: You know what Prometheus stores and what it costs, cut off expensive metrics, observe the stack itself and run through an upgrade including a rollback. · Duration: approx. 75 min · Prerequisite: Day 13
Briefing
- An observability stack is an application itself – with storage that fills up, queries that get too expensive and upgrades that can go wrong.
- The biggest cost driver in Prometheus is the number of time series in the head, the part of the data that is still in memory. Each time series costs a few kilobytes there.
- What isn’t needed shouldn’t be stored in the first place. Relabeling discards metrics at scrape time, before they take up space.
- Meta-monitoring means: Prometheus, Loki and the Collector report their own metrics. Whoever observes must be observed too.
Scrape ──► metricRelabelings ──► Head (RAM) ──► blocks (disk) ──► retention
(discard) │
prometheus_tsdb_*
Drill 1: What Prometheus Stores
curl -s localhost:30090/api/v1/status/tsdb | jq '.data.headStats'Expected (the numbers differ):
{
"numSeries": 128726,
"numLabelPairs": 6410,
"chunkCount": 253003,
"minTime": 1793871784853,
"maxTime": 1793874279677
}
Almost 130,000 time series. Which metrics bring the most?
curl -s localhost:30090/api/v1/status/tsdb \
| jq -r '.data.seriesCountByMetricName[:5][] | "\(.value)\t\(.name)"'Expected (roughly):
13702 apiserver_request_duration_seconds_bucket
10608 etcd_request_duration_seconds_bucket
9984 apiserver_request_body_size_bytes_bucket
8850 apiserver_request_sli_duration_seconds_bucket
7560 apiserver_watch_events_dispatch_duration_seconds_bucket
- The front-runners are histograms of the API server: many buckets times many resource types times many verbs. notes-api is far behind.
/api/v1/status/tsdbreturns the same data as the page Status → TSDB Status in the UI.
Save your progress? Sign up for free
Drill 2: What notes-api Costs
promtool query instant http://localhost:30090 'count({job="notes-api"})'
promtool query instant http://localhost:30090 \
'count by (__name__) ({job="notes-api", __name__=~"go_.*"})' | wc -lExpected (roughly):
{} => 212 @[...]
31
212 time series for two pods, plus 31 different Go runtime metrics.
Does anyone need go_gc_duration_seconds in this lab? No. A
metricRelabelings entry in the ServiceMonitor discards them
at scrape time – here is the whole file from Day 2:
cd ~/obs-lab
cat > notes-servicemonitor.yaml <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: notes-api
namespace: app
spec:
selector:
matchLabels:
app.kubernetes.io/name: notes-api
endpoints:
- port: http
path: /metrics
interval: 15s
metricRelabelings:
- sourceLabels: [__name__]
regex: go_(gc|memstats|sched|cpu)_.*
action: drop
EOF
kubectl apply -f notes-servicemonitor.yamlExpected:
servicemonitor.monitoring.coreos.com/notes-api configured
metricRelabelingsapplies after the scrape, before storage.sourceLabels: [__name__]checks the metric name againstregex.go_goroutinesandgo_infostay – they don’t match the pattern and are useful.- There is also
relabelings(without “metric”): it applies before the scrape to the targets, for example to exclude whole pods.
After a minute, Prometheus only writes the remaining time series. The old ones drop out of the query after five minutes without a new value:
promtool query instant http://localhost:30090 'count({job="notes-api"})'Expected (after about six minutes, roughly):
{} => 148 @[...]
Almost a third saved – without a line of code.
Save your progress? Sign up for free
Drill 3: The Stack Observes Itself
How many samples does Prometheus write per second?
promtool query instant http://localhost:30090 \
'sum(rate(prometheus_tsdb_head_samples_appended_total[5m]))'Expected (roughly):
{} => 4230.422222222222 @[...]
A good 4000 samples per second. sum combines the time
series for simple values and histograms, which Prometheus counts
separately.
And who delivers the most per scrape?
promtool query instant http://localhost:30090 \
'topk(3, sum by (job) (scrape_samples_post_metric_relabeling))'Expected (roughly):
{job="kubelet"} => 63867 @[...]
{job="apiserver"} => 46654 @[...]
{job="kps-grafana"} => 4796 @[...]
- Prometheus creates
scrape_samples_post_metric_relabelingitself for every target – after the relabeling from Drill 2. Without_post_metric_relabelingyou see the number before. - Kubelet and API server are the most expensive customers: cAdvisor delivers dozens of values per container, the API server histograms per resource and verb. In many clusters, metrics nobody queries are discarded there.
How much space does the data take on disk?
promtool query instant http://localhost:30090 \
'prometheus_tsdb_head_chunks_storage_size_bytes / 1024 / 1024'
promtool query instant http://localhost:30090 \
'prometheus_tsdb_storage_blocks_bytes / 1024 / 1024'Expected (shortened, roughly):
{..., job="kps-prometheus", ...} => 7.999890327453613 @[...]
{..., job="kps-prometheus", ...} => 0 @[...]
- About 8 MB are on disk as head chunks. Every two hours, Prometheus writes a block from them; the second number only grows then.
- Since the restart on Day 11, no block has been completed yet. With
retention: 3d, storage grows up to a limit and then stays there.
Save your progress? Sign up for free
Drill 4: A Safety Net Against Cardinality
What if an application suddenly delivers millions of time series? A ServiceMonitor can set an upper limit per scrape. Try it with one that’s too small:
kubectl -n app patch servicemonitor notes-api --type merge \
-p '{"spec":{"sampleLimit":50}}'Expected:
servicemonitor.monitoring.coreos.com/notes-api patched
After a minute:
promtool query instant http://localhost:30090 'up{job="notes-api"}'Expected (shortened):
up{..., pod="notes-api-76d9df4449-8rszn", ...} => 0 @[...]
up{..., pod="notes-api-76d9df4449-zgs5d", ...} => 0 @[...]
- notes-api delivers more than 50 samples. Prometheus discards the
whole scrape and sets
upto 0 – better no data than storage that overflows. - The Target health page names the reason: “sample
limit exceeded”.
scrape_samples_scrapedshows how many samples there were. NotesApiDownfrom Day 4 speaks up shortly – rightly so: to Prometheus, notes-api is invisible.
patch was only for the experiment. The file is the truth
– but apply doesn’t help here:
kubectl apply -f notes-servicemonitor.yaml
kubectl -n app get servicemonitor notes-api -o jsonpath='{.spec.sampleLimit}{"\n"}'Expected:
servicemonitor.monitoring.coreos.com/notes-api unchanged
50
apply only removes fields it once set itself. It
remembers its last state in the annotation
kubectl.kubernetes.io/last-applied-configuration.
sampleLimit came from patch –
apply knows nothing about it. replace replaces
the whole object with the file:
kubectl replace -f notes-servicemonitor.yamlExpected:
servicemonitor.monitoring.coreos.com/notes-api replaced
Save your progress? Sign up for free
Drill 5: Upgrade and Rollback
What has Helm installed, and in which revision?
helm list -n monitoring -o json | jq -r '.[] | [.name, .revision, .chart] | @tsv'Expected:
alloy 2 alloy-1.15.0
kps 4 kube-prometheus-stack-92.18.0
loki 2 loki-20.4.1
otel 4 opentelemetry-collector-0.178.2
tempo 1 tempo-3.2.1
Every change to the values was a revision. helm history
shows them:
helm history alloy -n monitoringExpected:
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
1 Thu Nov 5 09:07:47 2026 superseded alloy-1.15.0 v1.21.3 Install co...
2 Thu Nov 5 10:20:49 2026 deployed alloy-1.15.0 v1.21.3 Upgrade co...
Run through a rollback – revision 1 is Alloy without the events from Day 13:
helm rollback alloy 1 -n monitoring --wait
helm history alloy -n monitoring --max 1Expected:
Rollback was a success! Happy Helming!
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
3 Thu Nov 5 10:33:53 2026 deployed alloy-1.15.0 v1.21.3 Rollback to 1
- A rollback is a new revision, not a deletion of history.
- Now Alloy no longer collects events. But the state in
alloy.yamlis still the one with events – file and cluster have drifted apart.
Back to the state of the file:
helm upgrade alloy grafana/alloy --version 1.15.0 -n monitoring -f alloy.yaml --waitExpected (shortened):
Release "alloy" has been upgraded. Happy Helming!
...
REVISION: 4
Which newer versions are available?
helm search repo prometheus-community/kube-prometheus-stack --versions | head -4Expected (the versions differ):
NAME CHART VERSION APP VERSION DESCRIPTION
prometheus-community/kube-prometheus-stack 92.18.0 v0.94.2 kube-prom...
prometheus-community/kube-prometheus-stack 92.17.1 v0.94.2 kube-prom...
prometheus-community/kube-prometheus-stack 92.17.0 v0.94.2 kube-prom...
Warning: With kube-prometheus-stack,
helm upgradedoes not update the CRDs. Before jumping to a new major version, the chart’s instructions list the commands you use to apply the CRDs first. If you skip that, you end up with a new operator and old resource types.
Save your progress? Sign up for free
Checkpoint
promtool query instant http://localhost:30090 'count(up{job="notes-api"} == 1)'Expected:
{} => 2 @[...]
notes-api is being scraped again.
kubectl -n app get servicemonitor notes-api -o jsonpath='{.spec.sampleLimit}{"\n"}'Expected: an empty line. No limit anymore.
helm history alloy -n monitoring --max 1 -o json | jq -r '.[0].description'Expected:
Upgrade complete
Challenge
Set a sensible sampleLimit for notes-api: generous
enough for everyday use, small enough to catch an explosion. How many
samples does a pod deliver right now?
Hint:
scrape_samples_scraped{job="notes-api"}. A buffer of about
five times that is a good start.
Solution
promtool query instant http://localhost:30090 'scrape_samples_scraped{job="notes-api"}'Expected (shortened, roughly):
scrape_samples_scraped{..., pod="notes-api-76d9df4449-8rszn", ...} => 99 @[...]
scrape_samples_scraped{..., pod="notes-api-76d9df4449-zgs5d", ...} => 99 @[...]
scrape_samples_scraped counts before relabeling – the
limit applies after. About 75 samples remain; a limit of 500 leaves
room. Into the file, directly under spec:
cat > notes-servicemonitor.yaml <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: notes-api
namespace: app
spec:
selector:
matchLabels:
app.kubernetes.io/name: notes-api
sampleLimit: 500
endpoints:
- port: http
path: /metrics
interval: 15s
metricRelabelings:
- sourceLabels: [__name__]
regex: go_(gc|memstats|sched|cpu)_.*
action: drop
EOF
kubectl apply -f notes-servicemonitor.yamlExpected:
servicemonitor.monitoring.coreos.com/notes-api configured
Save your progress? Sign up for free
Enterprise Note
- Long-term storage: For months instead of days,
Prometheus sends its data via
remote_writeto Thanos, Mimir or VictoriaMetrics. Prometheus itself then keeps only hours. - Meta-monitoring from the outside: A Prometheus can’t report that it’s dead. A second, small Prometheus – or an external service with the Watchdog from Day 4 – monitors the first one.
- Rehearse upgrades: Values files in Git, a test
cluster with the same configuration,
helm diff upgrade(plugin) before every step. And always read the notes for the new major version.