Day 2: Prometheus in the Cluster
Goal: kube-prometheus-stack is running, and Prometheus finds the pods of notes-api on its own through a ServiceMonitor. · Duration: approx. 60 min · Prerequisite: Day 1
Briefing
- Prometheus fetches metrics by scraping, stores them as time series and answers queries in PromQL.
- kube-prometheus-stack is a Helm chart that brings a complete observability stack: Prometheus, Alertmanager, Grafana, exporters for nodes and Kubernetes objects, and ready-made rules.
- At its core is the Prometheus Operator. It
introduces new resource types, such as
ServiceMonitor. In it you describe what should be scraped, and the operator writes the Prometheus configuration from that. - A ServiceMonitor selects Services by labels. Prometheus then scrapes every pod behind the Service individually – exactly what Day 1 called for.
ServiceMonitor notes-api ──► Operator ──► Prometheus configuration
│
Service notes-api ── endpoints ── Pod A, Pod B ◄── scrape every 15 s
Drill 1: The Chart
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo updateExpected (shortened):
"prometheus-community" has been added to your repositories
...
Update Complete. ⎈Happy Helming!⎈
helm search repo prometheus-community/kube-prometheus-stackExpected:
NAME CHART VERSION APP VERSION DESCRIPTION
prometheus-community/kube-prometheus-stack 92.18.0 v0.94.2 kube-prom...
The app version is that of the operator. The chart brings Prometheus 3.15, Alertmanager 0.34 and Grafana 13.3.
Save your progress? Sign up for free
Drill 2: The Values
The chart has over 5000 lines of defaults. You only override what should be different in the lab:
cd ~/obs-lab
cat > kps.yaml <<'EOF'
fullnameOverride: kps
prometheus:
service:
type: NodePort
nodePort: 30090
prometheusSpec:
serviceMonitorSelectorNilUsesHelmValues: false
ruleSelectorNilUsesHelmValues: false
retention: 3d
resources:
requests: {cpu: 100m, memory: 300Mi}
limits: {memory: 800Mi}
alertmanager:
service:
type: NodePort
nodePort: 30093
grafana:
adminPassword: lab-grafana-pw-123
service:
type: NodePort
nodePort: 30300
kubeControllerManager: {enabled: false}
kubeScheduler: {enabled: false}
kubeProxy: {enabled: false}
kubeEtcd: {enabled: false}
EOFfullnameOverride: kpsshortens the names of all objects. Without it, the Prometheus pod would be calledprometheus-kps-kube-prometheus-stack-prometheus-0.serviceMonitorSelectorNilUsesHelmValues: falseis the most important line. Otherwise Prometheus only considers ServiceMonitors with the labelrelease: kps– and yours would be silently ignored. The same applies to rules withruleSelector….retention: 3d: Prometheus forgets data after three days. That keeps the VM lean.- With k3s, controller manager, scheduler, kube-proxy and etcd run in one process and can’t be reached from the outside. The chart would look for them in vain.
Save your progress? Sign up for free
Drill 3: Installing
helm install kps prometheus-community/kube-prometheus-stack --version 92.18.0 \
-n monitoring --create-namespace -f kps.yaml --wait --timeout 10mExpected (after one to three minutes, shortened):
NAME: kps
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
kubectl -n monitoring get podsExpected:
NAME READY STATUS RESTARTS AGE
alertmanager-kps-alertmanager-0 2/2 Running 0 42s
kps-grafana-76b59ccc8-746jn 3/3 Running 0 46s
kps-kube-state-metrics-dc46747f8-tjlrd 1/1 Running 0 46s
kps-operator-7c868dc646-jxcpw 1/1 Running 0 46s
kps-prometheus-node-exporter-2hzgk 1/1 Running 0 46s
prometheus-kps-prometheus-0 2/2 Running 0 40s
- node-exporter delivers metrics of the VM: CPU, memory, disk, network.
- kube-state-metrics translates the state of Kubernetes objects into metrics: how many replicas are desired, how many are ready?
- The pods
prometheus-…andalertmanager-…aren’t created by Helm but by the operator.
The new resource types:
kubectl get crd | grep monitoring.coreos.comExpected (shortened):
alertmanagerconfigs.monitoring.coreos.com Namespaced v1alpha1(storage) 2026-...
alertmanagers.monitoring.coreos.com Namespaced v1(storage) 2026-...
podmonitors.monitoring.coreos.com Namespaced v1(storage) 2026-...
...
prometheusrules.monitoring.coreos.com Namespaced v1(storage) 2026-...
servicemonitors.monitoring.coreos.com Namespaced v1(storage) 2026-...
Save your progress? Sign up for free
Drill 4: promtool
You reach the Prometheus web UI at
http://<VM-IP>:30090. For the shell there is
promtool, the tool from the Prometheus package:
cd ~/obs-lab
curl -fsSLO https://github.com/prometheus/prometheus/releases/download/\
v3.15.1/prometheus-3.15.1.linux-amd64.tar.gz
tar xzf prometheus-3.15.1.linux-amd64.tar.gz
mv prometheus-3.15.1.linux-amd64/promtool ~/.local/bin/
rm -r prometheus-3.15.1.linux-amd64*
promtool --version | head -1Expected:
promtool, version 3.15.1 (branch: HEAD, revision: bae3eced0a5369e050e83ea...)
The first query: which targets is Prometheus already scraping?
promtool query instant http://localhost:30090 'count by (job) (up)'Expected (the order changes):
{job="kubelet"} => 3 @[1793866605.06]
{job="kps-alertmanager"} => 2 @[1793866605.06]
{job="kube-state-metrics"} => 1 @[1793866605.06]
{job="kps-grafana"} => 1 @[1793866605.06]
{job="kps-prometheus"} => 2 @[1793866605.06]
{job="coredns"} => 1 @[1793866605.06]
{job="apiserver"} => 1 @[1793866605.06]
{job="node-exporter"} => 1 @[1793866605.06]
{job="kps-operator"} => 1 @[1793866605.06]
upis a metric that Prometheus creates itself: 1 if the last scrape worked, otherwise 0. One time series per target.count by (job)counts the targets per job. The job is named after the Service through which Prometheus found the target.kubelethas three targets: the kubelet itself, cAdvisor (container metrics) and the probes. Alertmanager and Prometheus count twice because their helper containerconfig-reloaderalso delivers metrics.- After
@is the time of the answer in Unix seconds. - notes-api is still missing. Prometheus doesn’t know about it.
Warning: Does
kubeletshow six targets instead of three? Then you already installed the chart once under a different name. The operator creates a Service for the kubelet inkube-system, andhelm uninstallleaves it behind.kubectl -n kube-system get svcshows it,kubectl deleteremoves the old one.
Save your progress? Sign up for free
Drill 5: The ServiceMonitor
cat > notes-servicemonitor.yaml <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: notes-api
namespace: app
spec:
selector:
matchLabels:
app.kubernetes.io/name: notes-api
endpoints:
- port: http
path: /metrics
interval: 15s
EOF
kubectl apply -f notes-servicemonitor.yamlExpected:
servicemonitor.monitoring.coreos.com/notes-api created
selectorlooks for Services with the labelapp.kubernetes.io/name: notes-api– in the same namespace as the ServiceMonitor.port: httpis the name of the port in the Service. That’s why it got a name on Day 0. A number doesn’t work here.interval: 15s: one scrape per pod every 15 seconds.
The operator needs a few seconds to rewrite the configuration, and Prometheus needs a bit until the first scrape. After about a minute:
promtool query instant http://localhost:30090 'up{job="notes-api"}'Expected (each time series is on one line, wrapped here):
up{container="notes-api", endpoint="http", instance="10.42.0.43:8080", job="notes-api",
namespace="app", pod="notes-api-57f4f47769-59ks2", service="notes-api"} => 1 @[...]
up{container="notes-api", endpoint="http", instance="10.42.0.42:8080", job="notes-api",
namespace="app", pod="notes-api-57f4f47769-m66nj", service="notes-api"} => 1 @[...]
Two time series, one per pod. Prometheus attached the labels
namespace, pod, service and
instance itself – from what it found out through
Kubernetes.
Save your progress? Sign up for free
Drill 6: The Metrics of notes-api
promtool query instant http://localhost:30090 'notes_created_total'Expected (shortened, the numbers differ):
notes_created_total{..., pod="notes-api-57f4f47769-59ks2", ...} => 146 @[...]
notes_created_total{..., pod="notes-api-57f4f47769-m66nj", ...} => 148 @[...]
These are the same numbers you read on Day 1 with
port-forward – except that Prometheus now fetches and
stores them for both pods every 15 seconds.
promtool query instant http://localhost:30090 'sum(notes_created_total)'Expected:
{} => 294 @[...]
sum adds across all time series. Without
by, all labels disappear in the process. That is the sum
from Day 1, Drill 6 – without doing the math.
What the configuration made of it is shown in the web UI under
Status → Target health. There, notes-api is listed as
serviceMonitor/app/notes-api/0 with two endpoints.
Save your progress? Sign up for free
Checkpoint
promtool query instant http://localhost:30090 'count(up{job="notes-api"} == 1)'Expected:
{} => 2 @[...]
== 1 keeps only the targets whose last scrape worked.
Two pods, both reachable.
kubectl -n monitoring get prometheus kps-prometheusExpected:
NAME VERSION DESIRED READY RECONCILED AVAILABLE AGE
kps-prometheus v3.15.1-distroless 1 1 True True 2m26s
Prometheus itself is also a resource of the operator.
Challenge
Scale notes-api to three pods. What do you have to change in the ServiceMonitor so that Prometheus scrapes the third pod?
Hint:
kubectl -n app scale deploy/notes-api --replicas 3, wait a
minute, then the query from the checkpoint.
Solution
Nothing:
kubectl -n app scale deploy/notes-api --replicas 3After a minute:
promtool query instant http://localhost:30090 'count(up{job="notes-api"} == 1)'Expected:
{} => 3 @[...]
The ServiceMonitor points to the Service, not to pods. Prometheus watches its endpoints and notices new pods on its own. That’s called service discovery. Back to two:
kubectl -n app scale deploy/notes-api --replicas 2Save your progress? Sign up for free
Enterprise Note
- Storage: In the lab, Prometheus data lives in an
emptyDirand is gone after a pod restart. In production, a PersistentVolume belongs underprometheusSpec.storageSpec. - High availability: Two Prometheus replicas scrape the same thing, Alertmanager removes duplicate alerts. For long retention and queries across many clusters, Thanos or Mimir come in.
- Who may scrape: ServiceMonitors in every namespace
mean that every team decides for itself what gets scraped. With
serviceMonitorNamespaceSelectoryou limit that to approved namespaces.