Skip to content
Hands-on IT IT Drills

Day 2: Prometheus in the Cluster

Goal: kube-prometheus-stack is running, and Prometheus finds the pods of notes-api on its own through a ServiceMonitor. · Duration: approx. 60 min · Prerequisite: Day 1

Briefing

  • Prometheus fetches metrics by scraping, stores them as time series and answers queries in PromQL.
  • kube-prometheus-stack is a Helm chart that brings a complete observability stack: Prometheus, Alertmanager, Grafana, exporters for nodes and Kubernetes objects, and ready-made rules.
  • At its core is the Prometheus Operator. It introduces new resource types, such as ServiceMonitor. In it you describe what should be scraped, and the operator writes the Prometheus configuration from that.
  • A ServiceMonitor selects Services by labels. Prometheus then scrapes every pod behind the Service individually – exactly what Day 1 called for.
 ServiceMonitor notes-api ──► Operator ──► Prometheus configuration
                                              │
 Service notes-api ── endpoints ── Pod A, Pod B ◄── scrape every 15 s

Drill 1: The Chart

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Expected (shortened):

"prometheus-community" has been added to your repositories
...
Update Complete. ⎈Happy Helming!⎈
helm search repo prometheus-community/kube-prometheus-stack

Expected:

NAME                                        CHART VERSION  APP VERSION  DESCRIPTION
prometheus-community/kube-prometheus-stack  92.18.0        v0.94.2      kube-prom...

The app version is that of the operator. The chart brings Prometheus 3.15, Alertmanager 0.34 and Grafana 13.3.

Save your progress? Sign up for free

Drill 2: The Values

The chart has over 5000 lines of defaults. You only override what should be different in the lab:

cd ~/obs-lab
cat > kps.yaml <<'EOF'
fullnameOverride: kps

prometheus:
  service:
    type: NodePort
    nodePort: 30090
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: false
    ruleSelectorNilUsesHelmValues: false
    retention: 3d
    resources:
      requests: {cpu: 100m, memory: 300Mi}
      limits: {memory: 800Mi}

alertmanager:
  service:
    type: NodePort
    nodePort: 30093

grafana:
  adminPassword: lab-grafana-pw-123
  service:
    type: NodePort
    nodePort: 30300

kubeControllerManager: {enabled: false}
kubeScheduler: {enabled: false}
kubeProxy: {enabled: false}
kubeEtcd: {enabled: false}
EOF
  • fullnameOverride: kps shortens the names of all objects. Without it, the Prometheus pod would be called prometheus-kps-kube-prometheus-stack-prometheus-0.
  • serviceMonitorSelectorNilUsesHelmValues: false is the most important line. Otherwise Prometheus only considers ServiceMonitors with the label release: kps – and yours would be silently ignored. The same applies to rules with ruleSelector….
  • retention: 3d: Prometheus forgets data after three days. That keeps the VM lean.
  • With k3s, controller manager, scheduler, kube-proxy and etcd run in one process and can’t be reached from the outside. The chart would look for them in vain.

Save your progress? Sign up for free

Drill 3: Installing

helm install kps prometheus-community/kube-prometheus-stack --version 92.18.0 \
  -n monitoring --create-namespace -f kps.yaml --wait --timeout 10m

Expected (after one to three minutes, shortened):

NAME: kps
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
kubectl -n monitoring get pods

Expected:

NAME                                     READY   STATUS    RESTARTS   AGE
alertmanager-kps-alertmanager-0          2/2     Running   0          42s
kps-grafana-76b59ccc8-746jn              3/3     Running   0          46s
kps-kube-state-metrics-dc46747f8-tjlrd   1/1     Running   0          46s
kps-operator-7c868dc646-jxcpw            1/1     Running   0          46s
kps-prometheus-node-exporter-2hzgk       1/1     Running   0          46s
prometheus-kps-prometheus-0              2/2     Running   0          40s
  • node-exporter delivers metrics of the VM: CPU, memory, disk, network.
  • kube-state-metrics translates the state of Kubernetes objects into metrics: how many replicas are desired, how many are ready?
  • The pods prometheus-… and alertmanager-… aren’t created by Helm but by the operator.

The new resource types:

kubectl get crd | grep monitoring.coreos.com

Expected (shortened):

alertmanagerconfigs.monitoring.coreos.com   Namespaced   v1alpha1(storage)   2026-...
alertmanagers.monitoring.coreos.com         Namespaced   v1(storage)         2026-...
podmonitors.monitoring.coreos.com           Namespaced   v1(storage)         2026-...
...
prometheusrules.monitoring.coreos.com       Namespaced   v1(storage)         2026-...
servicemonitors.monitoring.coreos.com       Namespaced   v1(storage)         2026-...

Save your progress? Sign up for free

Drill 4: promtool

You reach the Prometheus web UI at http://<VM-IP>:30090. For the shell there is promtool, the tool from the Prometheus package:

cd ~/obs-lab
curl -fsSLO https://github.com/prometheus/prometheus/releases/download/\
v3.15.1/prometheus-3.15.1.linux-amd64.tar.gz
tar xzf prometheus-3.15.1.linux-amd64.tar.gz
mv prometheus-3.15.1.linux-amd64/promtool ~/.local/bin/
rm -r prometheus-3.15.1.linux-amd64*
promtool --version | head -1

Expected:

promtool, version 3.15.1 (branch: HEAD, revision: bae3eced0a5369e050e83ea...)

The first query: which targets is Prometheus already scraping?

promtool query instant http://localhost:30090 'count by (job) (up)'

Expected (the order changes):

{job="kubelet"} => 3 @[1793866605.06]
{job="kps-alertmanager"} => 2 @[1793866605.06]
{job="kube-state-metrics"} => 1 @[1793866605.06]
{job="kps-grafana"} => 1 @[1793866605.06]
{job="kps-prometheus"} => 2 @[1793866605.06]
{job="coredns"} => 1 @[1793866605.06]
{job="apiserver"} => 1 @[1793866605.06]
{job="node-exporter"} => 1 @[1793866605.06]
{job="kps-operator"} => 1 @[1793866605.06]
  • up is a metric that Prometheus creates itself: 1 if the last scrape worked, otherwise 0. One time series per target.
  • count by (job) counts the targets per job. The job is named after the Service through which Prometheus found the target.
  • kubelet has three targets: the kubelet itself, cAdvisor (container metrics) and the probes. Alertmanager and Prometheus count twice because their helper container config-reloader also delivers metrics.
  • After @ is the time of the answer in Unix seconds.
  • notes-api is still missing. Prometheus doesn’t know about it.

Warning: Does kubelet show six targets instead of three? Then you already installed the chart once under a different name. The operator creates a Service for the kubelet in kube-system, and helm uninstall leaves it behind. kubectl -n kube-system get svc shows it, kubectl delete removes the old one.

Save your progress? Sign up for free

Drill 5: The ServiceMonitor

cat > notes-servicemonitor.yaml <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: notes-api
  namespace: app
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: notes-api
  endpoints:
    - port: http
      path: /metrics
      interval: 15s
EOF
kubectl apply -f notes-servicemonitor.yaml

Expected:

servicemonitor.monitoring.coreos.com/notes-api created
  • selector looks for Services with the label app.kubernetes.io/name: notes-api – in the same namespace as the ServiceMonitor.
  • port: http is the name of the port in the Service. That’s why it got a name on Day 0. A number doesn’t work here.
  • interval: 15s: one scrape per pod every 15 seconds.

The operator needs a few seconds to rewrite the configuration, and Prometheus needs a bit until the first scrape. After about a minute:

promtool query instant http://localhost:30090 'up{job="notes-api"}'

Expected (each time series is on one line, wrapped here):

up{container="notes-api", endpoint="http", instance="10.42.0.43:8080", job="notes-api",
namespace="app", pod="notes-api-57f4f47769-59ks2", service="notes-api"} => 1 @[...]
up{container="notes-api", endpoint="http", instance="10.42.0.42:8080", job="notes-api",
namespace="app", pod="notes-api-57f4f47769-m66nj", service="notes-api"} => 1 @[...]

Two time series, one per pod. Prometheus attached the labels namespace, pod, service and instance itself – from what it found out through Kubernetes.

Save your progress? Sign up for free

Drill 6: The Metrics of notes-api

promtool query instant http://localhost:30090 'notes_created_total'

Expected (shortened, the numbers differ):

notes_created_total{..., pod="notes-api-57f4f47769-59ks2", ...} => 146 @[...]
notes_created_total{..., pod="notes-api-57f4f47769-m66nj", ...} => 148 @[...]

These are the same numbers you read on Day 1 with port-forward – except that Prometheus now fetches and stores them for both pods every 15 seconds.

promtool query instant http://localhost:30090 'sum(notes_created_total)'

Expected:

{} => 294 @[...]

sum adds across all time series. Without by, all labels disappear in the process. That is the sum from Day 1, Drill 6 – without doing the math.

What the configuration made of it is shown in the web UI under Status → Target health. There, notes-api is listed as serviceMonitor/app/notes-api/0 with two endpoints.

Save your progress? Sign up for free

Checkpoint

promtool query instant http://localhost:30090 'count(up{job="notes-api"} == 1)'

Expected:

{} => 2 @[...]

== 1 keeps only the targets whose last scrape worked. Two pods, both reachable.

kubectl -n monitoring get prometheus kps-prometheus

Expected:

NAME             VERSION              DESIRED   READY   RECONCILED   AVAILABLE   AGE
kps-prometheus   v3.15.1-distroless   1         1       True         True        2m26s

Prometheus itself is also a resource of the operator.

Challenge

Scale notes-api to three pods. What do you have to change in the ServiceMonitor so that Prometheus scrapes the third pod?

Hint: kubectl -n app scale deploy/notes-api --replicas 3, wait a minute, then the query from the checkpoint.

Solution

Nothing:

kubectl -n app scale deploy/notes-api --replicas 3

After a minute:

promtool query instant http://localhost:30090 'count(up{job="notes-api"} == 1)'

Expected:

{} => 3 @[...]

The ServiceMonitor points to the Service, not to pods. Prometheus watches its endpoints and notices new pods on its own. That’s called service discovery. Back to two:

kubectl -n app scale deploy/notes-api --replicas 2

Save your progress? Sign up for free

Enterprise Note

  • Storage: In the lab, Prometheus data lives in an emptyDir and is gone after a pod restart. In production, a PersistentVolume belongs under prometheusSpec.storageSpec.
  • High availability: Two Prometheus replicas scrape the same thing, Alertmanager removes duplicate alerts. For long retention and queries across many clusters, Thanos or Mimir come in.
  • Who may scrape: ServiceMonitors in every namespace mean that every team decides for itself what gets scraped. With serviceMonitorNamespaceSelector you limit that to approved namespaces.