Skip to content
Hands-on IT IT Drills

Day 1: The First Pod

Goal: Create the namespace notes, start notes-api as a single Pod, examine it from the outside and the inside – and see why a Pod alone is not enough. · Duration: approx. 60 min · Prerequisite: Day 0

Briefing

  • A Pod is the smallest unit in Kubernetes: one or more containers with a shared IP. Today exactly one container, notes-api:0.1.0.
  • Every object has apiVersion, kind, metadata and usually spec (the desired state). The status (the actual state) is written by the cluster.
  • kubectl apply -f sends a file to the API server – that is the counterpart of docker compose up, just without starting containers on your machine.
  • A namespace separates objects with the same name. From today on, notes is your default.
 k8s/api-pod.yaml ──apply──► API server ──► etcd (desired state stored)
                                 │
                     scheduler picks lab-worker2
                                 │
                     kubelet ──► containerd ──► container notes-api:0.1.0

Drill 1: Namespace and Default Context

source ~/k8s-lab/setup/env.sh
mkdir -p ~/k8s-lab/tag01/k8s && cd ~/k8s-lab/tag01
cp -R "$LAB/code/notes/." .
cat > k8s/namespace.yaml <<'EOF'
# k8s/namespace.yaml – state at end of Day 1
apiVersion: v1
kind: Namespace
metadata:
  name: notes
EOF
kubectl apply --dry-run=server -f k8s/namespace.yaml
kubectl apply -f k8s/namespace.yaml
kubectl config set-context --current --namespace notes
kubectl config get-contexts
kubectl get pods

Expected (abridged):

namespace/notes created (server dry run)
namespace/notes created
Context "kind-lab" modified.
CURRENT   NAME       CLUSTER    AUTHINFO   NAMESPACE
*         kind-lab   kind-lab   kind-lab   notes
No resources found in notes namespace.

--dry-run=server has the API server fully validate the file without storing anything. You do this before every new manifest. The default namespace is stored in the context; without it, every command would need -n notes.

Save your progress? Sign up for free

Drill 2: The Pod Manifest

cat > k8s/api-pod.yaml <<'EOF'
# k8s/api-pod.yaml – state at end of Day 1
apiVersion: v1
kind: Pod
metadata:
  name: api
  labels:
    app.kubernetes.io/name: api
    app.kubernetes.io/part-of: notes
spec:
  containers:
    - name: api
      image: notes-api:0.1.0
      env:
        - name: NOTES_API_KEY
          value: lab-api-key-123
      ports:
        - containerPort: 8080
EOF
kubectl apply --dry-run=server -f k8s/api-pod.yaml

Expected:

pod/api created (server dry run)
Field Meaning Equivalent in Compose
apiVersion, kind Which object type in which API version –
metadata Name, namespace, labels service name
spec.containers Image, environment, ports image, environment
containerPort Documents the port, publishes nothing expose

You use the labels from Day 2 on. The API key in plain text is a stopgap; on Day 4 it becomes a Secret.

Save your progress? Sign up for free

Drill 3: Apply and Inspect

kubectl apply -f k8s/api-pod.yaml
kubectl get pods
sleep 4
kubectl get pods -o wide
kubectl get pod api -o jsonpath='{.status.phase} {.status.podIP} {.spec.nodeName}'; echo

Expected (abridged, names, IPs and times will differ on your machine):

pod/api created
NAME   READY   STATUS              RESTARTS   AGE
api    0/1     ContainerCreating   0          0s
NAME   READY   STATUS    RESTARTS   AGE   IP           NODE          …
api    1/1     Running   0          4s    10.244.2.2   lab-worker2   …
Running 10.244.2.2 lab-worker2

The scheduler picked a worker, and the Pod has its own IP from the Pod network. What the cluster made of your 17 lines is shown by -o yaml:

kubectl get pod api -o yaml | grep -E \
  'imagePullPolicy|restartPolicy|nodeName|dnsPolicy|phase|qosClass|^ +uid: [0-9]+$'

Expected:

    imagePullPolicy: IfNotPresent
  dnsPolicy: ClusterFirst
  nodeName: lab-worker2
  restartPolicy: Always
        uid: 65532
  phase: Running
  qosClass: BestEffort

Without the filter it is more than 100 lines: Kubernetes fills every field you leave out with a default value.

  • imagePullPolicy: IfNotPresent: Because the tag is not latest, the kubelet takes the image loaded with kind load and does not ask any registry.
  • restartPolicy: Always: If the process dies, the kubelet restarts the container in the same Pod.
  • uid: 65532 (under status.containerStatuses): The container runs as the user from the Dockerfile.

Save your progress? Sign up for free

Drill 4: describe and Events

kubectl describe pod api | sed -n '/^Events:/,$p'
kubectl events --for pod/api

Expected (abridged):

Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  9s    default-scheduler  Successfully assigned notes/api to lab-w…
  Normal  Pulled     9s    kubelet            spec.containers{api}: Container image "…
  Normal  Created    8s    kubelet            spec.containers{api}: Container created
  Normal  Started    8s    kubelet            spec.containers{api}: Container started
LAST SEEN   TYPE     REASON      OBJECT    MESSAGE
10s         Normal   Scheduled   Pod/api   Successfully assigned notes/api to lab-work…
10s         Normal   Pulled      Pod/api   Container image "notes-api:0.1.0" already pr…
9s          Normal   Created     Pod/api   Container created
9s          Normal   Started     Pod/api   Container started

Events are the cluster’s log: who did what to the object and when. If a Pod is stuck in Pending or ContainerCreating, the reason is almost always here. Events expire after one hour.

Save your progress? Sign up for free

Drill 5: Logs and exec

kubectl logs api
kubectl exec api -- sh

Expected (abridged):

time=… level=INFO msg="api key loaded" source=env
time=… level=INFO msg="store ready" store=memory
time=… level=INFO msg=listening addr=:8080 version=0.1.0 pid=1
error: Internal error occurred: … OCI runtime exec failed: exec failed: unable to start
container process: exec: "sh": executable file not found in $PATH

kubectl logs reads the container’s stdout and stderr, like docker logs. There is no shell: notes-api is a scratch image with exactly one file. Programs that are in the image can still be started with exec:

kubectl exec api -- /notes-api healthcheck; echo "Exit: $?"

Expected:

Exit: 0

How you can still look inside such a container without bloating the image is shown by kubectl debug on Day 11.

Save your progress? Sign up for free

Drill 6: Port Forwarding

The Pod has an IP in the cluster network; you can’t reach it from the Mac. kubectl port-forward tunnels a local port through the API server into the Pod:

kubectl port-forward pod/api 8080:8080 &
sleep 2
curl -s localhost:8080/version
curl -s localhost:8080/notes
curl -s -X POST localhost:8080/notes -H 'X-API-Key: lab-api-key-123' \
  -H 'Content-Type: application/json' -d '{"text":"Tag 1 lebt im Pod"}'
curl -s localhost:8080/notes
curl -s -X POST localhost:8080/notes -d '{"text":"ohne Key"}'
kill %1

Expected (abridged, IDs differ):

Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080
Handling connection for 8080
0.1.0
[]
{"id":"d847b492-…","text":"Tag 1 lebt im Pod"}
[{"id":"d847b492-…","text":"Tag 1 lebt im Pod"}]
{"error":"invalid api key"}

Port forwarding is a tool for you, not a path for users: it ends with the process and is tied to exactly one Pod. You build the proper path on Day 3.

Save your progress? Sign up for free

Drill 7: kubectl explain

The cluster itself knows which fields exist and what they mean:

kubectl explain pod.spec.containers.imagePullPolicy | head -12
kubectl api-resources | grep -E '^NAME|^pods |^namespaces |^deployments |^services '

Expected (abridged):

KIND:       Pod
VERSION:    v1

FIELD: imagePullPolicy <string>
ENUM:
    Always
    IfNotPresent
    Never

DESCRIPTION:
    Image pull policy. One of Always, Never, IfNotPresent. Defaults to Always if
    :latest tag is specified, or IfNotPresent otherwise. Cannot be updated. More
NAME          SHORTNAMES   APIVERSION   NAMESPACED   KIND
namespaces    ns           v1           false        Namespace
pods          po           v1           true         Pod
services      svc          v1           true         Service
deployments   deploy       apps/v1      true         Deployment

explain works its way through the structure with dots (pod.spec.containers.env), and --recursive shows the whole tree. api-resources provides the short names: kubectl get po is the same as kubectl get pods.

Save your progress? Sign up for free

Drill 8: Pod Gone, Note Gone

kubectl delete pod api
kubectl get pods

Expected:

pod "api" deleted from notes namespace
No resources found in notes namespace.

restartPolicy: Always doesn’t help here: it restarts containers in a Pod. If the Pod itself is gone – deleted, or its node failed – nobody replaces it. So create it again:

kubectl apply -f k8s/api-pod.yaml
kubectl wait --for=condition=Ready pod/api --timeout=60s
kubectl port-forward pod/api 8080:8080 &
sleep 2
curl -s localhost:8080/notes
kill %1

Expected (abridged):

pod/api created
pod/api condition met
Forwarding from 127.0.0.1:8080 -> 8080
…
[]

The new Pod may run on a different node, has a new IP – and your note is gone, because notes-api works in memory without a database. The first problem is solved tomorrow by a Deployment, the second by Postgres on Day 7.

Save your progress? Sign up for free

Checkpoint

kubectl config view --minify | grep namespace

Expected: notes is the default namespace in the context.

    namespace: notes
kubectl get pods --show-labels

Expected: exactly one Pod api, Running, with both labels.

NAME   READY   STATUS    RESTARTS   AGE   LABELS
api    1/1     Running   0          2s    app.kubernetes.io/name=api,app.kubernetes.io/…
kubectl logs api

Expected: version 0.1.0, store memory.

time=… level=INFO msg="api key loaded" source=env
time=… level=INFO msg="store ready" store=memory
time=… level=INFO msg=listening addr=:8080 version=0.1.0 pid=1

Challenge

Start a second Pod api-test with the image notes-api:9.9.9 (which doesn’t exist), use built-in tools to find out why it doesn’t start, and fix it without deleting it. Clean up afterwards.

Hint: kubectl run creates a Pod without a file; a container’s image is one of the few Pod fields that can be changed afterwards (kubectl set image).

Solution

kubectl run api-test --image=notes-api:9.9.9 --env NOTES_API_KEY=lab-api-key-123
sleep 15
kubectl get pod api-test
kubectl events --for pod/api-test
kubectl set image pod/api-test api-test=notes-api:0.1.0
kubectl wait --for=condition=Ready pod/api-test --timeout=90s
kubectl get pods
kubectl delete pod api-test

Expected (abridged):

pod/api-test created
NAME       READY   STATUS             RESTARTS   AGE
api-test   0/1     ImagePullBackOff   0          15s
LAST SEEN          TYPE      REASON      OBJECT         MESSAGE
16s                Normal    Scheduled   Pod/api-test   Successfully assigned notes/…
14s                Normal    BackOff     Pod/api-test   Back-off pulling image "notes-…
2s (x2 over 15s)   Normal    Pulling     Pod/api-test   Pulling image "notes-api:9.9.9"
1s (x2 over 14s)   Warning   Failed      Pod/api-test   Failed to pull image "notes-api…
…: pull access denied, repository does not exist or may require authorization: …
1s (x2 over 14s)   Warning   Failed      Pod/api-test   Error: ErrImagePull
pod/api-test image updated
pod/api-test condition met
NAME       READY   STATUS    RESTARTS   AGE
api        1/1     Running   0          19s
api-test   1/1     Running   0          17s
pod "api-test" deleted from notes namespace

The tag 9.9.9 doesn’t exist on any node, so the kubelet tries to pull docker.io/library/notes-api and fails. After every failure it waits longer (ImagePullBackOff). kubectl run names the container like the Pod – hence api-test=… in set image.

Save your progress? Sign up for free

Enterprise Note

  • Bare Pods practically don’t exist in production. Everything runs through controllers such as Deployment, StatefulSet or Job, which replace Pods (from Day 2 on).
  • An API key as value: ends up in plain text in Git and in etcd. That is what Secrets (Day 4) and external secret stores (“OpenBao secrets hands-on”) are for.
  • kubectl exec and port-forward are powerful permissions. In operations they are restricted by RBAC to a few people and recorded in the audit log.
  • ImagePullBackOff in production usually means: a typo in the tag, a missing pull secret for the private registry or a Docker Hub rate limit.