Skip to content
Hands-on IT IT Drills

Chapter 0: Building your lab

In this chapter you will learn:

  • which building blocks your practice lab has and how they work together
  • how to install all the tools for this book and check their versions
  • how to run two local registries as a “build zone” and a “target zone”
  • how to configure a kind cluster so that it pulls images from these registries
  • how to build the sample application, push it to the registry and start it in the cluster

Before we talk about secure supply chains, you need a workbench. A carpenter does not learn the trade from a catalog but at the workbench: tools within reach, material on the table, a test piece to practice on. That is exactly what you build in this chapter.

The lab is deliberately small. Everything runs on your machine in Docker. There are no cloud accounts, no licenses and no costs. Still, it reproduces the most important parts of a real supply chain: source code, build, registry, zone boundary and a Kubernetes cluster as the target.

At the end of this chapter, the sample application demo-app runs in your cluster and answers curl. All later chapters build on it. So take your time. A clean lab saves you many hours of troubleshooting later.

Overview of the lab

Picture the lab as a small company with two buildings. In the first building (build zone), things are built. In the second building (target zone), things go into production. Between the two is a gate. Only what has been checked there may cross over. In Chapter 4 we build this gate out properly. Today we only set up the two warehouses: one registry per zone (the name reg-ziel uses Ziel, German for “target”).

 Your machine (macOS or Linux, Docker)
 +------------------------------------------------------------------+
 |  CLI tools: docker kind kubectl helm git jq yq cosign syft       |
 |  grype trivy checkov semgrep gitleaks skopeo oras crane vault    |
 |  kyverno conftest opa argocd                                     |
 |                                                                  |
 |  ~/seclab/demo-app  --docker build/push-->  localhost:5001       |
 |                                                                  |
 |  Docker network "kind"                                           |
 |  +------------------------------------------------------------+  |
 |  |  ZONE BUILD/INTERNAL         |  ZONE TARGET/PRODUCTION     |  |
 |  |  +-----------------------+   |   +----------------------+  |  |
 |  |  | reg-build (registry:2)|   |   | reg-ziel (registry:2)|  |  |
 |  |  | Host: localhost:5001  |---+-->| Host: localhost:5002 |  |  |
 |  |  | Net:  reg-build:5000  | Transfer | Net:  reg-ziel:5000  |  |  |
 |  |  +-----------------------+  (Ch. 4)  +--------------------+  |  |
 |  |              \                           /                 |  |
 |  |               \------- pull (http) -----/                  |  |
 |  |  +------------------------------------------------------+  |  |
 |  |  | kind cluster "seclab" (node: seclab-control-plane)   |  |  |
 |  |  | containerd: certs.d/reg-build:5000 + reg-ziel:5000   |  |  |
 |  |  | Namespace demo: Deployment + Service demo-app        |  |  |
 |  |  +------------------------------------------------------+  |  |
 |  +------------------------------------------------------------+  |
 +------------------------------------------------------------------+

Memorize the most important names right away. They come up again in every chapter:

Building block From the host From the cluster Purpose
reg-build localhost:5001 reg-build:5000 built images
reg-ziel localhost:5002 reg-ziel:5000 released images
kind cluster context kind-seclab – target platform
demo-app ~/seclab/demo-app namespace demo sample application

Remember: The same registry has two addresses. Your machine reaches it through the published port (localhost:5001). The cluster reaches it through the container name in the Docker network (reg-build:5000). The image is the same, only the path to it differs.

What hardware you need

  • Docker Desktop, Colima or Docker Engine (Linux)
  • at least 8 GB RAM for Docker, 16 GB recommended (GitLab in Chapter 1 is hungry)
  • about 40 GB of free disk space
  • internet access for the installation (later we also simulate offline scenarios)

Installing the tools

We install everything at once. That way you do not have to stop in the middle of an exercise later. Don’t worry: you don’t need to know yet what each tool does. The table gives you a first impression.

Tip: What am I actually installing here? Appendix D: Tool Guide describes each tool in a line or two, shows typical commands and explains where tools such as grype and trivy overlap. Skim it now, and look things up later when a tool shows up for the first time.

Tool Used for Chapter
docker, kind, kubectl, helm containers, clusters, packages all
git, jq, yq, curl, openssl basic craft all
gitlab-runner (container), argocd CI/CD and GitOps 1
skopeo, crane, oras copying and inspecting images 3, 4
cosign signing and verifying 5
vault secrets management 6
syft, grype SBOM and vulnerabilities 7
trivy, checkov, semgrep, gitleaks scanners 2, 8
kyverno, conftest, opa policy as code 9

Container runtime on macOS

You have two options. Docker Desktop is convenient but requires a license in larger companies. Colima is a lean, free alternative.

Option A, Docker Desktop:

brew install --cask docker
open -a Docker

Then, under “Resources” in the settings, assign at least 8 GB RAM and 4 CPUs.

Option B, Colima:

brew install colima docker docker-compose docker-buildx
colima start --cpu 4 --memory 12 --disk 80

Tip: With Colima you have to register the Compose plugin once. Add the entry "cliPluginsExtraDirs": ["/opt/homebrew/lib/docker/cli-plugins"] to ~/.docker/config.json. On Intel Macs the path is /usr/local/lib/docker/cli-plugins.

Container runtime on Linux

Install Docker Engine following the official instructions for your distribution (docs.docker.com, section “Install Docker Engine”). The Compose plugin is already included there. Then add your user to the docker group and log in again:

sudo usermod -aG docker "$USER"

Security: Being a member of the docker group effectively means: root on this machine. Anyone allowed to use the Docker socket can mount any directory of the host into a container. For a lab that is fine. For build servers, we make a note of this point for Chapter 1.

CLI tools with Homebrew (macOS and Linux)

Homebrew also runs on Linux. If you use it there, you can use exactly the same commands. This is the easiest way to get the same versions on both systems. The three lines correspond to the three groups in Appendix D: basic craft, checking and signing, registries and policies.

brew install kind kubectl helm git jq yq curl openssl
brew install cosign syft grype trivy checkov semgrep gitleaks
brew install skopeo oras crane kyverno conftest opa argocd

Because of its license (BSL), Vault is no longer in the main Homebrew catalog. HashiCorp maintains its own “tap”:

brew tap hashicorp/tap
brew install hashicorp/tap/vault

If you prefer the free fork OpenBao, also install:

brew install openbao

Remember: In this book, yq always means the Go tool by Mike Farah (Homebrew package yq). There is a Python tool with the same name and a different syntax. Check with yq --version that “mikefarah” appears in the output.

CLI tools on Linux without Homebrew

If you don’t want to use Homebrew, use the official installation methods. Distribution package repositories are often outdated. Especially for security tools, you want current versions.

Tool Official method
kind binary from kind.sigs.k8s.io (“Quick Start”)
kubectl binary from dl.k8s.io (instructions on kubernetes.io)
helm script get-helm-3 or package repository (helm.sh)
trivy, syft, grype install.sh from the respective GitHub repo
cosign, crane, oras release binary from GitHub
kyverno, conftest, opa release binary from GitHub
argocd release binary from GitHub (argoproj/argo-cd)
vault HashiCorp package repository (apt/dnf)
checkov, semgrep pipx install checkov / pipx install semgrep
skopeo, jq, git, openssl the distribution’s package manager

Example: kubectl on Linux (amd64), as described in the Kubernetes docs:

STABLE=$(curl -L -s https://dl.k8s.io/release/stable.txt)
curl -LO "https://dl.k8s.io/release/${STABLE}/bin/linux/amd64/kubectl"
curl -LO "https://dl.k8s.io/release/${STABLE}/bin/linux/amd64/kubectl.sha256"
echo "$(cat kubectl.sha256)  kubectl" | sha256sum --check
sudo install -m 0755 kubectl /usr/local/bin/kubectl

Example: Trivy via the installation script:

curl -sfL \
  https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh \
  | sudo sh -s -- -b /usr/local/bin

Security: curl … | sh runs third-party code without checking it. That is convenient, but it is exactly the kind of trust we question in this book. Better: download the script first, read it, then run it. Even better: download the release binary and compare the checksum against the published checksums.txt, as with kubectl above.

gitlab-runner

We do not install the GitLab Runner on the host. In Chapter 1 it runs as a container (gitlab/gitlab-runner). You don’t need to do anything now.

Checking versions

Now you check whether everything is there. Create a small check script for this. You will need it again, for example after an operating system update.

mkdir -p ~/seclab
cat > ~/seclab/check-tools.sh <<'EOF'
#!/usr/bin/env bash
# Checks whether all lab tools are in the PATH.
set -u
tools="docker kind kubectl helm git jq yq curl openssl cosign syft grype
trivy checkov semgrep gitleaks skopeo oras crane vault kyverno conftest
opa argocd"
missing=0
for t in $tools; do
  if command -v "$t" >/dev/null 2>&1; then
    printf "OK      %s\n" "$t"
  else
    printf "FEHLT   %s\n" "$t"
    missing=$((missing + 1))
  fi
done
docker compose version >/dev/null 2>&1 \
  && echo "OK      docker compose" \
  || { echo "FEHLT   docker compose"; missing=$((missing + 1)); }
echo "Fehlend: $missing"
exit "$missing"
EOF
chmod +x ~/seclab/check-tools.sh
~/seclab/check-tools.sh

Expected output (shortened; FEHLT and Fehlend both mean “missing”):

OK      docker
OK      kind
OK      kubectl
...
OK      argocd
OK      docker compose
Fehlend: 0

Then take a look at the versions. Write them down. If a command later behaves differently than in the book, the version is the first suspect.

docker version --format 'Docker Server: {{.Server.Version}}'
docker compose version
kind version
kubectl version --client
helm version --short
cosign version
syft version
grype version
trivy --version
checkov --version
semgrep --version
gitleaks version
skopeo --version
oras version
crane version
vault version
kyverno version
conftest --version
opa version
argocd version --client
yq --version

What to look out for:

Tool Expectation
cosign v2.x (with v3, note the hints in Chapter 5)
kubectl at most one minor version away from the cluster
yq output contains “mikefarah”
docker compose v2.x (plugin, not the old docker-compose)

Tip: docker version must show both “Client” and “Server”. If the server part is missing, the Docker daemon is not running. Start Docker Desktop or colima start.

The working directory

All files of the book live under ~/seclab. Each chapter gets its own subdirectory. The sample application lives in ~/seclab/demo-app.

mkdir -p ~/seclab/{demo-app,containerd-certs.d,kap01,kap02,kap03,kap04,kap05}
mkdir -p ~/seclab/{kap06,kap07,kap08,kap09,kap10,kap11}
ls ~/seclab

Expected output:

check-tools.sh  containerd-certs.d  demo-app  kap01  kap02  kap03  kap04
kap05  kap06  kap07  kap08  kap09  kap10  kap11

The two registries

A registry is a warehouse for container images. We use the lean reference image registry:2. It has no UI, no users and no TLS. That’s no good for production. For a lab it is ideal: fast, transparent and reachable with standard tools.

Step 1: Start the registries

docker run -d --restart=always --name reg-build \
  -p 127.0.0.1:5001:5000 registry:2

docker run -d --restart=always --name reg-ziel \
  -p 127.0.0.1:5002:5000 registry:2

We deliberately bind the ports to 127.0.0.1 only. That way the registries are not reachable from your Wi-Fi.

Check:

docker ps --filter name=reg- --format '{{.Names}}\t{{.Ports}}'
curl -s http://localhost:5001/v2/_catalog
curl -s http://localhost:5002/v2/_catalog

Expected output:

reg-ziel   127.0.0.1:5002->5000/tcp
reg-build  127.0.0.1:5001->5000/tcp
{"repositories":[]}
{"repositories":[]}

Why without TLS?

Docker allows unencrypted registries without further configuration only for localhost. That is why docker push localhost:5001/... works right away. The cluster, on the other hand, addresses the registries by their container name. There we have to tell containerd explicitly: “For reg-build:5000, http is fine.” We take care of that in the kind configuration in a moment.

Warning: Unencrypted registries are purely a lab concession. In Chapter 3 you get to know Nexus with a clean configuration. In real environments, every registry belongs behind TLS with a certificate from your own PKI (see Chapter 5).

The kind cluster

kind stands for “Kubernetes in Docker”. Each Kubernetes node is a Docker container. Inside it, containerd runs as the container runtime. containerd pulls the images for your pods. So containerd must know how to reach reg-build:5000 and reg-ziel:5000.

How containerd finds registries

Since containerd 1.5 there is a simple directory scheme for this. In a directory, usually /etc/containerd/certs.d, each registry has a subdirectory with a file hosts.toml. The directory name is exactly the registry name from the image reference.

/etc/containerd/certs.d/
  reg-build:5000/hosts.toml   -> "use http://reg-build:5000"
  reg-ziel:5000/hosts.toml    -> "use http://reg-ziel:5000"

For containerd to read this directory, config_path must be set in its configuration. kind takes care of that via containerdConfigPatches. We create the hosts.toml files on the host and mount them into the node via extraMounts. That way everything is declarative and survives re-creating the cluster.

Step 2: Create the hosts.toml files

for reg in reg-build reg-ziel; do
  mkdir -p ~/seclab/containerd-certs.d/"${reg}:5000"
  cat > ~/seclab/containerd-certs.d/"${reg}:5000"/hosts.toml <<EOF
server = "http://${reg}:5000"

[host."http://${reg}:5000"]
  capabilities = ["pull", "resolve"]
EOF
done
cat ~/seclab/containerd-certs.d/reg-build:5000/hosts.toml

Expected output:

server = "http://reg-build:5000"

[host."http://reg-build:5000"]
  capabilities = ["pull", "resolve"]

The capabilities allow only reading (pull) and resolving tags to digests (resolve). The cluster should not write anything to the registry.

Step 3: Write kind.yaml

kind does not understand a tilde in hostPath. So we let the shell substitute $HOME when writing the file. Make sure the heredoc marker EOF is not in quotes here.

cat > ~/seclab/kind.yaml <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: seclab
containerdConfigPatches:
  - |-
    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = "/etc/containerd/certs.d"
nodes:
  - role: control-plane
    extraMounts:
      - hostPath: ${HOME}/seclab/containerd-certs.d
        containerPath: /etc/containerd/certs.d
        readOnly: true
EOF
cat ~/seclab/kind.yaml

On your machine the finished file looks roughly like this (with your home path in it):

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: seclab
containerdConfigPatches:
  - |-
    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = "/etc/containerd/certs.d"
nodes:
  - role: control-plane
    extraMounts:
      - hostPath: /Users/thomas/seclab/containerd-certs.d
        containerPath: /etc/containerd/certs.d
        readOnly: true

What the parts mean:

  • name: seclab sets the cluster name. The context is then called kind-seclab.
  • containerdConfigPatches adds config_path to the containerd configuration in the node.
  • extraMounts makes your host directory visible in the node.
  • A single node is enough. It saves RAM that GitLab needs in Chapter 1.

Remember: The official kind docs (“Local Registry”) show the same principle with config_path. There, the hosts.toml files are copied into the node afterwards with docker exec. Our way via extraMounts is equivalent but more reproducible.

Step 4: Create the cluster

kind create cluster --name seclab --config ~/seclab/kind.yaml

Expected output (shortened):

Creating cluster "seclab" ...
 ✓ Ensuring node image (kindest/node:v1.xx.x)
 ✓ Preparing nodes
 ✓ Writing configuration
 ✓ Starting control-plane
 ✓ Installing CNI
 ✓ Installing StorageClass
Set kubectl context to "kind-seclab"

The check marks come from kind itself. Check the cluster:

kubectl cluster-info --context kind-seclab
kubectl get nodes

Expected output:

Kubernetes control plane is running at https://127.0.0.1:xxxxx
...
NAME                   STATUS   ROLES           AGE   VERSION
seclab-control-plane   Ready    control-plane   1m    v1.xx.x

Also check that the registry configuration has arrived in the node:

docker exec seclab-control-plane ls /etc/containerd/certs.d
docker exec seclab-control-plane grep config_path /etc/containerd/config.toml

Expected output:

reg-build:5000
reg-ziel:5000
    config_path = "/etc/containerd/certs.d"

The Docker network kind

When creating the cluster, kind set up a Docker network named kind. The node seclab-control-plane is attached to it. In such user-defined networks, Docker runs a built-in DNS service. Containers in the same network find each other by name.

Our registries, however, are still attached to the default network bridge. So the node cannot find them. Let’s change that.

Step 5: Attach the registries to the kind network

docker network connect kind reg-build
docker network connect kind reg-ziel
docker network inspect kind \
  --format '{{range .Containers}}{{.Name}} {{end}}'

Expected output (order may differ):

seclab-control-plane reg-build reg-ziel

If Docker reports endpoint with name reg-build already exists in network kind, the container is already attached to the network. That is not an error.

Remember: The registries are now attached to two networks. Through bridge and the published port, you reach them from the host. Through kind, the cluster node reaches them. In Chapter 1 we also attach GitLab and the CI jobs to this network.

A quick connection test from inside the node:

docker exec seclab-control-plane \
  curl -s http://reg-build:5000/v2/_catalog

Expected output:

{"repositories":[]}

The sample application

Now you need something that travels through the supply chain. The demo-app is a tiny Python web application with Flask. It is deliberately simple. The content is not the topic, the path from source code into the cluster is.

The structure at the end:

~/seclab/demo-app/
  app.py
  requirements.txt
  requirements-dev.txt
  tests/test_app.py
  Dockerfile
  .dockerignore
  .gitignore
  k8s/deployment.yaml
  k8s/service.yaml

Step 6: app.py

cd ~/seclab/demo-app
cat > app.py <<'EOF'
"""Minimale Demo-Anwendung für das Lieferketten-Labor."""
import os

from flask import Flask, jsonify

app = Flask(__name__)
APP_VERSION = os.environ.get("APP_VERSION", "1.0.0")


@app.get("/")
def index():
    """Einfacher Lebenszeichen-Endpunkt."""
    return jsonify(status="ok")


@app.get("/version")
def version():
    """Gibt die ausgelieferte Version zurück."""
    return jsonify(version=APP_VERSION)


if __name__ == "__main__":
    # Local tests only. In the container, gunicorn starts the app.
    app.run(host="127.0.0.1", port=8000)
EOF

Step 7: Dependencies

cat > requirements.txt <<'EOF'
flask==3.0.3
gunicorn==22.0.0
# Deliberately outdated: known vulnerabilities for the scan exercises
# in Chapters 7 and 8. The app does not use requests.
requests==2.25.0
EOF

cat > requirements-dev.txt <<'EOF'
pytest==8.3.3
EOF

Warning: requests==2.25.0 is deliberately a version with known vulnerabilities (CVEs). It is our practice object for SBOM, scanners and thresholds. Never copy this line into a real project.

All versions are pinned (==). That is the first small building block of a secure supply chain: today’s build and tomorrow’s build should use the same packages.

Step 8: A test

Chapter 1 runs this test in the pipeline.

mkdir -p tests
cat > tests/test_app.py <<'EOF'
from app import app


def test_index_returns_ok():
    client = app.test_client()
    resp = client.get("/")
    assert resp.status_code == 200
    assert resp.get_json() == {"status": "ok"}


def test_version_has_value():
    client = app.test_client()
    resp = client.get("/version")
    assert resp.status_code == 200
    assert "version" in resp.get_json()
EOF

Running it locally is optional, but it makes a good quick test:

python3 -m venv .venv
.venv/bin/pip install -q -r requirements.txt -r requirements-dev.txt
.venv/bin/python -m pytest -q

Expected output:

..                                                    [100%]
2 passed in 0.05s

Step 9: Dockerfile

The Dockerfile has two stages (multi-stage). The first stage installs the packages into a virtual environment. The second stage takes over only the result. Tools that are only needed for building therefore don’t end up in the finished image. In addition, the app does not run as root but as a user with UID 10001.

cat > Dockerfile <<'EOF'
# ---- Stage 1: install dependencies ----
FROM python:3.12-slim AS build
WORKDIR /app
COPY requirements.txt .
RUN python -m venv /opt/venv \
 && /opt/venv/bin/pip install --no-cache-dir -r requirements.txt

# ---- Stage 2: lean runtime image ----
FROM python:3.12-slim
ENV PATH="/opt/venv/bin:$PATH" \
    PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1
RUN groupadd --system --gid 10001 app \
 && useradd --system --uid 10001 --gid 10001 --no-create-home app
WORKDIR /app
COPY --from=build /opt/venv /opt/venv
COPY app.py .
USER 10001:10001
EXPOSE 8000
CMD ["gunicorn", "--bind", "0.0.0.0:8000", "--workers", "2", "app:app"]
EOF

What the most important lines do:

Line Effect
FROM ... AS build named build stage
COPY --from=build take over only the finished venv
USER 10001:10001 process runs without root
PYTHONDONTWRITEBYTECODE=1 no .pyc files, fits read-only

Tip: Here we still use python:3.12-slim by tag. A tag is a movable pointer. In Chapter 2 you learn to pin base images by digest, and in Chapter 3 they come only from your mirror.

In addition, a .dockerignore so that no local leftovers end up in the image, and a .gitignore:

cat > .dockerignore <<'EOF'
.git
.venv
__pycache__
*.pyc
tests
k8s
EOF

cat > .gitignore <<'EOF'
.venv/
__pycache__/
*.pyc
.pytest_cache/
EOF

Step 10: Kubernetes manifests

mkdir -p k8s
cat > k8s/deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-app
  namespace: demo
  labels:
    app: demo-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: demo-app
  template:
    metadata:
      labels:
        app: demo-app
    spec:
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        runAsGroup: 10001
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: demo-app
          image: reg-build:5000/demo-app:1.0.0
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8000
              name: http
          env:
            - name: APP_VERSION
              value: "1.0.0"
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
            limits:
              memory: 256Mi
          readinessProbe:
            httpGet:
              path: /
              port: http
            initialDelaySeconds: 3
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /
              port: http
            initialDelaySeconds: 10
            periodSeconds: 20
          volumeMounts:
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: tmp
          emptyDir: {}
EOF

cat > k8s/service.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: demo-app
  namespace: demo
  labels:
    app: demo-app
spec:
  type: ClusterIP
  selector:
    app: demo-app
  ports:
    - name: http
      port: 80
      targetPort: http
EOF

A few details deserve an explanation:

  • readOnlyRootFilesystem: The container may not modify its file system. That way an attacker cannot download and drop a tool.
  • emptyDir under /tmp: gunicorn needs a writable temp directory for its workers. Without this mount, the pod fails to start.
  • capabilities drop ALL: The process gets no special Linux privileges.
  • automountServiceAccountToken: false: The app does not need access to the Kubernetes API, so it does not get a token either.
  • image: This contains the cluster address reg-build:5000, not localhost:5001. From the node’s point of view, localhost would be the node itself.

Remember: These security settings are not optional extras. In Chapter 9 you write Kyverno rules that enforce exactly these fields. The demo-app then passes these rules from the start.

Step 11: Create the Git repository

Chapter 1 pushes the app to GitLab. Create the repository now:

cd ~/seclab/demo-app
git init -b main
git add .
git commit -m "demo-app: erste Version 1.0.0"
git log --oneline

Expected output:

a1b2c3d demo-app: erste Version 1.0.0

Thanks to .gitignore, the .venv does not show up in the commit. Check that with git status --ignored.

Building and pushing the image

Step 12: Build

cd ~/seclab/demo-app
docker build -t localhost:5001/demo-app:1.0.0 .
docker image ls localhost:5001/demo-app

Expected output (size varies):

REPOSITORY                TAG     IMAGE ID       CREATED          SIZE
localhost:5001/demo-app   1.0.0   3f2c...        10 seconds ago   ~150MB

Tip: On a Mac with Apple silicon, Docker builds an arm64 image. The kind node runs on the same architecture, so that fits. Problems only arise when you bring the image to an amd64 server. Then docker buildx build --platform linux/amd64 helps.

Step 13: Try it locally

Before the image goes into the cluster, a quick test directly with Docker:

docker run --rm -d --name demo-test -p 127.0.0.1:8000:8000 \
  localhost:5001/demo-app:1.0.0
sleep 2
curl -s http://localhost:8000/
curl -s http://localhost:8000/version
docker exec demo-test id
docker stop demo-test

Expected output:

{"status":"ok"}
{"version":"1.0.0"}
uid=10001(app) gid=10001(app) groups=10001(app)
demo-test

The line with uid=10001 proves it: the process does not run as root.

Step 14: Push and determine the digest

docker push localhost:5001/demo-app:1.0.0
curl -s http://localhost:5001/v2/_catalog
crane ls localhost:5001/demo-app
crane digest localhost:5001/demo-app:1.0.0

Expected output:

The push refers to repository [localhost:5001/demo-app]
...
1.0.0: digest: sha256:9c1e... size: 1789
{"repositories":["demo-app"]}
1.0.0
sha256:9c1e...

The digest is the fingerprint of the image. A tag like 1.0.0 can point to a different image tomorrow. The digest cannot. That is why, from Chapter 4 on, we work almost exclusively with image@sha256:….

Remember: crane automatically talks to localhost over http. For other registry names you would need --insecure. That is a quirk of the go-containerregistry library that crane is built on.

Step 15: Test the pull from the cluster

Now the decisive test: does containerd in the node find the image under its cluster name?

docker exec seclab-control-plane \
  crictl pull reg-build:5000/demo-app:1.0.0

Expected output:

Image is up to date for sha256:4e7a...

The number after sha256: here is the image ID, not the manifest digest. Both are hashes, but over different things. That is normal.

Test the target registry right away too. We copy the image over with crane copy. This is a foretaste of the transfer in Chapter 4.

crane copy localhost:5001/demo-app:1.0.0 localhost:5002/demo-app:1.0.0
crane digest localhost:5002/demo-app:1.0.0
docker exec seclab-control-plane \
  crictl pull reg-ziel:5000/demo-app:1.0.0

The digest in reg-ziel is identical to the one in reg-build. crane copy copies byte for byte. This matters: a signature that points to the digest therefore stays valid across the zone boundary.

Deploying and testing in the cluster

Step 16: Apply the namespace and manifests

kubectl create namespace demo
kubectl apply -f ~/seclab/demo-app/k8s/
kubectl -n demo rollout status deployment/demo-app --timeout=120s
kubectl -n demo get pods,svc

Expected output:

namespace/demo created
deployment.apps/demo-app created
service/demo-app created
deployment "demo-app" successfully rolled out
NAME                            READY   STATUS    RESTARTS   AGE
pod/demo-app-6d9f8b7c5d-x2k4q   1/1     Running   0          20s

NAME               TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/demo-app   ClusterIP   10.96.123.45   <none>        80/TCP    20s

Step 17: Test with port-forward and curl

The service has no external address. For tests, you forward a local port to it. The command blocks the terminal, so it runs in the background.

kubectl -n demo port-forward svc/demo-app 8000:80 >/dev/null 2>&1 &
PF_PID=$!
sleep 2
curl -s http://localhost:8000/
curl -s http://localhost:8000/version
kill "$PF_PID"

Expected output:

{"status":"ok"}
{"version":"1.0.0"}

Congratulations: the supply chain works once from start to finish. Source code, build, registry, cluster. It is still completely unprotected. We change that chapter by chapter.

Where did the image really come from? The pod shows you:

kubectl -n demo get pod -l app=demo-app \
  -o jsonpath='{.items[0].status.containerStatuses[0].imageID}{"\n"}'

Expected output:

reg-build:5000/demo-app@sha256:9c1e...

That is the same digest that crane digest reported.

Pausing and restarting the lab

The lab does not need to run all the time. This saves RAM and battery:

# Pause
docker stop seclab-control-plane

# Start again
docker start seclab-control-plane
kubectl get nodes

Thanks to --restart=always, the registries start automatically with Docker. If the cluster does not come up cleanly after a restart, delete it and create it again. Thanks to kind.yaml, that only takes a minute:

kind delete cluster --name seclab
kind create cluster --name seclab --config ~/seclab/kind.yaml
docker network connect kind reg-build
docker network connect kind reg-ziel
kubectl create namespace demo
kubectl apply -f ~/seclab/demo-app/k8s/

Tip: Usually the registries stay connected to the kind network. The two docker network connect commands are in the list anyway, because they do no harm: if the connection already exists, Docker merely reports “already exists”. The images in the registries are preserved in any case.

Check your lab

Tick off every item before you move on to Chapter 1. Each row has a check command.

Check Command Expectation
Tools complete ~/seclab/check-tools.sh Fehlend: 0
Registries running docker ps --filter name=reg- two containers
Image in reg-build crane ls localhost:5001/demo-app 1.0.0
Context correct kubectl config current-context kind-seclab
Node ready kubectl get nodes Ready
Network connected docker network inspect kind reg-build, reg-ziel
Cluster pull works crictl pull in the node Image is up to date
App running kubectl -n demo get pods 1/1 Running
App responds port-forward + curl {"status":"ok"}
Git repo exists git -C ~/seclab/demo-app log one commit

When everything is green, you can record this state as a reference:

crane digest localhost:5001/demo-app:1.0.0 > ~/seclab/demo-app-1.0.0.digest
cat ~/seclab/demo-app-1.0.0.digest

Common pitfalls

Symptom Cause Solution
Cannot connect to the Docker daemon Docker is not running Docker Desktop or colima start
port is already allocated on start another service uses 5001/5002 check with lsof -i :5001
Pod ErrImagePull, “http: server gave HTTP response to HTTPS client” hosts.toml missing or config_path not set check steps 2–4, re-create the cluster
Pod ErrImagePull, “no such host” registry not in the kind network docker network connect kind reg-build
Pod ErrImagePull, “not found” image not pushed or typo crane ls localhost:5001/demo-app
kind create reports a mount error hostPath contains ~ or directory missing regenerate kind.yaml with $HOME
Pod CrashLoopBackOff, gunicorn error about tmp /tmp not writable mount emptyDir under /tmp
Pod CreateContainerConfigError runAsNonRoot and image without a numeric user set USER 10001:10001 numerically
curl on port 8000 hangs port-forward not active check the process with jobs, restart
yq commands fail Python yq installed install the Go variant (mikefarah)
Pull on an amd64 server: “exec format error” image built for arm64 build with --platform linux/amd64

Tip: For pull errors, kubectl -n demo describe pod -l app=demo-app is your best friend. At the very bottom, under “Events”, you find the exact error message from containerd.

Summary

What you learned:

  • Your lab consists of two registries (reg-build on port 5001, reg-ziel on port 5002), a kind cluster seclab and the sample application demo-app.
  • The same registry has two addresses: localhost:5001 from the host and reg-build:5000 from the cluster. The Docker network kind with its built-in DNS makes the second address possible.
  • containerd finds registries via hosts.toml files in /etc/containerd/certs.d. kind sets config_path for this via containerdConfigPatches, and the files get into the node via extraMounts.
  • A multi-stage Dockerfile with a fixed, unprivileged user (UID 10001) is the foundation for a lean, secure image.
  • Security settings such as readOnlyRootFilesystem, drop: ALL and runAsNonRoot belong in the deployment from the start.
  • A tag is a movable pointer, a digest a fixed fingerprint. crane digest shows it to you, and crane copy preserves it when copying.
  • With ~/seclab/check-tools.sh and the checklist, you can check at any time whether your lab is intact.

In the next chapter, you replace manual operation with a real CI/CD chain: GitLab builds the image, Argo CD brings it into the cluster.