Skip to content
Hands-on IT IT Drills

Appendix A: Cheat Sheet

All 34 tools on a few pages: installation and the commands you practiced in the book. The number in parentheses is the day. Variables like $REG are set by source ~/ct-lab/setup/env.sh; the registry is registry.ct.lab:5000, login lab/lab-registry-pw-123. When in doubt, the project’s docs or --help on the VM take precedence.

Installation at a glance

Tool Version in this book Source Day
podman, Quadlet 5.7.0 Ubuntu package podman 1–3
podman-compose 1.5.0 Ubuntu package podman-compose 4
lazydocker, ctop 0.25.2, 0.7.7 (outdated) Release binary 4
buildah 1.42.1 Ubuntu package buildah 5
ko 0.19.1 Release binary, plus golang-go 5
ctr, nerdctl, buildctl v2.4.2, 2.4.2, v0.34.1 nerdctl-full bundle 6
buildctl, Lima (Mac) 0.34.1, 2.2.1 Homebrew, VM via template://buildkit 6
crane, krane 0.22.1 Release binary (go-containerregistry) 7, 10
skopeo 1.21.0 Ubuntu package skopeo 8
regctl, regsync v0.11.7 Release binary (regclient) 8, 9
oras 1.3.6 Release binary 9
kind, crictl v0.33.0, v1.37.0 Release binary 10
dive, diffoci 0.13.1, v0.1.8 Release binary 11
container-diff v0.19.0 (archived) Self-built (go mod download) 11
hadolint, dockle 2.15.1, 0.4.15 Release binary 12
ShellCheck 0.11.0 Ubuntu package shellcheck 12
trivy, grype 0.76.1, 0.121.0 Release binary 13
docker scout v1.29.0, Docker CE 29.9.2 Docker apt repository + CLI plugin 13
syft, cosign 1.55.0, v3.1.4 Release binary 14
docker, Buildx, Compose 29.9.2, v0.38.1, v5.6.1 Docker apt repository (plugins) 13, 15
Colima (Mac), Lima v0.10.3, 2.2.1 Homebrew 15
Scaffdog 4.1.0 npm, via npx in node:26.13.0-alpine 15

Every release binary in four steps – download, verify checksum, unpack, install (0). URL and checksum are in setup/tools.lock:

sudo apt-get install -y podman buildah skopeo jq apache2-utils     # packages (0)
arch=$(dpkg --print-architecture)           # one binary by hand (0, 7, 8)
read -r _ _ url sum < <(grep "^regctl $arch " "$LAB/setup/tools.lock")
curl -fsSLo /tmp/regctl "$url"
echo "$sum  /tmp/regctl" | sha256sum -c -
sudo install -m 0755 /tmp/regctl /usr/local/bin/regctl && rm /tmp/regctl
sha256sum -c --ignore-missing checksums.txt # against the project's file (0, 4)

Podman and systemd (Days 1–4)

podman

podman pull docker.io/library/nginx:1.30.6-alpine   # always fully qualified (1)
podman run -d --name web -p 127.0.0.1:8080:80 docker.io/library/nginx:1.30.6-alpine
podman ps --format '{{.Names}}\t{{.Status}}\t{{.Ports}}'
podman logs --tail 1 web
podman top web user huser pid hpid args         # UID in the container and on the host (1)
podman build --format docker -t localhost/notes-api:0.1.0 \
  --build-arg VERSION=0.1.0 notes-api           # Docker format keeps HEALTHCHECK (1)
podman image inspect localhost/notes-api:0.1.0 \
  --format '{{.Config.Healthcheck.Test}} {{.ManifestType}}'
podman healthcheck run notes-api && echo "gesund"
podman save localhost/notes-api:0.1.0 | sudo podman load      # rootless → rootful (1)
podman info --format '{{.Store.GraphRoot}}'
podman image prune -f                           # only the user's unnamed images (1)

Rootless in detail, pods and Kubernetes YAML (2):

podman unshare cat /proc/self/uid_map           # 0 → 1000, 1 → 100000 … (2)
podman run -d --name pg-bind --user 70:70 -e POSTGRES_PASSWORD=lab-db-pw-123 \
  -v ./pgdata-bind:/var/lib/postgresql:U docker.io/library/postgres:18.6-alpine
podman unshare rm -rf pgdata-bind               # delete files with foreign UIDs (2)
podman pod create --name notes -p 127.0.0.1:8080:8080 -p 127.0.0.1:3000:3000
podman run -d --pod notes --name notes-db … docker.io/library/postgres:18.6-alpine
podman kube generate notes -f notes-pod.yaml
podman kube play --replace notes-pod.yaml
podman kube down notes-pod.yaml                 # volumes remain (without --force)

Observing (4):

podman stats --no-stream --format 'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}'
podman pod stats --no-stream --format '{{.Pod}} {{.Name}} {{.MemUsage}}'
podman events --since 30s --until 1s --filter container=notes-api \
  --format '{{.Status}} {{.Name}}'

Quadlet

Units in ~/.config/containers/systemd/, daemon-reload after every change (3):

# ~/.config/containers/systemd/notes-api.container – template (3)
[Unit]
Requires=notes-migrate.container
After=notes-migrate.container

[Container]
ContainerName=notes-api
Image=localhost/notes-api:0.1.0
Pod=notes.pod
Environment=NOTES_API_KEY=lab-api-key-123
Notify=healthy
AutoUpdate=local

[Service]
Restart=on-failure
/usr/libexec/podman/quadlet -dryrun -user       # generated units, typos (3, 16)
QUADLET_UNIT_DIRS=/tmp/qtest /usr/libexec/podman/quadlet -dryrun -user
systemctl --user daemon-reload
systemctl --user start notes-pod.service        # not enable: the units are generated
systemctl --user list-units 'notes*' --all --no-pager
journalctl --user -u notes-api.service --no-pager -o cat
sudo loginctl enable-linger "$USER"             # services without a login session (3)
podman auto-update --dry-run
systemctl --user enable --now podman-auto-update.timer

Podman socket and podman-compose

systemctl --user enable --now podman.socket     # Docker API 1.41 (4)
S="$XDG_RUNTIME_DIR/podman/podman.sock"
curl -s --unix-socket "$S" http://d/_ping; echo
export DOCKER_HOST="unix://$XDG_RUNTIME_DIR/podman/podman.sock"   # in ~/.bashrc
podman --remote ps --format '{{.Names}}'
sudo apt-get install -y podman-compose          # provider for podman compose (4)
podman compose up -d
podman compose logs worker | tail -1
podman compose down

lazydocker

lazydocker                                      # reads DOCKER_HOST (4)
lazydocker -c | grep -E '^commandTemplates:|dockerCompose:'   # default values

In ~/.config/lazydocker/config.yml: commandTemplates: → dockerCompose: podman compose. Keys: 1…6 panels, [ ] tabs (logs, stats …), x menu, q quit.

ctop

ctop -v && ctop                                 # no release since 2022 – outdated (4)

Building (Days 5–6)

buildah

sudo apt-get install -y buildah
buildah build --format docker -t localhost/notes-api:0.2.0 \
  --build-arg VERSION=0.2.0 notes-api           # without --layers: one layer (5)
buildah build -q --format docker --timestamp 0 …   # reproducible image ID (5)
b=$(buildah from --name api-build docker.io/library/golang:1.27-alpine)
buildah copy "$b" notes-api /src
buildah run --workingdir /src -e CGO_ENABLED=0 "$b" -- \
  go build -trimpath -ldflags="-s -w -X main.version=0.2.0" -o /out/notes-api .
f=$(buildah from --format docker --name api-final scratch)
buildah copy --from "$b" --chmod 555 "$f" /out/notes-api /notes-api
buildah config --user 65532:65532 --entrypoint '["/notes-api"]' \
  --healthcheck 'CMD /notes-api healthcheck' "$f"
buildah commit --format docker --rm "$f" localhost/notes-api:0.2.0-buildah
buildah rm "$b" && buildah containers
buildah unshare sh -c '
  m=$(buildah mount api-final); ls -ln "$m"; buildah umount api-final'
buildah push localhost/notes-api:0.2.0 oci:./out:notes-api     # OCI: health check gone
buildah push localhost/notes-api:0.2.0 \
  docker-archive:./out/notes-api.tar:localhost/notes-api:0.2.0

ko

sudo apt-get install -y golang-go
ko build --push=false .                         # without KO_DOCKER_REPO, build only (5)
ko build --local .                              # via DOCKER_HOST to ko.local/…
export VERSION=0.2.0                            # for {{.Env.VERSION}} in .ko.yaml
KO_DOCKER_REPO=ko.local/notes-api ko build --bare -t 0.2.0 --sbom-dir ../out/sbom .
KO_DOCKER_REPO=localhost/notes-api ko build --bare -t 0.2.0-ko --push=false \
  --tarball ../out/ko.tar .
podman load -i ../out/ko.tar
podman run -d --name api-ko --health-cmd '["/ko-app/notes-api", "healthcheck"]' …

ctr

sudo systemctl daemon-reload && sudo systemctl enable --now containerd
sudo ctr ns ls
sudo ctr images pull docker.io/library/nginx:1.30.6-alpine   # no short names
sudo ctr run -d docker.io/library/nginx:1.30.6-alpine web
sudo ctr tasks ls
sudo ctr tasks exec --exec-id e1 web nginx -v
sudo ctr tasks kill web && sudo ctr tasks rm web && sudo ctr containers rm web
podman save localhost/notes-api:0.2.0 | sudo ctr images import -   # Podman → containerd

nerdctl

sudo nerdctl run -d --name web -p 127.0.0.1:8081:80 \
  docker.io/library/nginx:1.30.6-alpine
sudo nerdctl ps && sudo nerdctl images
sudo nerdctl network ls
sudo nerdctl compose -f compose.yaml -f compose.nerdctl.yaml up -d db   # db first (6)
sudo nerdctl compose -f compose.yaml -f compose.nerdctl.yaml up -d
sudo nerdctl build -t localhost/notes-api:0.2.0 --build-arg VERSION=0.2.0 notes-api
sudo systemctl disable --now containerd buildkit

buildctl

sudo systemctl enable --now buildkit
sudo buildctl debug workers -v
sudo buildctl build --frontend dockerfile.v0 \
  --local context=notes-api --local dockerfile=notes-api \
  --opt build-arg:VERSION=0.2.0 \
  --output type=oci,dest=out/notes-api-oci.tar,name=localhost/notes-api:0.2.0-buildctl \
  --export-cache type=local,dest=out/buildcache
sudo buildctl build … --import-cache type=local,src=out/buildcache
sudo buildctl build … --output type=docker,name=localhost/notes-api:0.3.0,dest=- \
  | sudo ctr images import -
sudo buildctl prune --all

Standalone BuildKit with Lima (Mac)

brew install buildkit lima                       # buildctl on the Mac (6)
limactl start --name=buildkit --cpus=2 --memory=2 --disk=20 template://buildkit
export BUILDKIT_HOST="unix://$HOME/.lima/buildkit/sock/buildkitd.sock"   # ~/.zshrc
buildctl debug workers
buildctl build --frontend dockerfile.v0 --local context=. --local dockerfile=. \
  --output type=oci,dest=myapp.tar,name=localhost/myapp:0.1.0
buildctl build … --output "type=image,name=$REG/lab/myapp:0.1.0,push=true"   # from 7 on
limactl stop buildkit; limactl list; limactl start buildkit

BuildKit only builds: no images, no run. Run the result via podman load -i myapp.tar.

Registry and moving images (Days 7–10)

Your own registry

./run-registry.sh [kind]                        # restart the registry (7, 10)
curl -sS -u "$REG_USER:$REG_PASS" "https://$REG/v2/_catalog"; echo
curl -sS -u "$REG_USER:$REG_PASS" "https://$REG/v2/notes/notes-api/tags/list"; echo
podman exec registry registry garbage-collect --dry-run /etc/distribution/config.yml
Tool Writes login to Also reads
podman, buildah, skopeo $XDG_RUNTIME_DIR/containers/auth.json ~/.docker/config.json
crane, krane ~/.docker/config.json Podman auth.json
regctl ~/.regctl/config.json ~/.docker/config.json
oras, cosign ~/.docker/config.json –

With DOCKER_CONFIG set, crane and oras read and write there instead of in ~/.docker; podman login then writes there too, as long as REGISTRY_AUTH_FILE isn’t set (16).

crane

echo "$REG_PASS" | crane auth login "$REG" -u "$REG_USER" --password-stdin
crane catalog "$REG"
crane ls docker.io/library/nginx | grep '^1\.30\.6-alpine'
crane digest docker.io/library/nginx:1.30.6-alpine              # HEAD: doesn't count
crane digest --platform linux/arm64 docker.io/library/nginx:1.30.6-alpine
crane manifest "$REG/library/nginx:1.30.6-alpine" | jq -r .mediaType
crane config --platform linux/arm64 "$REG/library/nginx:1.30.6-alpine" | jq .config
crane copy docker.io/library/nginx:1.30.6-alpine "$REG/library/nginx:1.30.6-alpine"
crane tag "$REG/lab/notes-api:oci" stabil
crane delete "$REG/lab/notes-api@$(crane digest "$REG/lab/notes-api:oci")"
crane mutate "$REG/lab/notes-api:0.1.0" --label lab.tag=07 -t "$REG/lab/notes-api:mut"
crane append -b "$REG/lab/notes-api:0.1.0" -f extra.tar -t "$REG/lab/notes-api:extra"
crane export "$REG/lab/notes-api:extra" - | tar -t
crane validate --remote "$REG/lab/notes-api:extra"
echo "$REG" | crane auth get; echo

skopeo

skopeo inspect "docker://$REG/library/nginx:1.30.6-alpine" | jq '{Digest, Architecture}'
skopeo inspect --config --raw "docker://$REG/notes/notes-api:0.1.0" \
  | jq -c .config.Healthcheck.Test              # --raw: original config (8)
skopeo copy -q --all --preserve-digests "docker://$REG/library/nginx:1.30.6-alpine" \
  "docker://$REG/mirror/nginx:1.30.6-alpine-all"    # a true mirror
skopeo copy -q "docker://$REG/notes/notes-api:0.1.0" oci:api-oci:0.1.0
podman unshare skopeo copy -q dir:api-dir \
  containers-storage:localhost/notes-api:von-dir    # the only way rootless (8)
skopeo sync --src docker --dest dir --scoped "$REG/notes/notes-api" backup/
skopeo list-tags "docker://$REG/notes/notes-api" | jq -c .Tags
skopeo delete "docker://$REG/backup/notes-web:0.1.0"
skopeo --policy policy-lab.json copy -q "docker://$REG/notes/notes-api:0.1.0" dir:p1

Transports: docker:// (registry), dir:, oci:path:tag, docker-archive:file, containers-storage: (Podman storage).

regctl

echo "$REG_PASS" | regctl registry login "$REG" -u "$REG_USER" --pass-stdin
regctl registry whoami "$REG"                   # only knows ~/.regctl/config.json
regctl tag ls --include '1\.30\.6-alpine.*' docker.io/library/nginx
regctl image ratelimit docker.io/library/nginx:1.30.6-alpine
regctl manifest head "$REG/library/nginx:1.30.6-alpine"
regctl manifest get "$REG/notes/notes-api:0.1.0" --format raw-body | jq -r .mediaType
regctl image digest --platform linux/arm64 "$REG/library/nginx:1.30.6-alpine"
regctl image copy "$REG/notes/notes-api:0.1.0" "$REG/lab/notes-api:rc"
regctl image mod "$REG/lab/notes-api:rc" --to-oci --label lab.tag=08 \
  --create 0.1.0-oci08                          # health check stays (8)
regctl image get-file "$REG/notes/notes-api:0.1.0" /etc/ssl/certs/ca-certificates.crt
regctl image export "$REG/notes/notes-api:0.1.0" api-regctl.tar
regctl tag rm "$REG/lab/notes-api:rc"           # only the tag
regctl repo ls "$REG"
regctl artifact list "$REG/release/notes-api:0.1.0"

regsync

regsync config -c regsync.yml                   # shows passwords in plain text!
regsync check -c regsync.yml
regsync once -c regsync.yml
regsync once --missing -c regsync.yml
timeout 15 regsync server -c regsync.yml
systemd-analyze verify --user systemd/regsync.service systemd/regsync.timer
# regsync.yml – template (9)
version: 1
creds:
  - registry: '{{ env "REG" }}'
    user: '{{ env "REG_USER" }}'
    pass: '{{ env "REG_PASS" }}'
defaults:
  ratelimit: {min: 20, retry: 15m}
sync:
  - source: docker.io/library/postgres
    target: '{{ env "REG" }}/mirror/postgres'
    type: repository
    platform: linux/arm64
    tags:
      allow: ['18\.6-alpine']

oras

echo "$REG_PASS" | oras login -u "$REG_USER" --password-stdin "$REG"
oras repo ls "$REG" && oras repo tags "$REG/notes/notes-api"
oras push "$REG/lab/regsync-config:tag09" \
  --artifact-type application/vnd.ct-lab.config regsync.yml:application/yaml
oras pull -o /tmp/zurueck "$REG/lab/regsync-config:tag09"
oras manifest fetch "$REG/charts/notes:0.1.0" | jq .
oras attach --artifact-type application/spdx+json "$REG/notes/notes-api:0.1.0" \
  notes-api.spdx.json:application/spdx+json
oras discover "$REG/notes/notes-api:0.1.0"
oras discover --artifact-type text/markdown "$REG/notes/notes-web:0.1.0"
oras cp -r "$REG/notes/notes-api:0.1.0" "$REG/release/notes-api:0.1.0"   # with referrers
oras manifest delete --force "$REG/notes/notes-api@$S"

kind

export KIND_EXPERIMENTAL_PROVIDER=podman
CONTAINERS_CONF_OVERRIDE=$PWD/kind/containers-kind.conf \
  kind create cluster --config kind/kind-ct-lab.yaml        # rootless (10)
N=ct-lab-control-plane
podman exec $N kubectl get pods -A                         # kubectl in the node (10)
podman save -q -o api-020.tar localhost/notes-api:0.2.0
kind load image-archive api-020.tar --name ct-lab
kind delete cluster --name ct-lab

crictl

podman cp /usr/local/bin/crictl ct-lab-control-plane:/usr/local/bin/crictl
N=ct-lab-control-plane
podman exec $N crictl pods                                    # crictl in the node (10)
podman exec $N crictl ps
podman exec $N crictl images
CID=$(podman exec $N crictl ps --name notes-api -q)
podman exec $N crictl inspect "$CID" | jq -r .status.imageRef
podman exec $N crictl logs --tail 1 "$CID"
podman exec $N crictl exec "$CID" /notes-api healthcheck
podman exec $N crictl pull --creds lab:lab-registry-pw-123 \
  registry.ct.lab:5000/notes/notes-web:0.1.0
podman exec $N crictl rmi registry.ct.lab:5000/notes/notes-web:0.1.0

krane

krane version && krane ls "$REG/notes/notes-api"   # crane + cloud keychains (10)
DOCKER_CONFIG=/tmp/leer env -u XDG_RUNTIME_DIR krane ls "$REG/notes/notes-api"

krane reads the Docker config, Google, GitHub, ECR, ACR – no kubeconfig, no pull secrets. A pull secret is a Docker config: mount it as a file and point DOCKER_CONFIG at it.

Checking and securing (Days 11–14)

dive

dive docker.io/library/nginx:1.30.6-alpine          # UI via DOCKER_HOST
CI=true dive podman://docker.io/library/nginx:1.30.6-alpine     # calls podman save
dive --ci --source podman localhost/notes-api:schlecht   # rules from .dive-ci (11)
dive --source podman localhost/notes-api:schlecht -j dive.json
dive --ci docker-archive://nginx.tar
# .dive-ci – template (11)
rules:
  lowestEfficiency: 0.98
  highestWastedBytes: 10MB
  highestUserWastedPercent: 0.05

diffoci

diffoci diff --semantic podman://localhost/notes-api:0.1.0 \
  podman://localhost/notes-api:0.2.0            # exit 1 on differences (11)
diffoci diff --semantic --verbose podman://… podman://…
diffoci diff --semantic podman://localhost/notes-api:0.1.0 "$REG/notes/notes-api:0.1.0"

container-diff

Archived, only for comparison with diffoci (11):

podman run --rm -v "$PWD/bin:/out:Z" -e CGO_ENABLED=0 \
  docker.io/library/golang:1.27-alpine sh -c '
  m=github.com/GoogleContainerTools/[email protected]
  cd "$(go mod download -json "$m" | sed -n "s/.*\"Dir\": \"\(.*\)\",/\1/p")"
  go build -o /out/container-diff .'
./bin/container-diff diff api-010.tar api-020.tar --type=file --type=size

hadolint

hadolint --no-color notes-api/Dockerfile        # reads .hadolint.yaml in the folder (12)
hadolint -t error notes-api/Containerfile.lint
hadolint --no-fail notes-api/Containerfile.lint
hadolint --disable-ignore-pragma notes-api/Containerfile.lint
hadolint -f sarif notes-api/Containerfile.lint -o lint.sarif

Exceptions: # hadolint ignore=DL3008 above the line, --ignore DL3059 for one run, ignored:, override: and trustedRegistries: in .hadolint.yaml.

dockle

export DOCKLE_HOST="unix://$XDG_RUNTIME_DIR/podman/podman.sock"   # not DOCKER_HOST
dockle --no-color -c 1 localhost/notes-worker:0.1.0     # .dockleignore in the folder (12)
dockle --no-color --accept-key KEY_SHA512 --exit-code 1 \
  docker.io/library/nginx:1.30.6-alpine
dockle --no-color --input api.tar               # archive from podman save
dockle -f json localhost/notes-api:0.1.0 | jq -c '.summary'

ShellCheck

sudo apt-get install -y shellcheck
shellcheck push-alle.sh                          # findings with SC code, suggestion (12)
shellcheck -x push-alle.sh                       # check source files too
shellcheck -S warning -f gcc tag12/*.sh          # warnings only, format for editor/CI
shellcheck -f json1 push-alle.sh | jq -c '.comments[] | [.line, .code]'

Exceptions: # shellcheck disable=SC2086 above the line, # shellcheck source=file for source, source-path=SCRIPTDIR and external-sources=true in .shellcheckrc.

trivy

trivy image -q --scanners vuln --severity HIGH,CRITICAL --ignore-unfixed \
  --exit-code 1 localhost/notes-api:0.1.0       # gate, reads .trivyignore (13)
trivy image -q --scanners vuln --show-suppressed localhost/notes-api:0.1.0
trivy image -q --scanners vuln --ignorefile '' localhost/notes-api:0.1.1
trivy image -q --scanners secret --exit-code 1 localhost/notes-api:schlecht
trivy fs -q --scanners vuln --table-mode summary .
trivy config -q notes-api/Containerfile.besser
trivy sbom -q --table-mode summary nginx.cdx.json
trivy image -q --skip-db-update … && trivy clean --vuln-db
# .trivyignore – template (13): every exception with a reason and an expiry
CVE-2026-56851 exp:2026-12-31

grype

grype db update        # approx. 4–5 min, 3.2 GB (13)
grype -q podman:docker.io/library/nginx:1.30.6-alpine
grype -q "registry:$REG/notes/notes-api:0.1.0" --only-fixed --by-cve
grype -q podman:localhost/notes-api:0.1.0 --only-fixed --fail-on high   # exit 2
grype -q sbom:nginx.cdx.json --only-fixed --fail-on critical
grype db status

docker scout

Needs Docker CE and a Docker account; sees only Docker’s storage (13):

docker scout quickview docker.io/library/nginx:1.30.6-alpine
docker scout cves --only-severity critical,high --only-fixed --exit-code \
  docker.io/library/nginx:1.30.6-alpine
syft -q podman:docker.io/library/nginx:1.30.6-alpine -o spdx-json \
  | docker scout cves sbom://
docker scout recommendations docker.io/library/nginx:1.30.6-alpine

syft

syft -q podman:localhost/notes-api:0.1.0        # table, Go modules from the binary
syft -q dir:. -o purls
syft -q "registry:$REG/notes/notes-api:0.1.0" -o syft-json=api.syft.json \
  -o cyclonedx-json=api.cdx.json -o spdx-json=api.spdx.json      # three formats (14)
syft convert api.syft.json -o cyclonedx-json=konvertiert.cdx.json

cosign

export COSIGN_PASSWORD=lab-cosign-pw-123
cosign generate-key-pair                        # in ~/ct-lab/keys, never in the repo (14)
cosign signing-config create --out lab-signing-config.json   # without Rekor/TSA
img="$REG/notes/notes-api@$(crane digest "$REG/notes/notes-api:0.1.0")"   # digest!
cosign sign --key ~/ct-lab/keys/cosign.key --signing-config lab-signing-config.json \
  --yes -a version=0.1.0 "$img"
cosign verify --key ~/ct-lab/keys/cosign.pub --insecure-ignore-tlog=true "$img"
cosign attest --key ~/ct-lab/keys/cosign.key --signing-config lab-signing-config.json \
  --yes --type cyclonedx --predicate api.cdx.json "$img"
cosign verify-attestation --key ~/ct-lab/keys/cosign.pub \
  --insecure-ignore-tlog=true --type cyclonedx "$img"
cosign tree "$img"
cosign download attestation "$img"              # verifies nothing
cosign verify … --registry-cacert "$LAB/registry/certs/ca.crt" "$img"

Docker tools (Day 15)

docker and contexts

docker context create podman --docker "host=unix://$XDG_RUNTIME_DIR/podman/podman.sock"
docker context ls                               # DOCKER_HOST overrides every context
unset DOCKER_HOST                               # then the active context applies (15)
docker --context podman ps                      # per command
docker context use podman                       # permanently
docker version --format '{{.Client.Version}} / {{.Server.Version}}'
docker build -q -t ct-lab/notes-api:0.1.0 notes-api
docker ps --format '{{.Names}}\t{{.Status}}'
docker stats --no-stream; docker system df

Buildx and Bake

docker buildx ls
docker buildx create --name ct-lab --driver docker-container \
  --driver-opt image=docker.io/moby/buildkit:v0.34.1 --driver-opt network=host \
  --buildkitd-config buildkitd.toml --bootstrap    # registry CA in the TOML (15)
echo "$REG_PASS" | docker login "$REG" -u "$REG_USER" --password-stdin
docker buildx build --builder ct-lab --platform linux/amd64,linux/arm64 \
  -t "$REG/lab/notes-api:0.1.0" --push notes-api
docker buildx imagetools inspect "$REG/lab/notes-api:0.1.0"
docker buildx bake --print                      # resolved Bake file
docker buildx bake --builder ct-lab --push      # group default
VERSION=0.2.0 docker buildx bake --builder ct-lab --push report   # one target

Docker Compose

docker compose up -d --build --wait             # build, start, health checks (15)
docker compose ps --format '{{.Service}}\t{{.Status}}'
docker compose logs api; docker compose exec api /notes-api healthcheck
docker compose down                             # -v also deletes the volumes

Colima (Mac)

brew install colima docker docker-buildx docker-compose
colima start --cpu 2 --memory 4 --disk 40 --vm-type vz --vz-rosetta   # (15)
colima status; colima list; colima ssh
colima stop; colima delete                      # delete the VM including images
docker context use default                      # back to another context

The CLI finds plugins from Homebrew via "cliPluginsExtraDirs" in ~/.docker/config.json ($(brew --prefix)/lib/docker/cli-plugins).

Scaffdog

docker run --rm -u "$(id -u):$(id -g)" -e HOME=/tmp \
  -v "$PWD":/work -w /work \
  docker.io/library/node:26.13.0-alpine npx --yes [email protected] list   # (15)
# instead of list: generate go-dienst -a name:notes-report -a port:8097 [--dry-run|--force]

Pitfalls on Ubuntu 26.04

All of them occurred in the lab and are shown in the respective chapter.

AppArmor

  • pasta when removing networks (2, 4): When you remove the last container of a custom network rootless (including kube down, docker compose down), Podman reports rootless netns: kill network process: permission denied – the pasta profile denies Podman’s signal. The objects are gone anyway; Docker Compose needs a second down.
  • nerdctl-default, docker-default and gosu (6, 15): Under nerdctl and Docker CE, Postgres hangs with full CPU – the profiles nerdctl-default and docker-default forbid gosu (and Postgres itself) from sending signals to their own threads and processes (journalctl -k | grep DENIED). Lab solution: security_opt: [apparmor=unconfined] for db in compose.nerdctl.yaml or compose.yaml (Day 15); in production, a custom profile or an image without gosu.
  • skopeo and containers-storage: (8, 16): kernel.apparmor_restrict_unprivileged_userns = 1 allows user namespaces only for programs with a profile; podman and buildah have one, skopeo doesn’t → Error during unshare(...): Operation not permitted. Remedy: podman unshare skopeo copy … or podman save.
  • Rootless containerd (6): containerd-rootless-setuptool.sh check fails because /usr/local/bin/rootlesskit has no profile. In this book, containerd stays rootful.

Names and formats

  • Short names (1, 16): registries.conf has no unqualified-search-registries, shortnames.conf doesn’t know nginx and postgres. A short name only matches images that are already local – on a fresh machine, the same build fails. Always write docker.io/library/….
  • HEALTHCHECK disappears (1, 5, 8, 12, 16): Without --format docker, Podman and Buildah build OCI and only warn. buildah push … oci:, skopeo copy --format oci and ko lose it too; regctl image mod --to-oci doesn’t.
  • ctr doesn’t know short names (6): nginx:1.30.6-alpine becomes parse "dummy://nginx…": invalid port.

Packages and services

  • docker-compose-v2 from apt (4): pulls in docker.io and containerd – don’t install it; use podman-compose or the Docker Compose binary.
  • Ports after stopping (4): After systemctl --user stop notes-pod.service, wait a good 3 s, otherwise you get internal libpod error for the next stack on the same ports.
  • Ports below 1024 (2, 16): rootless Listen failed … Permission denied (net.ipv4.ip_unprivileged_port_start = 1024); as a Quadlet service, the message only appears in the unit’s journal.
  • buildkitd (6): exits with exit 1 when stopped, systemctl is-active buildkit shows failed – sudo systemctl reset-failed buildkit.
  • Go in Ubuntu (5): golang-go is 1.26, notes-api requires 1.27 – GOTOOLCHAIN=auto downloads the matching toolchain to ~/go/pkg/mod.

Logins, TLS, registry

  • Podman login in RAM (7): $XDG_RUNTIME_DIR/containers/auth.json is gone after a reboot. regctl and oras never read it (table above).
  • Certificate without SAN (7, 16): curl accepts the CN, none of the Go tools do: x509: certificate relies on legacy Common Name field. Issue the certificate with subjectAltName.
  • Registry 3.1.2 without Referrers API (9, 14): oras and cosign create the tag sha256-<digest>; take it along when copying (oras cp -r).
  • cosign v3 (14): --tlog-upload=false no longer works – an empty signing config and --insecure-ignore-tlog=true when verifying.
  • Docker Hub (7): anonymous: 100 pulls per hour and IP; HEAD (crane digest, regctl image digest) doesn’t count.

Tool quirks

  • dockle (12): ignores DOCKER_HOST and takes localhost/… for a registry – set DOCKLE_HOST or use --input with an archive.
  • grype (13): local Podman images only with the podman: prefix and a running Podman socket.
  • kind (10): kind load docker-image doesn’t find Podman’s localhost/ images (→ image-archive); the node inherits the VM’s /etc/hosts (→ base_hosts_file = "none" via CONTAINERS_CONF_OVERRIDE).
  • dive (11): deleted folders don’t count as waste – look at the layer sizes as well.
  • hadolint (12): -t none still returned exit 1 – use --no-fail.
  • Quadlet (3, 16): typos make the unit silently disappear (Unit … not found); only the dry run names the line. Generated units can’t be enabled.