Skip to content
Hands-on IT IT Drills

Day 6: containerd – ctr, nerdctl, buildctl

Goal: Start containerd and BuildKit as system services, handle images and containers at the lowest level with ctr, run notes with nerdctl and nerdctl compose and build directly with buildctl. · Duration: approx. 90 min · Prerequisite: Day 5

Briefing

  • containerd is the runtime underneath Docker and Kubernetes: a daemon that manages images, snapshots and containers. Unlike Podman, it needs a service.
  • ctr is containerd’s debug client – bare, with no conveniences. nerdctl is the Docker-compatible client, buildctl talks to the build daemon BuildKit.
  • containerd separates everything into namespaces: default for ctr and nerdctl, k8s.io for Kubernetes, buildkit for BuildKit.
  • Podman and containerd have separate storage. An image only moves across via an archive: podman save | sudo ctr images import -.
 ubuntu (rootless)                     root (system services)
 ┌────────────────────────────┐        ┌───────────────────────────────────────────┐
 │ podman, buildah, ko        │ save/  │ ctr ─────┐                                │
 │ ~/.local/share/containers  │ import │ nerdctl ─┼► containerd.service ► runc     │
 │                            │───────►│          │  /var/lib/containerd           │
 └────────────────────────────┘        │ buildctl ─► buildkit.service              │
                                       │             /var/lib/buildkit             │
                                       └───────────────────────────────────────────┘

Drill 1: Install nerdctl-full, start containerd

containerd, nerdctl, BuildKit, runc and the CNI plugins are available as one bundle: nerdctl-full. You already unpacked it to /usr/local on Day 0. On its own, it looks like this:

source ~/ct-lab/setup/env.sh
mkdir -p ~/ct-lab/tag06 && cd ~/ct-lab/tag06
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag05/notes/." .
mkdir -p /tmp/dl && cd /tmp/dl
V=2.4.2
BASE=https://github.com/containerd/nerdctl/releases/download/v$V
curl -fsSLO "$BASE/nerdctl-full-$V-linux-arm64.tar.gz"
curl -fsSLO "$BASE/SHA256SUMS"
grep "nerdctl-full-$V-linux-arm64" SHA256SUMS
sha256sum -c --ignore-missing SHA256SUMS
tar -tzf "nerdctl-full-$V-linux-arm64.tar.gz" | grep -E 'system/.+|^bin/(ctr|runc)$'
sudo tar -C /usr/local -xzf "nerdctl-full-$V-linux-arm64.tar.gz"
cd ~/ct-lab/tag06 && rm -rf /tmp/dl
nerdctl --version; containerd --version | cut -d' ' -f1,3; ctr --version
buildctl --version | cut -d' ' -f1,3; runc --version | head -1

Expected (abridged, the download is just under 300 MB):

fe0ef0e9840ee8932640e6058f84b7fa98050f9726e44af44883a5b804ae830e  nerdctl-full-2.4.2-l…
nerdctl-full-2.4.2-linux-arm64.tar.gz: OK
bin/ctr
bin/runc
lib/systemd/system/buildkit.service
lib/systemd/system/containerd.service
lib/systemd/system/stargz-snapshotter.service
nerdctl version 2.4.2
containerd v2.4.2
ctr github.com/containerd/containerd/v2 v2.4.2
buildctl v0.34.1
runc version 1.5.2

The checksum is the same as in tools.lock. tar -C /usr/local unpacks everything to its place: programs to bin/, CNI plugins to libexec/cni/, three systemd units to lib/systemd/system/. systemd already knows that directory:

systemd-analyze unit-paths | grep local
sudo systemctl daemon-reload
sudo systemctl enable --now containerd
systemctl status containerd --no-pager | sed -n 1,3p
ls -l /run/containerd/containerd.sock
ls /etc/containerd; containerd config default | sed -n 1,3p

Expected (abridged):

/usr/local/lib/systemd/system
Created symlink '/etc/systemd/system/multi-user.target.wants/containerd.service' → …
● containerd.service - containerd container runtime
     Loaded: loaded (/usr/local/lib/systemd/system/containerd.service; enabled; preset:…
     Active: active (running) since Fri 2026-10-30 20:20:19 UTC; 47ms ago
srw-rw---- 1 root root 0 Oct 30 20:20 /run/containerd/containerd.sock
ls: cannot access '/etc/containerd': No such file or directory
version = 4
root = '/var/lib/containerd'
state = '/run/containerd'
  • The socket belongs to root, mode 0660 – that is why every ctr and nerdctl today is preceded by sudo.
  • There is no config.toml: containerd runs with built-in defaults, data under /var/lib/containerd. containerd config default shows them.

Save your progress? Sign up for free

Drill 2: ctr – namespaces and images

Profile ctr
Purpose Operate containerd directly: images, containers, tasks, snapshots
Special feature debug tool, no networks, no ports, no short names
Project containerd.io, github.com/containerd/containerd
Version in the book v2.4.2 (from nerdctl-full)
sudo ctr ns ls
sudo ctr images pull nginx:1.30.6-alpine; echo "Exit: $?"
sudo ctr images pull docker.io/library/nginx:1.30.6-alpine 2>&1 | tail -2
sudo ctr ns ls
sudo ctr images ls -q

Expected (abridged, IDs and times will differ on your machine):

NAME LABELS
ctr: failed to resolve image: parse "dummy://nginx:1.30.6-alpine": invalid port ":1.3…
Exit: 1
application/vnd.oci.image.index.v1+json sha256:0e52bd054126cdc040b1276c0af347aeb02a41…
Completed pull from OCI Registry (docker.io/library/nginx:1.30.6-alpine)    elapsed: 7.2 s…
NAME    LABELS
default
docker.io/library/nginx:1.30.6-alpine
  • ctr adds nothing: no docker.io, no library/. It reads nginx:1.30.6-alpine as host nginx with port 1.30.6-alpine.
  • The index digest 0e52bd054126… is the same as with Podman on Day 1 – same registry, same image, just different storage.
  • The namespace default is created with the first object. Without -n, ctr works there.

Save your progress? Sign up for free

Drill 3: ctr run – containers, tasks and snapshots

containerd distinguishes two things that Docker calls “container”: the container is the description (image, configuration, snapshot), the task is the running process that goes with it.

I=docker.io/library/nginx:1.30.6-alpine
sudo ctr run -d "$I" web
sudo ctr containers ls
sudo ctr tasks ls
sudo ctr tasks exec --exec-id e1 web nginx -v
sudo ctr snapshots ls | grep web

Expected (abridged):

CONTAINER    IMAGE                                    RUNTIME
web          docker.io/library/nginx:1.30.6-alpine    io.containerd.runc.v2
TASK    PID       STATUS
web     135129    RUNNING
nginx version: nginx/1.30.6
web    sha256:3b40866184a698343e4e4669f485c61c12afbf74c3cec956c4afafaae96e97c5 Active
  • exec needs its own --exec-id: every additional process is a separate entry in the task.
  • The snapshot web is the writable layer on top of the image layers (an upperdir with overlayfs). Its parent is the last image layer.

Network? ctr run creates an empty network namespace – only lo, no ports:

sudo ctr run --rm "$I" net-test ip -o addr | cut -c1-48

Expected:

1: lo    inet 127.0.0.1/8 scope host lo\
1: lo    inet6 ::1/128 scope host \

For networks and ports, ctr would need CNI by hand – nerdctl takes care of that (Drill 5). Cleaning up takes three steps: stop the task, delete the task, delete the container.

sudo ctr tasks kill web; sleep 2
sudo ctr tasks ls
sudo ctr tasks rm web
sudo ctr containers rm web
sudo ctr snapshots ls | grep -c web

Expected:

TASK    PID       STATUS
web     135129    STOPPED
0

The snapshot disappears along with the container. The image stays.

Save your progress? Sign up for free

Drill 4: Podman images into containerd

The notes images live in your user’s Podman storage, containerd does not see them. podman save writes an archive, ctr images import reads it – through a pipe:

podman save localhost/notes-api:0.2.0 | sudo ctr images import -
sudo ctr images ls -q
podman image inspect localhost/notes-api:0.2.0 --format '{{.Id}}' | cut -c1-12
sudo ctr images inspect localhost/notes-api:0.2.0 | sed -n 4,6p | cut -c1-88

Expected (abridged):

localhost/notes api:0.2.0                   saved
application/vnd.docker.distribution.manifest.v2+json sha256:cea8daf0f92891ea59719b6e…
Importing   elapsed: 1.8 s  total:   0.0 B  (0.0 B/s)
docker.io/library/nginx:1.30.6-alpine
localhost/notes-api:0.2.0
3cac2607c210
└── application/vnd.docker.distribution.manifest.v2+json @sha256:cea8daf0f92891ea5971…
    ├── application/vnd.docker.container.image.v1+json @sha256:3cac2607c2107e9249e036…
    └── application/vnd.docker.image.rootfs.diff.tar @sha256:f1fca665eac4fa423f4d0c9…

Podman’s image ID is the digest of the config (3cac2607…). In containerd, the same config hangs under a manifest – the image arrived bit for bit. The sudo is only in front of ctr: podman save still reads your rootless storage.

Save your progress? Sign up for free

Drill 5: nerdctl – Docker commands for containerd

Profile nerdctl
Purpose Docker-compatible CLI for containerd: run, ps, build, compose
Special feature CNI networks, namespaces, lazy pulling, image encryption, rootless
Project github.com/containerd/nerdctl
Version in the book 2.4.2 (nerdctl-full)
sudo nerdctl namespace ls
sudo nerdctl images
sudo nerdctl run -d --name web -p 127.0.0.1:8081:80 \
  docker.io/library/nginx:1.30.6-alpine
sudo nerdctl ps
curl -sI 127.0.0.1:8081 | head -2
sudo ctr containers ls

Expected (abridged):

NAME       CONTAINERS    IMAGES    VOLUMES    LABELS
default    0             2         0
IMAGE                        ID              DISK USAGE    CONTENT SIZE    EXTRA
localhost/notes-api:0.2.0    cea8daf0f928    23.8MB        11.9MB
nginx:1.30.6-alpine          0e52bd054126    261MB         195MB
61386ca7a7cd80269011bf0267a9d4d0132bbaeee58901d4a136b1324d87f464
CONTAINER ID    IMAGE                                    COMMAND                   …
61386ca7a7cd    docker.io/library/nginx:1.30.6-alpine    "/docker-entrypoint.…"    …
HTTP/1.1 200 OK
Server: nginx/1.30.6
CONTAINER                                                           IMAGE          …
61386ca7a7cd80269011bf0267a9d4d0132bbaeee58901d4a136b1324d87f464    docker.io/libr…
  • nerdctl and ctr share the namespace default: the nginx pulled by ctr and the imported notes-api are there, and the nerdctl container shows up in ctr containers ls.
  • Like Docker, nerdctl shortens docker.io/library/ in the display – and adds it when pulling. nerdctl images shows the manifest digest as ID, not the config.

The network comes from CNI, as a bridge just like with Docker:

sudo nerdctl network ls
ip -br addr show nerdctl0
sudo nerdctl inspect web --format '{{.NetworkSettings.IPAddress}}'
sudo nerdctl rm -f web

Expected:

NETWORK ID      NAME      FILE
17f29b073143    bridge    /etc/cni/net.d/nerdctl-bridge.conflist
                host
                none
nerdctl0         UP             10.4.0.1/24 fe80::7cf7:baff:fe2e:5fcf/64
10.4.0.2
web

The bridge nerdctl0 (10.4.0.0/24) is the counterpart to docker0; nerdctl writes the configuration to /etc/cni/net.d/ on first start.

Save your progress? Sign up for free

Drill 6: nerdctl compose with notes

The same compose.yaml as on Day 4, now on containerd. First the Quadlet pod has to release ports 8080 and 3000 (wait 3 s, Day 4), then the three images move across in one archive:

systemctl --user stop notes-pod.service && sleep 3
podman save -m localhost/notes-api:0.1.0 localhost/notes-web:0.1.0 \
  localhost/notes-worker:0.1.0 | sudo ctr images import - >/dev/null
sudo nerdctl images --format '{{.Repository}}:{{.Tag}}' | grep notes
sudo nerdctl volume create notes-pgdata

Expected:

localhost/notes-worker:0.1.0
localhost/notes-web:0.1.0
localhost/notes-api:0.1.0
localhost/notes-api:0.2.0
notes-pgdata

-m (multi-image-archive) packs several images into one docker-archive. The volume notes-pgdata is marked as external in compose.yaml – containerd has its own, empty one. Now start:

sudo nerdctl compose up -d 2>&1 | grep -E 'warn' | cut -c1-88
sleep 30
sudo nerdctl compose ps -a
sudo nerdctl exec notes-db-1 ps -o pid,time,args | head -2
sudo journalctl -k --since '-2 min' | grep -m1 DENIED | grep -o 'profile=.*' \
  | cut -c1-88

Expected (abridged):

time="…" level=warning msg="service db: healthcheck: CMD exec form is not supported,…
time="…" level=warning msg="Ignoring: service migrate: depends_on: db: condition ser…
time="…" level=warning msg="Ignoring: service api: depends_on: migrate: condition se…
…
NAME               IMAGE                                     …  SERVICE    STATUS
notes-api-1        localhost/notes-api:0.1.0                 …  api        Exited (1) …
notes-migrate-1    localhost/notes-api:0.1.0                 …  migrate    Exited (1) …
notes-web-1        localhost/notes-web:0.1.0                 …  web        running
notes-worker-1     localhost/notes-worker:0.1.0              …  worker     running
notes-db-1         docker.io/library/postgres:18.6-alpine    …  db         running
PID   TIME  COMMAND
    1  0:33 gosu postgres /usr/local/bin/docker-entrypoint.sh postgres
profile="nerdctl-default" pid=182735 comm="gosu" requested_mask="send" denied_mask="se…

Two problems at once:

  • nerdctl compose ignores condition: in depends_on, and says so. migrate and api start immediately, find no database and exit.
  • The database never comes up: gosu hangs at full CPU. On Ubuntu 26.04, the AppArmor profile nerdctl-default forbids it from sending a signal to its own threads, which Go needs when switching the user ID. Podman uses a different profile – that is where Postgres ran on Days 2–4.

The lab solution: an override file that turns off the profile for db, and a start in two stages:

cat > compose.nerdctl.yaml <<'EOF'
# compose.nerdctl.yaml – Stand Ende Tag 6
# Ergänzung zu compose.yaml für nerdctl compose (containerd) auf Ubuntu 26.04:
# Das AppArmor-Profil nerdctl-default verbietet gosu (Postgres-Entrypoint) ein Signal
# an die eigenen Threads, postgres startet nie. Nur im Labor: Profil für db aus.
services:
  db:
    security_opt:
      - apparmor=unconfined
EOF
F="-f compose.yaml -f compose.nerdctl.yaml"
sudo nerdctl compose $F down 2>/dev/null
sudo nerdctl compose $F up -d db 2>/dev/null
until sudo nerdctl inspect -f '{{.State.Health.Status}}' notes-db-1 2>/dev/null \
  | grep -q '^healthy'; do sleep 1; done; echo "db healthy"
sudo nerdctl compose $F up -d 2>/dev/null
sleep 3
sudo nerdctl compose ps -a

Expected (abridged):

db healthy
NAME               IMAGE                                     …  SERVICE    STATUS
notes-api-1        localhost/notes-api:0.1.0                 …  api        running   …
notes-worker-1     localhost/notes-worker:0.1.0              …  worker     running
notes-web-1        localhost/notes-web:0.1.0                 …  web        running   …
notes-migrate-1    localhost/notes-api:0.1.0                 …  migrate    Exited (0) …
notes-db-1         docker.io/library/postgres:18.6-alpine    …  db         running
curl -s 127.0.0.1:8080/version; echo
curl -s -X POST 127.0.0.1:8080/notes -H 'X-API-Key: lab-api-key-123' \
  -H 'Content-Type: application/json' -d '{"text":"Tag 6 mit nerdctl compose"}'; echo
curl -s 127.0.0.1:3000/api/notes; echo
sudo nerdctl inspect -f '{{.State.Health.Status}}' notes-api-1
sudo nerdctl compose $F down 2>&1 | tail -1
systemctl --user start notes-pod.service

Expected (abridged):

0.1.0
{"id":"de8ce9aa-…","text":"Tag 6 mit nerdctl compose"}
[{"id":"de8ce9aa-…","text":"Tag 6 mit nerdctl compose"}]
healthy
time="…" level=info msg="Removing network notes_default"

Only one note: the volume in containerd is a different one from Podman’s (Drill 8). The HEALTHCHECK from the image also works under nerdctl.

Security: apparmor=unconfined takes away a layer of protection from the container. In production, you write your own profile instead (nerdctl apparmor) or use a Postgres image that starts directly as a user without gosu.

Save your progress? Sign up for free

Drill 7: BuildKit – nerdctl build and buildctl

Profile buildctl
Purpose Client for the build daemon BuildKit (buildkitd)
Special feature frontends (Dockerfile and others), cache export, output as OCI archive
Project github.com/moby/buildkit
Version in the book v0.34.1 (nerdctl-full)

nerdctl build does not build by itself – it needs buildkitd:

sudo nerdctl build -t localhost/notes-api:0.2.0 notes-api 2>&1 | tail -1 | cut -c1-88
sudo systemctl enable --now buildkit
sudo buildctl debug workers -v | grep -E '^ID|executor|namespace'

Expected (abridged):

time="…" level=fatal msg="no buildkit host is available, tried 2 candidates: failed …
Created symlink '/etc/systemd/system/multi-user.target.wants/buildkit.service' → …
ID:     rhhdbavuhzrnk9xvsgrqbanzu
    org.mobyproject.buildkit.worker.executor:       oci
ID:     x2p7whwk7om974z41wou8k9l7
    org.mobyproject.buildkit.worker.containerd.namespace:   buildkit
    org.mobyproject.buildkit.worker.executor:       containerd

buildkitd comes with two workers: oci (its own runc, data under /var/lib/buildkit) and containerd (namespace buildkit). The first one is used. Now the build works:

sudo nerdctl build -t localhost/notes-api:0.2.0 --build-arg VERSION=0.2.0 \
  notes-api 2>&1 | grep -E '^#[0-9]+ (\[final|exporting (manifest|config))'
sudo nerdctl images --format '{{.Repository}}:{{.Tag}} {{.ID}}' | grep 0.2.0
sudo nerdctl image inspect localhost/notes-api:0.2.0 --format '{{.Id}}' | cut -c1-19

Expected (abridged, about 1.5 minutes):

#13 [final 2/2] COPY --link --from=build --chmod=555 /out/notes-api /notes-api
#14 exporting manifest sha256:46c20b4c7047a1e3a5b968c4de5e3e62fed5a4cc34ccac330706e9…
#14 exporting config sha256:9cedc11a94a5d11560e7a591c223291f1c1644ccf4e4c62925712a1d…
localhost/notes-api:0.2.0 46c20b4c7047
sha256:9cedc11a94a5

nerdctl has BuildKit produce an archive and imports it into containerd itself – the tag 0.2.0 now points to the BuildKit build (config 9cedc11a…), no longer to the Buildah import. BuildKit evaluates # syntax=docker/dockerfile:1 in the Dockerfile: it loads the Dockerfile frontend as an image.

buildctl talks to BuildKit directly. Instead of an image name, you tell it which frontend describes the build, which local directories it may read and where the result goes:

mkdir -p out
sudo buildctl build --frontend dockerfile.v0 \
  --local context=notes-api --local dockerfile=notes-api \
  --opt build-arg:VERSION=0.2.0 \
  --output type=oci,dest=out/notes-api-oci.tar,name=localhost/notes-api:0.2.0-buildctl \
  --export-cache type=local,dest=out/buildcache 2>&1 \
  | grep -E 'exporting manifest|writing cache image'
ls -lh out/notes-api-oci.tar; ls out/buildcache
tar -xOf out/notes-api-oci.tar index.json \
  | jq -c '.manifests[] | {mediaType, ref: .annotations["io.containerd.image.name"]}'

Expected (abridged):

#14 exporting manifest sha256:46c20b4c7047a1e3a5b968c4de5e3e62fed5a4cc34ccac330706e9…
#15 writing cache image manifest sha256:704ae5aaaa8c7afdcb7cbd530eb659001d6d8cd3a0792…
-rw-rw-rw- 1 root root 4.3M Oct 30 20:52 out/notes-api-oci.tar
blobs  cache.lock  index.json  ingest  oci-layout
{"mediaType":"application/vnd.oci.image.manifest.v1+json","ref":"localhost/notes-api:…
  • --local context=… and --local dockerfile=… are the two directories BuildKit may read. Build args are written --opt build-arg:NAME=value.
  • The same manifest digest 46c20b4c… as with nerdctl build: all steps came from the daemon’s cache.
  • --export-cache type=local writes this cache to disk as an OCI layout – for CI runners that start empty every time.

An empty daemon shows how much the cache is worth:

sudo buildctl prune --all >/dev/null
time (sudo buildctl build --frontend dockerfile.v0 --local context=notes-api \
  --local dockerfile=notes-api --opt build-arg:VERSION=0.2.0 \
  --output type=oci,dest=/dev/null \
  --import-cache type=local,src=out/buildcache 2>&1 | grep -c CACHED)

Expected (abridged):

6

real    0m4.913s
…

With the cache, 5 seconds instead of a minute and a half – six steps came from out/buildcache.

Save your progress? Sign up for free

Drill 8: Two worlds – Podman and containerd

The same name, two images:

podman images --format '{{.Repository}}:{{.Tag}} {{.ID}}' localhost/notes-api:0.2.0
sudo nerdctl image inspect localhost/notes-api:0.2.0 --format '{{.Id}}' | cut -c1-19
podman volume inspect notes-pgdata --format '{{.Mountpoint}}'
sudo nerdctl volume inspect notes-pgdata --format '{{.Mountpoint}}'
sudo du -sh /var/lib/containerd /var/lib/buildkit /var/lib/nerdctl

Expected (abridged):

localhost/notes-api:0.2.0 3cac2607c210
sha256:9cedc11a94a5
/home/ubuntu/.local/share/containers/storage/volumes/notes-pgdata/_data
/var/lib/nerdctl/1935db59/volumes/default/notes-pgdata/_data
1.2G    /var/lib/containerd
59M /var/lib/buildkit
47M /var/lib/nerdctl
Podman (rootless) containerd (root)
Images ~/.local/share/containers/storage /var/lib/containerd
Volumes …/storage/volumes /var/lib/nerdctl/<id>/volumes
Build cache in the same storage /var/lib/buildkit
Bridge podman save, podman load ctr images import, nerdctl save

Podman also reads an archive from buildctl directly:

podman load -i out/notes-api-oci.tar | tail -1
podman images --format '{{.Repository}}:{{.Tag}} {{.ID}}' \
  --filter reference='localhost/notes-api:0.2.0*'

Expected:

Loaded image: localhost/notes-api:0.2.0-buildctl
localhost/notes-api:0.2.0-buildctl 9cedc11a94a5
localhost/notes-api:0.2.0-buildah 666976c1aaff
localhost/notes-api:0.2.0 3cac2607c210
localhost/notes-api:0.2.0-ko 2acfdf58f9a3

The same config 9cedc11a… as in containerd – an OCI archive is the common format.

Rootless containerd exists too: containerd-rootless-setuptool.sh install sets up containerd as a user service with RootlessKit. Ubuntu 26.04 blocks that at first:

containerd-rootless-setuptool.sh check 2>&1 | grep -E 'apparmor|ERROR' | cut -c1-88

Expected (abridged):

time="…" level=warning msg="[rootlesskit:parent] This error might have happened becau…
time="…" level=warning msg="[rootlesskit:parent] Hint: try running the following comm…
[ERROR] RootlessKit failed, see the error messages and https://rootlesscontaine.rs/ge…

apparmor_restrict_unprivileged_userns=1 forbids unknown programs from using user namespaces; the script suggests an AppArmor profile for /usr/local/bin/rootlesskit. In the book, containerd stays rootful. Time to call it a day for the services – they take up RAM, and you only need them again in the challenge:

sudo systemctl disable --now containerd buildkit
systemctl is-active containerd buildkit

Expected (abridged):

Removed '/etc/systemd/system/multi-user.target.wants/containerd.service'.
Removed '/etc/systemd/system/multi-user.target.wants/buildkit.service'.
inactive
failed

failed for buildkit is not a problem in the lab: buildkitd exits with exit code 1 when stopped, and systemd remembers that. sudo systemctl reset-failed buildkit resets the state. The data under /var/lib/containerd, /var/lib/buildkit and /var/lib/nerdctl stays where it is.

Save your progress? Sign up for free

Drill 9: BuildKit standalone – on the Mac with Lima

BuildKit needs neither Docker nor containerd on your machine: the client buildctl talks via a socket to a buildkitd that can run anywhere. On the Mac, Lima starts a small Linux VM for that.

Profile Lima
Purpose Linux VMs on macOS with file and socket forwarding
Special feature templates via template://…; template://buildkit starts buildkitd rootless
Project github.com/lima-vm/lima (Apache-2.0)
Version in the book 2.2.1, plus buildkit 0.34.1 (both Homebrew)

On the Mac, not in the VM ct-lab. The Homebrew package buildkit only brings the client buildctl, the daemon runs in the Lima VM:

brew install buildkit lima
limactl start --name=buildkit --cpus=2 --memory=2 --disk=20 template://buildkit

The template installs containerd and BuildKit for the normal user in an Ubuntu VM and forwards the socket to the Mac. At the end, limactl prints the template’s hint (roughly):

To run `buildkit` on the host (assumes buildctl is installed), run the following
commands:
-------
export BUILDKIT_HOST="unix:///Users/…/.lima/buildkit/sock/buildkitd.sock"
buildctl debug workers
-------

2 GiB RAM are enough for small images; for larger builds with Go, Java or Node.js, give the VM more (--memory=4). BUILDKIT_HOST tells buildctl where the daemon is waiting – permanently in ~/.zshrc (if you manage your dotfiles with chezmoi, add it there):

echo 'export BUILDKIT_HOST="unix://$HOME/.lima/buildkit/sock/buildkitd.sock"' \
  >> ~/.zshrc
source ~/.zshrc
buildctl --version
buildctl debug workers

Expected (abridged, verified in the VM ct-lab against a standalone buildkitd):

buildctl github.com/moby/buildkit v0.34.1 f5b59a1f2fb25a0cb7072c398cab9aed7835da4e
ID              PLATFORMS
fo2svu2b6g2pjez8jwnntz8bv   linux/amd64,linux/amd64/v2,linux/amd64/v3,linux/386

At least one worker – on a Mac with Apple Silicon it shows linux/arm64. Now a first image, entirely without Docker:

mkdir -p ~/buildkit-test/myapp && cd ~/buildkit-test/myapp
cat > Dockerfile <<'EOF'
# myapp/Dockerfile – Stand Ende Tag 6: kleinstes Testimage für buildctl
FROM docker.io/library/alpine:3.24
CMD ["echo", "gebaut von BuildKit ohne Docker"]
EOF
buildctl build --frontend dockerfile.v0 --local context=. --local dockerfile=. \
  --output type=oci,dest=myapp.tar,name=localhost/myapp:0.1.0 2>&1 | tail -4
ls -lh myapp.tar

Expected (abridged):

#5 exporting to oci image format
#5 sending tarball
#5 sending tarball 0.2s done
#5 DONE 0.7s
-rw-rw-rw- 1 ubuntu ubuntu 3.7M Nov  1 12:48 myapp.tar

The most important difference from Docker: BuildKit is only a build engine. There is no docker images and no docker run; the result ends up as an OCI archive on disk or in a registry. A runtime can run it – for example Podman in the VM ct-lab:

multipass transfer myapp.tar ct-lab:/home/ubuntu/ct-lab/tag06/
multipass exec ct-lab -- podman load -q -i /home/ubuntu/ct-lab/tag06/myapp.tar
multipass exec ct-lab -- podman run --rm localhost/myapp:0.1.0

Expected:

Loaded image: localhost/myapp:0.1.0
gebaut von BuildKit ohne Docker

name= in --output gave the archive the image name, which podman load adopts. The Lima VM takes up memory as long as it runs:

limactl stop buildkit
limactl list
limactl start buildkit

limactl list shows name, status (Stopped/Running), CPUs, memory and disk. The build cache lives on the VM’s disk and survives a stop.

From Day 7 on, once the lab registry is running, BuildKit writes straight to it: instead of type=oci a type=image with push=true – tags are versions, never latest. buildctl reads the login from ~/.docker/config.json (Day 7). The daemon has to reach the registry and trust its CA. The registry in the VM ct-lab listens only on 127.0.0.1, which is why buildkitd runs standalone as a container in the VM for this – with the host’s network and its CA list:

mkdir -p "$XDG_RUNTIME_DIR/bk"
podman run -d --name buildkitd --privileged --network host \
  -v "$XDG_RUNTIME_DIR/bk:/run/buildkit" \
  -v /etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt:ro \
  docker.io/moby/buildkit:v0.34.1
export BUILDKIT_HOST="unix://$XDG_RUNTIME_DIR/bk/buildkitd.sock"
buildctl build --frontend dockerfile.v0 --local context=. --local dockerfile=. \
  --output "type=image,name=$REG/lab/myapp:0.1.0,push=true" 2>&1 | tail -3 | cut -c1-80
crane ls "$REG/lab/myapp"

Expected (abridged):

#6 pushing manifest for registry.ct.lab:5000/lab/myapp:0.1.0@sha256:d85cecfb28e121…
#6 pushing manifest for registry.ct.lab:5000/lab/myapp:0.1.0@sha256:d85cecfb28e121…
#6 DONE 0.3s
0.1.0

Afterwards podman rm -f buildkitd – or you leave it running as a build server for Day 7.

Save your progress? Sign up for free

Checkpoint

containerd and BuildKit are stopped and disabled:

cd ~/ct-lab/tag06
systemctl is-enabled containerd buildkit
systemctl is-active containerd buildkit

Expected (failed: buildkit exits with exit code 1 on stop, see above):

disabled
disabled
inactive
failed

containerd’s data is still there:

sudo ls /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256 | wc -l

Expected (the number will differ):

1319

OCI archive and build cache from buildctl:

tar -tf out/notes-api-oci.tar | grep index.json
ls out/buildcache

Expected (abridged):

index.json
blobs
cache.lock
index.json
ingest
oci-layout

Compose file for nerdctl, the image in Podman, the Quadlet pod:

grep apparmor compose.nerdctl.yaml
podman images --filter reference=localhost/notes-api:0.2.0-buildctl --format '{{.Tag}}'
systemctl --user is-active notes-pod.service

Expected:

      - apparmor=unconfined
0.2.0-buildctl
active

Challenge

Build notes-api in version 0.3.0 with buildctl directly into containerd’s storage – without an archive on disk – and start it with nerdctl on 127.0.0.1:8081. Afterwards, both services are off again.

Hint: An --output of type docker writes to stdout with dest=-, and ctr images import - reads from stdin.

Solution

cd ~/ct-lab/tag06
sudo systemctl start containerd buildkit
sudo buildctl build --frontend dockerfile.v0 --local context=notes-api \
  --local dockerfile=notes-api --opt build-arg:VERSION=0.3.0 \
  --output type=docker,name=localhost/notes-api:0.3.0,dest=- 2>/dev/null \
  | sudo ctr images import -
sudo nerdctl run -d --name api3 -p 127.0.0.1:8081:8080 \
  -e NOTES_API_KEY=lab-api-key-123 localhost/notes-api:0.3.0
sleep 3
curl -s 127.0.0.1:8081/version; echo
sudo nerdctl ps --format '{{.Names}} {{.Status}}'
sudo nerdctl rm -f api3
sudo systemctl stop containerd buildkit

Expected (abridged):

localhost/notes api:0.3.0                   saved
application/vnd.oci.image.manifest.v1+json sha256:c676cd0f88658497dba03d5daf4c704e505…
Importing   elapsed: 85.1s  total:   0.0 B  (0.0 B/s)
b0c520c3e8f3…
0.3.0
api3 Up (healthy)
time="…" level=warning msg="force cleanup timed out for container b0c520c3e8f3…, but …
api3

This is exactly how nerdctl build works internally: BuildKit delivers an archive, nerdctl imports it. systemctl start also works for disabled units – only the automatic start at boot is missing.

Save your progress? Sign up for free

Enterprise Note

  • In Kubernetes, the kubelet talks via CRI to containerd in the namespace k8s.io (Day 10). ctr and nerdctl with -n k8s.io are troubleshooting tools there, not operating tools.
  • containerd configuration (/etc/containerd/config.toml) for registry mirrors, snapshotters and runtime classes belongs in configuration management, not in manual edits.
  • BuildKit in CI: as a dedicated service or rootless, with the cache stored via --export-cache type=registry in an internal registry instead of on the runner’s disk.
  • Do not switch off AppArmor profiles across the board: test nerdctl-default and your own profiles, and document and justify deviations per service.
  • Separate storage means double the disk space and two versions of the truth per day. In production, images travel via a registry (Day 7), not via save | import.