Day 6: containerd – ctr, nerdctl, buildctl
Goal: Start containerd and BuildKit as system services, handle images and containers at the lowest level with ctr, run notes with nerdctl and
nerdctl composeand build directly with buildctl. · Duration: approx. 90 min · Prerequisite: Day 5
Briefing
- containerd is the runtime underneath Docker and Kubernetes: a daemon that manages images, snapshots and containers. Unlike Podman, it needs a service.
- ctr is containerd’s debug client – bare, with no conveniences. nerdctl is the Docker-compatible client, buildctl talks to the build daemon BuildKit.
- containerd separates everything into namespaces:
defaultfor ctr and nerdctl,k8s.iofor Kubernetes,buildkitfor BuildKit. - Podman and containerd have separate storage. An
image only moves across via an archive:
podman save | sudo ctr images import -.
ubuntu (rootless) root (system services)
┌────────────────────────────┐ ┌───────────────────────────────────────────┐
│ podman, buildah, ko │ save/ │ ctr ─────┐ │
│ ~/.local/share/containers │ import │ nerdctl ─┼► containerd.service ► runc │
│ │───────►│ │ /var/lib/containerd │
└────────────────────────────┘ │ buildctl ─► buildkit.service │
│ /var/lib/buildkit │
└───────────────────────────────────────────┘
Drill 1: Install nerdctl-full, start containerd
containerd, nerdctl, BuildKit, runc and the CNI plugins are available
as one bundle: nerdctl-full. You already unpacked it to
/usr/local on Day 0. On its own, it looks like this:
source ~/ct-lab/setup/env.sh
mkdir -p ~/ct-lab/tag06 && cd ~/ct-lab/tag06
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag05/notes/." .
mkdir -p /tmp/dl && cd /tmp/dl
V=2.4.2
BASE=https://github.com/containerd/nerdctl/releases/download/v$V
curl -fsSLO "$BASE/nerdctl-full-$V-linux-arm64.tar.gz"
curl -fsSLO "$BASE/SHA256SUMS"
grep "nerdctl-full-$V-linux-arm64" SHA256SUMS
sha256sum -c --ignore-missing SHA256SUMS
tar -tzf "nerdctl-full-$V-linux-arm64.tar.gz" | grep -E 'system/.+|^bin/(ctr|runc)$'
sudo tar -C /usr/local -xzf "nerdctl-full-$V-linux-arm64.tar.gz"
cd ~/ct-lab/tag06 && rm -rf /tmp/dl
nerdctl --version; containerd --version | cut -d' ' -f1,3; ctr --version
buildctl --version | cut -d' ' -f1,3; runc --version | head -1Expected (abridged, the download is just under 300 MB):
fe0ef0e9840ee8932640e6058f84b7fa98050f9726e44af44883a5b804ae830e nerdctl-full-2.4.2-l…
nerdctl-full-2.4.2-linux-arm64.tar.gz: OK
bin/ctr
bin/runc
lib/systemd/system/buildkit.service
lib/systemd/system/containerd.service
lib/systemd/system/stargz-snapshotter.service
nerdctl version 2.4.2
containerd v2.4.2
ctr github.com/containerd/containerd/v2 v2.4.2
buildctl v0.34.1
runc version 1.5.2
The checksum is the same as in tools.lock.
tar -C /usr/local unpacks everything to its place: programs
to bin/, CNI plugins to libexec/cni/, three
systemd units to lib/systemd/system/. systemd already knows
that directory:
systemd-analyze unit-paths | grep local
sudo systemctl daemon-reload
sudo systemctl enable --now containerd
systemctl status containerd --no-pager | sed -n 1,3p
ls -l /run/containerd/containerd.sock
ls /etc/containerd; containerd config default | sed -n 1,3pExpected (abridged):
/usr/local/lib/systemd/system
Created symlink '/etc/systemd/system/multi-user.target.wants/containerd.service' → …
● containerd.service - containerd container runtime
Loaded: loaded (/usr/local/lib/systemd/system/containerd.service; enabled; preset:…
Active: active (running) since Fri 2026-10-30 20:20:19 UTC; 47ms ago
srw-rw---- 1 root root 0 Oct 30 20:20 /run/containerd/containerd.sock
ls: cannot access '/etc/containerd': No such file or directory
version = 4
root = '/var/lib/containerd'
state = '/run/containerd'
- The socket belongs to root, mode 0660 – that is why every
ctrandnerdctltoday is preceded bysudo. - There is no
config.toml: containerd runs with built-in defaults, data under/var/lib/containerd.containerd config defaultshows them.
Save your progress? Sign up for free
Drill 2: ctr – namespaces and images
| Profile | ctr |
|---|---|
| Purpose | Operate containerd directly: images, containers, tasks, snapshots |
| Special feature | debug tool, no networks, no ports, no short names |
| Project | containerd.io,
github.com/containerd/containerd |
| Version in the book | v2.4.2 (from nerdctl-full) |
sudo ctr ns ls
sudo ctr images pull nginx:1.30.6-alpine; echo "Exit: $?"
sudo ctr images pull docker.io/library/nginx:1.30.6-alpine 2>&1 | tail -2
sudo ctr ns ls
sudo ctr images ls -qExpected (abridged, IDs and times will differ on your machine):
NAME LABELS
ctr: failed to resolve image: parse "dummy://nginx:1.30.6-alpine": invalid port ":1.3…
Exit: 1
application/vnd.oci.image.index.v1+json sha256:0e52bd054126cdc040b1276c0af347aeb02a41…
Completed pull from OCI Registry (docker.io/library/nginx:1.30.6-alpine) elapsed: 7.2 s…
NAME LABELS
default
docker.io/library/nginx:1.30.6-alpine
- ctr adds nothing: no
docker.io, nolibrary/. It readsnginx:1.30.6-alpineas hostnginxwith port1.30.6-alpine. - The index digest
0e52bd054126…is the same as with Podman on Day 1 – same registry, same image, just different storage. - The namespace
defaultis created with the first object. Without-n, ctr works there.
Save your progress? Sign up for free
Drill 3: ctr run – containers, tasks and snapshots
containerd distinguishes two things that Docker calls “container”: the container is the description (image, configuration, snapshot), the task is the running process that goes with it.
I=docker.io/library/nginx:1.30.6-alpine
sudo ctr run -d "$I" web
sudo ctr containers ls
sudo ctr tasks ls
sudo ctr tasks exec --exec-id e1 web nginx -v
sudo ctr snapshots ls | grep webExpected (abridged):
CONTAINER IMAGE RUNTIME
web docker.io/library/nginx:1.30.6-alpine io.containerd.runc.v2
TASK PID STATUS
web 135129 RUNNING
nginx version: nginx/1.30.6
web sha256:3b40866184a698343e4e4669f485c61c12afbf74c3cec956c4afafaae96e97c5 Active
execneeds its own--exec-id: every additional process is a separate entry in the task.- The snapshot
webis the writable layer on top of the image layers (anupperdirwith overlayfs). Its parent is the last image layer.
Network? ctr run creates an empty network namespace –
only lo, no ports:
sudo ctr run --rm "$I" net-test ip -o addr | cut -c1-48Expected:
1: lo inet 127.0.0.1/8 scope host lo\
1: lo inet6 ::1/128 scope host \
For networks and ports, ctr would need CNI by hand – nerdctl takes care of that (Drill 5). Cleaning up takes three steps: stop the task, delete the task, delete the container.
sudo ctr tasks kill web; sleep 2
sudo ctr tasks ls
sudo ctr tasks rm web
sudo ctr containers rm web
sudo ctr snapshots ls | grep -c webExpected:
TASK PID STATUS
web 135129 STOPPED
0
The snapshot disappears along with the container. The image stays.
Save your progress? Sign up for free
Drill 4: Podman images into containerd
The notes images live in your user’s Podman storage, containerd does
not see them. podman save writes an archive,
ctr images import reads it – through a pipe:
podman save localhost/notes-api:0.2.0 | sudo ctr images import -
sudo ctr images ls -q
podman image inspect localhost/notes-api:0.2.0 --format '{{.Id}}' | cut -c1-12
sudo ctr images inspect localhost/notes-api:0.2.0 | sed -n 4,6p | cut -c1-88Expected (abridged):
localhost/notes api:0.2.0 saved
application/vnd.docker.distribution.manifest.v2+json sha256:cea8daf0f92891ea59719b6e…
Importing elapsed: 1.8 s total: 0.0 B (0.0 B/s)
docker.io/library/nginx:1.30.6-alpine
localhost/notes-api:0.2.0
3cac2607c210
└── application/vnd.docker.distribution.manifest.v2+json @sha256:cea8daf0f92891ea5971…
├── application/vnd.docker.container.image.v1+json @sha256:3cac2607c2107e9249e036…
└── application/vnd.docker.image.rootfs.diff.tar @sha256:f1fca665eac4fa423f4d0c9…
Podman’s image ID is the digest of the config
(3cac2607…). In containerd, the same config hangs under a
manifest – the image arrived bit for bit. The sudo is only
in front of ctr: podman save still reads your
rootless storage.
Save your progress? Sign up for free
Drill 5: nerdctl – Docker commands for containerd
| Profile | nerdctl |
|---|---|
| Purpose | Docker-compatible CLI for containerd: run, ps, build, compose |
| Special feature | CNI networks, namespaces, lazy pulling, image encryption, rootless |
| Project | github.com/containerd/nerdctl |
| Version in the book | 2.4.2 (nerdctl-full) |
sudo nerdctl namespace ls
sudo nerdctl images
sudo nerdctl run -d --name web -p 127.0.0.1:8081:80 \
docker.io/library/nginx:1.30.6-alpine
sudo nerdctl ps
curl -sI 127.0.0.1:8081 | head -2
sudo ctr containers lsExpected (abridged):
NAME CONTAINERS IMAGES VOLUMES LABELS
default 0 2 0
IMAGE ID DISK USAGE CONTENT SIZE EXTRA
localhost/notes-api:0.2.0 cea8daf0f928 23.8MB 11.9MB
nginx:1.30.6-alpine 0e52bd054126 261MB 195MB
61386ca7a7cd80269011bf0267a9d4d0132bbaeee58901d4a136b1324d87f464
CONTAINER ID IMAGE COMMAND …
61386ca7a7cd docker.io/library/nginx:1.30.6-alpine "/docker-entrypoint.…" …
HTTP/1.1 200 OK
Server: nginx/1.30.6
CONTAINER IMAGE …
61386ca7a7cd80269011bf0267a9d4d0132bbaeee58901d4a136b1324d87f464 docker.io/libr…
- nerdctl and ctr share the namespace
default: the nginx pulled by ctr and the imported notes-api are there, and the nerdctl container shows up inctr containers ls. - Like Docker, nerdctl shortens
docker.io/library/in the display – and adds it when pulling.nerdctl imagesshows the manifest digest as ID, not the config.
The network comes from CNI, as a bridge just like with Docker:
sudo nerdctl network ls
ip -br addr show nerdctl0
sudo nerdctl inspect web --format '{{.NetworkSettings.IPAddress}}'
sudo nerdctl rm -f webExpected:
NETWORK ID NAME FILE
17f29b073143 bridge /etc/cni/net.d/nerdctl-bridge.conflist
host
none
nerdctl0 UP 10.4.0.1/24 fe80::7cf7:baff:fe2e:5fcf/64
10.4.0.2
web
The bridge nerdctl0 (10.4.0.0/24) is the counterpart to
docker0; nerdctl writes the configuration to
/etc/cni/net.d/ on first start.
Save your progress? Sign up for free
Drill 6: nerdctl compose with
notes
The same compose.yaml as on Day 4, now on containerd.
First the Quadlet pod has to release ports 8080 and 3000 (wait 3 s, Day
4), then the three images move across in one archive:
systemctl --user stop notes-pod.service && sleep 3
podman save -m localhost/notes-api:0.1.0 localhost/notes-web:0.1.0 \
localhost/notes-worker:0.1.0 | sudo ctr images import - >/dev/null
sudo nerdctl images --format '{{.Repository}}:{{.Tag}}' | grep notes
sudo nerdctl volume create notes-pgdataExpected:
localhost/notes-worker:0.1.0
localhost/notes-web:0.1.0
localhost/notes-api:0.1.0
localhost/notes-api:0.2.0
notes-pgdata
-m (multi-image-archive) packs several images into one
docker-archive. The volume notes-pgdata is marked as
external in compose.yaml – containerd has its
own, empty one. Now start:
sudo nerdctl compose up -d 2>&1 | grep -E 'warn' | cut -c1-88
sleep 30
sudo nerdctl compose ps -a
sudo nerdctl exec notes-db-1 ps -o pid,time,args | head -2
sudo journalctl -k --since '-2 min' | grep -m1 DENIED | grep -o 'profile=.*' \
| cut -c1-88Expected (abridged):
time="…" level=warning msg="service db: healthcheck: CMD exec form is not supported,…
time="…" level=warning msg="Ignoring: service migrate: depends_on: db: condition ser…
time="…" level=warning msg="Ignoring: service api: depends_on: migrate: condition se…
…
NAME IMAGE … SERVICE STATUS
notes-api-1 localhost/notes-api:0.1.0 … api Exited (1) …
notes-migrate-1 localhost/notes-api:0.1.0 … migrate Exited (1) …
notes-web-1 localhost/notes-web:0.1.0 … web running
notes-worker-1 localhost/notes-worker:0.1.0 … worker running
notes-db-1 docker.io/library/postgres:18.6-alpine … db running
PID TIME COMMAND
1 0:33 gosu postgres /usr/local/bin/docker-entrypoint.sh postgres
profile="nerdctl-default" pid=182735 comm="gosu" requested_mask="send" denied_mask="se…
Two problems at once:
- nerdctl compose ignores
condition:independs_on, and says so. migrate and api start immediately, find no database and exit. - The database never comes up:
gosuhangs at full CPU. On Ubuntu 26.04, the AppArmor profilenerdctl-defaultforbids it from sending a signal to its own threads, which Go needs when switching the user ID. Podman uses a different profile – that is where Postgres ran on Days 2–4.
The lab solution: an override file that turns off the profile for
db, and a start in two stages:
cat > compose.nerdctl.yaml <<'EOF'
# compose.nerdctl.yaml – Stand Ende Tag 6
# Ergänzung zu compose.yaml für nerdctl compose (containerd) auf Ubuntu 26.04:
# Das AppArmor-Profil nerdctl-default verbietet gosu (Postgres-Entrypoint) ein Signal
# an die eigenen Threads, postgres startet nie. Nur im Labor: Profil für db aus.
services:
db:
security_opt:
- apparmor=unconfined
EOF
F="-f compose.yaml -f compose.nerdctl.yaml"
sudo nerdctl compose $F down 2>/dev/null
sudo nerdctl compose $F up -d db 2>/dev/null
until sudo nerdctl inspect -f '{{.State.Health.Status}}' notes-db-1 2>/dev/null \
| grep -q '^healthy'; do sleep 1; done; echo "db healthy"
sudo nerdctl compose $F up -d 2>/dev/null
sleep 3
sudo nerdctl compose ps -aExpected (abridged):
db healthy
NAME IMAGE … SERVICE STATUS
notes-api-1 localhost/notes-api:0.1.0 … api running …
notes-worker-1 localhost/notes-worker:0.1.0 … worker running
notes-web-1 localhost/notes-web:0.1.0 … web running …
notes-migrate-1 localhost/notes-api:0.1.0 … migrate Exited (0) …
notes-db-1 docker.io/library/postgres:18.6-alpine … db running
curl -s 127.0.0.1:8080/version; echo
curl -s -X POST 127.0.0.1:8080/notes -H 'X-API-Key: lab-api-key-123' \
-H 'Content-Type: application/json' -d '{"text":"Tag 6 mit nerdctl compose"}'; echo
curl -s 127.0.0.1:3000/api/notes; echo
sudo nerdctl inspect -f '{{.State.Health.Status}}' notes-api-1
sudo nerdctl compose $F down 2>&1 | tail -1
systemctl --user start notes-pod.serviceExpected (abridged):
0.1.0
{"id":"de8ce9aa-…","text":"Tag 6 mit nerdctl compose"}
[{"id":"de8ce9aa-…","text":"Tag 6 mit nerdctl compose"}]
healthy
time="…" level=info msg="Removing network notes_default"
Only one note: the volume in containerd is a different one from
Podman’s (Drill 8). The HEALTHCHECK from the image also
works under nerdctl.
Security:
apparmor=unconfinedtakes away a layer of protection from the container. In production, you write your own profile instead (nerdctl apparmor) or use a Postgres image that starts directly as a user withoutgosu.
Save your progress? Sign up for free
Drill 7: BuildKit – nerdctl build and buildctl
| Profile | buildctl |
|---|---|
| Purpose | Client for the build daemon BuildKit (buildkitd) |
| Special feature | frontends (Dockerfile and others), cache export, output as OCI archive |
| Project | github.com/moby/buildkit |
| Version in the book | v0.34.1 (nerdctl-full) |
nerdctl build does not build by itself – it needs
buildkitd:
sudo nerdctl build -t localhost/notes-api:0.2.0 notes-api 2>&1 | tail -1 | cut -c1-88
sudo systemctl enable --now buildkit
sudo buildctl debug workers -v | grep -E '^ID|executor|namespace'Expected (abridged):
time="…" level=fatal msg="no buildkit host is available, tried 2 candidates: failed …
Created symlink '/etc/systemd/system/multi-user.target.wants/buildkit.service' → …
ID: rhhdbavuhzrnk9xvsgrqbanzu
org.mobyproject.buildkit.worker.executor: oci
ID: x2p7whwk7om974z41wou8k9l7
org.mobyproject.buildkit.worker.containerd.namespace: buildkit
org.mobyproject.buildkit.worker.executor: containerd
buildkitd comes with two workers: oci
(its own runc, data under /var/lib/buildkit) and
containerd (namespace buildkit). The first one
is used. Now the build works:
sudo nerdctl build -t localhost/notes-api:0.2.0 --build-arg VERSION=0.2.0 \
notes-api 2>&1 | grep -E '^#[0-9]+ (\[final|exporting (manifest|config))'
sudo nerdctl images --format '{{.Repository}}:{{.Tag}} {{.ID}}' | grep 0.2.0
sudo nerdctl image inspect localhost/notes-api:0.2.0 --format '{{.Id}}' | cut -c1-19Expected (abridged, about 1.5 minutes):
#13 [final 2/2] COPY --link --from=build --chmod=555 /out/notes-api /notes-api
#14 exporting manifest sha256:46c20b4c7047a1e3a5b968c4de5e3e62fed5a4cc34ccac330706e9…
#14 exporting config sha256:9cedc11a94a5d11560e7a591c223291f1c1644ccf4e4c62925712a1d…
localhost/notes-api:0.2.0 46c20b4c7047
sha256:9cedc11a94a5
nerdctl has BuildKit produce an archive and imports it into
containerd itself – the tag 0.2.0 now points to the
BuildKit build (config 9cedc11a…), no longer to the Buildah
import. BuildKit evaluates # syntax=docker/dockerfile:1 in
the Dockerfile: it loads the Dockerfile frontend as an image.
buildctl talks to BuildKit directly. Instead of an image name, you tell it which frontend describes the build, which local directories it may read and where the result goes:
mkdir -p out
sudo buildctl build --frontend dockerfile.v0 \
--local context=notes-api --local dockerfile=notes-api \
--opt build-arg:VERSION=0.2.0 \
--output type=oci,dest=out/notes-api-oci.tar,name=localhost/notes-api:0.2.0-buildctl \
--export-cache type=local,dest=out/buildcache 2>&1 \
| grep -E 'exporting manifest|writing cache image'
ls -lh out/notes-api-oci.tar; ls out/buildcache
tar -xOf out/notes-api-oci.tar index.json \
| jq -c '.manifests[] | {mediaType, ref: .annotations["io.containerd.image.name"]}'Expected (abridged):
#14 exporting manifest sha256:46c20b4c7047a1e3a5b968c4de5e3e62fed5a4cc34ccac330706e9…
#15 writing cache image manifest sha256:704ae5aaaa8c7afdcb7cbd530eb659001d6d8cd3a0792…
-rw-rw-rw- 1 root root 4.3M Oct 30 20:52 out/notes-api-oci.tar
blobs cache.lock index.json ingest oci-layout
{"mediaType":"application/vnd.oci.image.manifest.v1+json","ref":"localhost/notes-api:…
--local context=…and--local dockerfile=…are the two directories BuildKit may read. Build args are written--opt build-arg:NAME=value.- The same manifest digest
46c20b4c…as withnerdctl build: all steps came from the daemon’s cache. --export-cache type=localwrites this cache to disk as an OCI layout – for CI runners that start empty every time.
An empty daemon shows how much the cache is worth:
sudo buildctl prune --all >/dev/null
time (sudo buildctl build --frontend dockerfile.v0 --local context=notes-api \
--local dockerfile=notes-api --opt build-arg:VERSION=0.2.0 \
--output type=oci,dest=/dev/null \
--import-cache type=local,src=out/buildcache 2>&1 | grep -c CACHED)Expected (abridged):
6
real 0m4.913s
…
With the cache, 5 seconds instead of a minute and a half – six steps
came from out/buildcache.
Save your progress? Sign up for free
Drill 8: Two worlds – Podman and containerd
The same name, two images:
podman images --format '{{.Repository}}:{{.Tag}} {{.ID}}' localhost/notes-api:0.2.0
sudo nerdctl image inspect localhost/notes-api:0.2.0 --format '{{.Id}}' | cut -c1-19
podman volume inspect notes-pgdata --format '{{.Mountpoint}}'
sudo nerdctl volume inspect notes-pgdata --format '{{.Mountpoint}}'
sudo du -sh /var/lib/containerd /var/lib/buildkit /var/lib/nerdctlExpected (abridged):
localhost/notes-api:0.2.0 3cac2607c210
sha256:9cedc11a94a5
/home/ubuntu/.local/share/containers/storage/volumes/notes-pgdata/_data
/var/lib/nerdctl/1935db59/volumes/default/notes-pgdata/_data
1.2G /var/lib/containerd
59M /var/lib/buildkit
47M /var/lib/nerdctl
| Podman (rootless) | containerd (root) | |
|---|---|---|
| Images | ~/.local/share/containers/storage |
/var/lib/containerd |
| Volumes | …/storage/volumes |
/var/lib/nerdctl/<id>/volumes |
| Build cache | in the same storage | /var/lib/buildkit |
| Bridge | podman save, podman load |
ctr images import, nerdctl save |
Podman also reads an archive from buildctl directly:
podman load -i out/notes-api-oci.tar | tail -1
podman images --format '{{.Repository}}:{{.Tag}} {{.ID}}' \
--filter reference='localhost/notes-api:0.2.0*'Expected:
Loaded image: localhost/notes-api:0.2.0-buildctl
localhost/notes-api:0.2.0-buildctl 9cedc11a94a5
localhost/notes-api:0.2.0-buildah 666976c1aaff
localhost/notes-api:0.2.0 3cac2607c210
localhost/notes-api:0.2.0-ko 2acfdf58f9a3
The same config 9cedc11a… as in containerd – an OCI
archive is the common format.
Rootless containerd exists too:
containerd-rootless-setuptool.sh install sets up containerd
as a user service with RootlessKit. Ubuntu 26.04 blocks that at
first:
containerd-rootless-setuptool.sh check 2>&1 | grep -E 'apparmor|ERROR' | cut -c1-88Expected (abridged):
time="…" level=warning msg="[rootlesskit:parent] This error might have happened becau…
time="…" level=warning msg="[rootlesskit:parent] Hint: try running the following comm…
[ERROR] RootlessKit failed, see the error messages and https://rootlesscontaine.rs/ge…
apparmor_restrict_unprivileged_userns=1 forbids unknown
programs from using user namespaces; the script suggests an AppArmor
profile for /usr/local/bin/rootlesskit. In the book,
containerd stays rootful. Time to call it a day for the services – they
take up RAM, and you only need them again in the challenge:
sudo systemctl disable --now containerd buildkit
systemctl is-active containerd buildkitExpected (abridged):
Removed '/etc/systemd/system/multi-user.target.wants/containerd.service'.
Removed '/etc/systemd/system/multi-user.target.wants/buildkit.service'.
inactive
failed
failed for buildkit is not a problem in the lab:
buildkitd exits with exit code 1 when stopped, and systemd remembers
that. sudo systemctl reset-failed buildkit resets the
state. The data under /var/lib/containerd,
/var/lib/buildkit and /var/lib/nerdctl stays
where it is.
Save your progress? Sign up for free
Drill 9: BuildKit standalone – on the Mac with Lima
BuildKit needs neither Docker nor containerd on your machine: the
client buildctl talks via a socket to a
buildkitd that can run anywhere. On the Mac, Lima starts a
small Linux VM for that.
| Profile | Lima |
|---|---|
| Purpose | Linux VMs on macOS with file and socket forwarding |
| Special feature | templates via template://…;
template://buildkit starts buildkitd rootless |
| Project | github.com/lima-vm/lima (Apache-2.0) |
| Version in the book | 2.2.1, plus buildkit 0.34.1 (both Homebrew) |
On the Mac, not in the VM ct-lab. The Homebrew
package buildkit only brings the client
buildctl, the daemon runs in the Lima VM:
brew install buildkit lima
limactl start --name=buildkit --cpus=2 --memory=2 --disk=20 template://buildkitThe template installs containerd and BuildKit for the normal user in
an Ubuntu VM and forwards the socket to the Mac. At the end,
limactl prints the template’s hint (roughly):
To run `buildkit` on the host (assumes buildctl is installed), run the following
commands:
-------
export BUILDKIT_HOST="unix:///Users/…/.lima/buildkit/sock/buildkitd.sock"
buildctl debug workers
-------
2 GiB RAM are enough for small images; for larger builds with Go,
Java or Node.js, give the VM more (--memory=4).
BUILDKIT_HOST tells buildctl where the daemon
is waiting – permanently in ~/.zshrc (if you manage your
dotfiles with chezmoi, add it there):
echo 'export BUILDKIT_HOST="unix://$HOME/.lima/buildkit/sock/buildkitd.sock"' \
>> ~/.zshrc
source ~/.zshrc
buildctl --version
buildctl debug workersExpected (abridged, verified in the VM ct-lab against a standalone buildkitd):
buildctl github.com/moby/buildkit v0.34.1 f5b59a1f2fb25a0cb7072c398cab9aed7835da4e
ID PLATFORMS
fo2svu2b6g2pjez8jwnntz8bv linux/amd64,linux/amd64/v2,linux/amd64/v3,linux/386
At least one worker – on a Mac with Apple Silicon it shows
linux/arm64. Now a first image, entirely without
Docker:
mkdir -p ~/buildkit-test/myapp && cd ~/buildkit-test/myapp
cat > Dockerfile <<'EOF'
# myapp/Dockerfile – Stand Ende Tag 6: kleinstes Testimage für buildctl
FROM docker.io/library/alpine:3.24
CMD ["echo", "gebaut von BuildKit ohne Docker"]
EOF
buildctl build --frontend dockerfile.v0 --local context=. --local dockerfile=. \
--output type=oci,dest=myapp.tar,name=localhost/myapp:0.1.0 2>&1 | tail -4
ls -lh myapp.tarExpected (abridged):
#5 exporting to oci image format
#5 sending tarball
#5 sending tarball 0.2s done
#5 DONE 0.7s
-rw-rw-rw- 1 ubuntu ubuntu 3.7M Nov 1 12:48 myapp.tar
The most important difference from Docker: BuildKit is
only a build engine. There is no
docker images and no docker run; the result
ends up as an OCI archive on disk or in a registry. A runtime can run it
– for example Podman in the VM ct-lab:
multipass transfer myapp.tar ct-lab:/home/ubuntu/ct-lab/tag06/
multipass exec ct-lab -- podman load -q -i /home/ubuntu/ct-lab/tag06/myapp.tar
multipass exec ct-lab -- podman run --rm localhost/myapp:0.1.0Expected:
Loaded image: localhost/myapp:0.1.0
gebaut von BuildKit ohne Docker
name= in --output gave the archive the
image name, which podman load adopts. The Lima VM takes up
memory as long as it runs:
limactl stop buildkit
limactl list
limactl start buildkitlimactl list shows name, status
(Stopped/Running), CPUs, memory and disk. The
build cache lives on the VM’s disk and survives a stop.
From Day 7 on, once the lab registry is running,
BuildKit writes straight to it: instead of type=oci a
type=image with push=true – tags are versions,
never latest. buildctl reads the login from
~/.docker/config.json (Day 7). The daemon has to reach the
registry and trust its CA. The registry in the VM ct-lab listens only on
127.0.0.1, which is why buildkitd runs standalone as a
container in the VM for this – with the host’s network and its CA
list:
mkdir -p "$XDG_RUNTIME_DIR/bk"
podman run -d --name buildkitd --privileged --network host \
-v "$XDG_RUNTIME_DIR/bk:/run/buildkit" \
-v /etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt:ro \
docker.io/moby/buildkit:v0.34.1
export BUILDKIT_HOST="unix://$XDG_RUNTIME_DIR/bk/buildkitd.sock"
buildctl build --frontend dockerfile.v0 --local context=. --local dockerfile=. \
--output "type=image,name=$REG/lab/myapp:0.1.0,push=true" 2>&1 | tail -3 | cut -c1-80
crane ls "$REG/lab/myapp"Expected (abridged):
#6 pushing manifest for registry.ct.lab:5000/lab/myapp:0.1.0@sha256:d85cecfb28e121…
#6 pushing manifest for registry.ct.lab:5000/lab/myapp:0.1.0@sha256:d85cecfb28e121…
#6 DONE 0.3s
0.1.0
Afterwards podman rm -f buildkitd – or you leave it
running as a build server for Day 7.
Save your progress? Sign up for free
Checkpoint
containerd and BuildKit are stopped and disabled:
cd ~/ct-lab/tag06
systemctl is-enabled containerd buildkit
systemctl is-active containerd buildkitExpected (failed: buildkit exits with exit code 1 on
stop, see above):
disabled
disabled
inactive
failed
containerd’s data is still there:
sudo ls /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256 | wc -lExpected (the number will differ):
1319
OCI archive and build cache from buildctl:
tar -tf out/notes-api-oci.tar | grep index.json
ls out/buildcacheExpected (abridged):
index.json
blobs
cache.lock
index.json
ingest
oci-layout
Compose file for nerdctl, the image in Podman, the Quadlet pod:
grep apparmor compose.nerdctl.yaml
podman images --filter reference=localhost/notes-api:0.2.0-buildctl --format '{{.Tag}}'
systemctl --user is-active notes-pod.serviceExpected:
- apparmor=unconfined
0.2.0-buildctl
active
Challenge
Build notes-api in version 0.3.0 with buildctl directly into
containerd’s storage – without an archive on disk – and start it with
nerdctl on 127.0.0.1:8081. Afterwards, both services are
off again.
Hint: An --output of type
docker writes to stdout with dest=-, and
ctr images import - reads from stdin.
Solution
cd ~/ct-lab/tag06
sudo systemctl start containerd buildkit
sudo buildctl build --frontend dockerfile.v0 --local context=notes-api \
--local dockerfile=notes-api --opt build-arg:VERSION=0.3.0 \
--output type=docker,name=localhost/notes-api:0.3.0,dest=- 2>/dev/null \
| sudo ctr images import -
sudo nerdctl run -d --name api3 -p 127.0.0.1:8081:8080 \
-e NOTES_API_KEY=lab-api-key-123 localhost/notes-api:0.3.0
sleep 3
curl -s 127.0.0.1:8081/version; echo
sudo nerdctl ps --format '{{.Names}} {{.Status}}'
sudo nerdctl rm -f api3
sudo systemctl stop containerd buildkitExpected (abridged):
localhost/notes api:0.3.0 saved
application/vnd.oci.image.manifest.v1+json sha256:c676cd0f88658497dba03d5daf4c704e505…
Importing elapsed: 85.1s total: 0.0 B (0.0 B/s)
b0c520c3e8f3…
0.3.0
api3 Up (healthy)
time="…" level=warning msg="force cleanup timed out for container b0c520c3e8f3…, but …
api3
This is exactly how nerdctl build works internally:
BuildKit delivers an archive, nerdctl imports it.
systemctl start also works for disabled units – only the
automatic start at boot is missing.
Save your progress? Sign up for free
Enterprise Note
- In Kubernetes, the kubelet talks via CRI to containerd in the
namespace
k8s.io(Day 10). ctr and nerdctl with-n k8s.ioare troubleshooting tools there, not operating tools. - containerd configuration (
/etc/containerd/config.toml) for registry mirrors, snapshotters and runtime classes belongs in configuration management, not in manual edits. - BuildKit in CI: as a dedicated service or rootless, with the cache
stored via
--export-cache type=registryin an internal registry instead of on the runner’s disk. - Do not switch off AppArmor profiles across the board: test
nerdctl-defaultand your own profiles, and document and justify deviations per service. - Separate storage means double the disk space and two versions of the
truth per day. In production, images travel via a registry (Day 7), not
via
save | import.