Skip to content
Hands-on IT IT Drills

Day 2: Rootless and pods

Goal: Understand how rootless Podman maps UIDs, file permissions and ports, run notes as a pod with Postgres, and export the pod as Kubernetes YAML and start it again. · Duration: approx. 75 min · Prerequisite: Day 1

Briefing

  • In rootless mode, every container runs in a user namespace: UID 0 in the container is your user, all other UIDs come from your range in /etc/subuid.
  • Files a container creates therefore often belong to “foreign” UIDs on the host, such as 100069. podman unshare opens a shell in exactly this namespace.
  • Networking is done by pasta: it forwards ports from the host into the container. A normal user may not open ports below 1024.
  • A pod is a group of containers with a shared network: they all reach each other via 127.0.0.1. The concept comes from Kubernetes – podman kube speaks its YAML.
 Pod notes (infra container holds network namespace)      Host
 ┌────────────────────────────────────────────────┐
 │ notes-db :5432 ◄─ notes-api :8080 ◄─ notes-web :3000 ◄──── 127.0.0.1:3000
 │      │            (127.0.0.1)                  │ ◄───────── 127.0.0.1:8080
 │      └── Volume notes-pgdata                   │
 └────────────────────────────────────────────────┘

Drill 1: subuid, subgid and newuidmap

Start of the day with the state of Day 1:

source ~/ct-lab/setup/env.sh
mkdir -p ~/ct-lab/tag02 && cd ~/ct-lab/tag02
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag01/notes/." .
id
grep "^$USER:" /etc/subuid /etc/subgid
ls -l /usr/bin/newuidmap /usr/bin/newgidmap

Expected (abridged):

uid=1000(ubuntu) gid=1000(ubuntu) groups=1000(ubuntu),4(adm),24(cdrom),27(sudo),…
/etc/subuid:ubuntu:100000:65536
/etc/subgid:ubuntu:100000:65536
-rwsr-xr-x 1 root root 41944 Feb  2  2026 /usr/bin/newgidmap
-rwsr-xr-x 1 root root 41944 Feb  2  2026 /usr/bin/newuidmap
  • ubuntu:100000:65536 means: your user may use the 65,536 UIDs starting at 100000. useradd enters such ranges automatically for every new user.
  • newuidmap and newgidmap are setuid root (s in the mode). They write the mapping into the kernel – the only piece of root that rootless Podman needs.

Save your progress? Sign up for free

Drill 2: podman unshare and the UID mapping

cat /proc/self/uid_map
podman unshare cat /proc/self/uid_map
podman unshare id
podman run --rm docker.io/library/nginx:1.30.6-alpine cat /proc/self/uid_map

Expected:

         0          0 4294967295
         0       1000          1
         1     100000      65536
uid=0(root) gid=0(root) groups=0(root)
         0       1000          1
         1     100000      65536

The three columns: UID inside, UID outside, count. Outside, your process sees all UIDs 1:1. In the namespace, UID 0 = 1000 (you), UID 1 = 100000, so UID 70 is 100000 + 70 − 1 = 100069. Check it with Postgres, which runs as UID 70:

podman run --rm docker.io/library/postgres:18.6-alpine id postgres
podman run -d --name pg-test -e POSTGRES_PASSWORD=lab-db-pw-123 \
  docker.io/library/postgres:18.6-alpine
sleep 3
ps -o user,pid,args -u 100069 | head -3
podman rm -f -t 0 pg-test

Expected (abridged, PIDs will differ):

Trying to pull docker.io/library/postgres:18.6-alpine...
…
uid=70(postgres) gid=70(postgres) groups=70(postgres),70(postgres)
1f0c…
USER         PID COMMAND
100069     73958 bash /usr/local/bin/docker-entrypoint.sh postgres
100069     73988 initdb --username=postgres --pwfile=/dev/fd/63
pg-test

Postgres is initializing its database right now – as 100069. For ps this is a UID without a name – it belongs to nobody on the host. If a process breaks out of the container, it does not even have your permissions.

Save your progress? Sign up for free

Drill 3: Volumes and permissions, :U

A bind mount is a directory of yours. Postgres starts as root and sets up the permissions itself; many images, however, start directly as a normal user. You simulate that with --user 70:70:

mkdir pgdata-bind && ls -ldn pgdata-bind
podman run --name pg-bind --user 70:70 -e POSTGRES_PASSWORD=lab-db-pw-123 \
  -v ./pgdata-bind:/var/lib/postgresql docker.io/library/postgres:18.6-alpine \
  2>&1 | tail -1
podman rm pg-bind

Expected:

drwxr-xr-x 2 1000 1000 4096 Oct 30 19:00 pgdata-bind
mkdir: can't create directory '/var/lib/postgresql/18/': Permission denied
pg-bind

The directory belongs to you (1000) – so to root inside the container. UID 70 may not create anything there. The option :U adjusts the owner to the container UID before the start:

podman run -d --name pg-bind --user 70:70 -e POSTGRES_PASSWORD=lab-db-pw-123 \
  -v ./pgdata-bind:/var/lib/postgresql:U docker.io/library/postgres:18.6-alpine
sleep 5
podman ps -a --filter name=pg-bind --format '{{.Names}} {{.Status}}'
ls -ldn pgdata-bind
podman unshare ls -ldn pgdata-bind

Expected (abridged):

915f7541…
pg-bind Up 5 seconds
drwxr-xr-x 3 100069 100069 4096 Oct 30 19:01 pgdata-bind
drwxr-xr-x 3 70 70 4096 Oct 30 19:01 pgdata-bind

The same file, two views: 100069 outside, 70 in the namespace. The downside: you yourself can no longer get at the files.

podman rm -f pg-bind
rm -rf pgdata-bind; echo "Exit: $?"
podman unshare rm -rf pgdata-bind && ls -d pgdata-bind

Expected (abridged):

pg-bind
rm: cannot remove 'pgdata-bind/18/docker': Permission denied
Exit: 1
ls: cannot access 'pgdata-bind': No such file or directory

In the namespace you are root and may do anything with what lies in your range. Named volumes (-v notes-pgdata:/…) do not have this problem: they live under ~/.local/share/containers/storage/volumes, and Podman sets up the permissions itself on the first mount.

Save your progress? Sign up for free

Drill 4: Networking with pasta, ports below 1024

podman run --rm docker.io/library/nginx:1.30.6-alpine ip -4 addr show eth0
podman run --rm -p 127.0.0.1:80:80 docker.io/library/nginx:1.30.6-alpine true
echo "Exit: $?"
sysctl net.ipv4.ip_unprivileged_port_start

Expected (abridged, addresses will differ):

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc fq_codel state UNKNOWN …
    inet 192.168.64.…/24 brd 192.168.64.255 scope global eth0
       valid_lft forever preferred_lft forever
Error: pasta failed with exit code 1:
Listen failed for HOST TCP port 127.0.0.1/80: Permission denied
Couldn't listen on requested TCP ports

Exit: 126
net.ipv4.ip_unprivileged_port_start = 1024
  • pasta gives the container a copy of your VM’s address and translates connections in user space. No bridge interface, no iptables rules, no root.
  • Ports below 1024 are privileged. You could unlock them system-wide with the sysctl net.ipv4.ip_unprivileged_port_start; in the lab we stick with 8080 and 3000.

Warning: Ubuntu 26.04 protects pasta with an AppArmor profile that rejects signals from Podman. If you remove the last container of a custom network in rootless mode, Podman reports rootless netns: kill network process: permission denied. The container is gone anyway, and you can ignore the message. Pods in the default network are not affected.

Save your progress? Sign up for free

Drill 5: The pod notes

podman pod create --name notes -p 127.0.0.1:8080:8080 -p 127.0.0.1:3000:3000
podman pod ps
podman ps -a --pod --format '{{.Names}}\t{{.PodName}}\t{{.Status}}'

Expected (abridged, IDs and times will differ on your machine):

16f6dcaf5f98…
POD ID        NAME        STATUS      CREATED                 INFRA ID      # OF CON…
16f6dcaf5f98  notes       Created     Less than a second ago  fedd3f40d7cc  1
16f6dcaf5f98-infra  notes   Created

The ports belong to the pod, not to a container. The infra container does nothing except hold the network namespace. Now the database with a named volume:

podman volume create notes-pgdata
podman run -d --pod notes --name notes-db \
  -e POSTGRES_USER=notes -e POSTGRES_DB=notes -e POSTGRES_PASSWORD=lab-db-pw-123 \
  -v notes-pgdata:/var/lib/postgresql docker.io/library/postgres:18.6-alpine
until podman exec notes-db pg_isready -h 127.0.0.1 -U notes -d notes; do sleep 1; done

Expected (abridged):

notes-pgdata
c2f1c1c9…
127.0.0.1:5432 - no response
127.0.0.1:5432 - accepting connections

api and web find the database and each other at 127.0.0.1 – like processes on the same machine:

DB=postgres://notes:[email protected]:5432/notes
podman run --rm --pod notes -e NOTES_DB_URL=$DB localhost/notes-api:0.1.0 migrate
podman run -d --pod notes --name notes-api \
  -e NOTES_API_KEY=lab-api-key-123 -e NOTES_DB_URL=$DB localhost/notes-api:0.1.0
podman run -d --pod notes --name notes-web -e NOTES_API_URL=http://127.0.0.1:8080 \
  -e NOTES_API_KEY=lab-api-key-123 localhost/notes-web:0.1.0
sleep 4
podman ps --pod --format '{{.Names}}\t{{.PodName}}\t{{.Status}}'
curl -s -X POST 127.0.0.1:3000/api/notes -H 'Content-Type: application/json' \
  -d '{"text":"Tag 2 im Pod"}'; echo
curl -s 127.0.0.1:8080/notes; echo
podman exec notes-db netstat -ltn

Expected (abridged):

time=… level=INFO msg="migration complete" table=notes
beea6244…
35374…
16f6dcaf5f98-infra  notes   Up 11 seconds
notes-db    notes   Up 11 seconds
notes-api   notes   Up 5 seconds (healthy)
notes-web   notes   Up 4 seconds
{"id":"1ac607e9-…","text":"Tag 2 im Pod"}
[{"id":"1ac607e9-…","text":"Tag 2 im Pod"}]
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:5432            0.0.0.0:*               LISTEN
tcp        0      0 :::3000                 :::*                    LISTEN
tcp        0      0 :::5432                 :::*                    LISTEN
tcp        0      0 :::8080                 :::*                    LISTEN

netstat in the db container shows all three services: one network namespace for the whole pod. Postgres listens on 5432 but is not published – from the host you only get to 8080 and 3000.

Save your progress? Sign up for free

Drill 6: podman kube generate

podman kube generate notes -f notes-pod.yaml
grep -v -E '^#|SandboxID|creationTimestamp' notes-pod.yaml

Expected (abridged):

apiVersion: v1
kind: Pod
metadata:
  annotations:
  labels:
    app: notes
  name: notes
spec:
  containers:
  - args:
    - postgres
    env:
    - name: POSTGRES_DB
      value: notes
    - name: POSTGRES_USER
      value: notes
    - name: POSTGRES_PASSWORD
      value: lab-db-pw-123
    image: docker.io/library/postgres:18.6-alpine
    name: notes-db
    ports:
    - containerPort: 3000
      hostIP: 127.0.0.1
      hostPort: 3000
    - containerPort: 8080
      hostIP: 127.0.0.1
      hostPort: 8080
    volumeMounts:
    - mountPath: /var/lib/postgresql
      name: notes-pgdata-pvc
  - env:
    - name: NOTES_API_KEY
      value: lab-api-key-123
    - name: NOTES_DB_URL
      value: postgres://notes:[email protected]:5432/notes
    image: localhost/notes-api:0.1.0
    name: notes-api
    securityContext: {}
  - args:
    - node
    - server.js
    env:
    - name: NOTES_API_URL
      value: http://127.0.0.1:8080
    - name: NOTES_API_KEY
      value: lab-api-key-123
    image: localhost/notes-web:0.1.0
    name: notes-web
    securityContext: {}
  volumes:
  - name: notes-pgdata-pvc
    persistentVolumeClaim:
      claimName: notes-pgdata
  • The pod ports are attached to the first container (hostPort); Kubernetes has no infra container to put them on.
  • The named volume becomes a PersistentVolumeClaim with claimName: notes-pgdata.
  • The passwords are in plain text – fine for lab values, in production they belong in a Secret (Enterprise Note).

Save your progress? Sign up for free

Drill 7: podman kube play and kube down

Remove the pod, play the YAML again:

podman pod rm -f notes
podman volume ls
podman kube play notes-pod.yaml
sleep 5
podman ps --pod --format '{{.Names}}\t{{.PodName}}\t{{.Status}}'
curl -s 127.0.0.1:8080/notes; echo

Expected (abridged):

16f6dcaf5f98…
DRIVER      VOLUME NAME
local       notes-pgdata
Pod:
9ac3b313…
Containers:
480e459f…
4cab69ee…
4e5da64e…

9ac3b3130d8f-infra  notes   Up 6 seconds
notes-notes-db  notes   Up 5 seconds
notes-notes-api notes   Up 5 seconds (starting)
notes-notes-web notes   Up 6 seconds
[{"id":"1ac607e9-…","text":"Tag 2 im Pod"}]
  • pod rm deletes containers, but no named volumes. kube play finds the claim notes-pgdata and mounts the existing volume – the note is still there.
  • The containers are now named <pod>-<container>, so notes-notes-api.
  • kube play attaches pods to the network podman-default-kube-network, not to the default network.

Tearing down works with the same file:

podman kube down notes-pod.yaml
podman volume ls -q
podman kube play notes-pod.yaml >/dev/null

Expected (abridged):

Pods stopped:
Error: stopping container 4eae3488…: removing container 4eae3488… network: 1 error
occurred:
    * rootless netns: kill network process: permission denied

Pods removed:
9ac3b3130d8f…
Secrets removed:
Volumes removed:
notes-pgdata

The error message is the AppArmor trap from Drill 4 (custom network) – the pod is removed anyway. kube down deletes volumes only with --force. For the checkpoint the pod is running again.

Save your progress? Sign up for free

Checkpoint

Your user has a subuid range, unshare makes you root in the namespace:

cd ~/ct-lab/tag02
grep "^$USER:" /etc/subuid
podman unshare id -u

Expected:

ubuntu:100000:65536
0

Volume, pod definition and running pod:

podman volume ls --format '{{.Name}}'
grep 'claimName' notes-pod.yaml
podman pod ps --filter name=notes --format '{{.Name}} {{.Status}}'

Expected (the first line is an anonymous volume, its ID will differ):

07d0786783aa…
notes-pgdata
      claimName: notes-pgdata
notes Running

The note comes from the volume, the bind directory is gone:

curl -s 127.0.0.1:8080/notes | jq -r '.[].text'
ls -d pgdata-bind

Expected:

Tag 2 im Pod
ls: cannot access 'pgdata-bind': No such file or directory

Challenge

Add notes-worker to the pod – only via the YAML, not with podman run. The worker should summarize every 5 seconds.

Hint: The worker reads NOTES_API_URL, NOTES_API_KEY and INTERVAL; kube play --replace replaces a running pod.

Solution

Create a copy and insert a fourth container before volumes: (complete file: code/tag02/challenge/notes-pod-worker.yaml):

  - env:
    - name: NOTES_API_URL
      value: http://127.0.0.1:8080
    - name: NOTES_API_KEY
      value: lab-api-key-123
    - name: INTERVAL
      value: "5"
    image: localhost/notes-worker:0.1.0
    name: notes-worker
podman kube play --replace notes-pod-worker.yaml >/dev/null 2>&1
sleep 12
podman ps --pod --format '{{.Names}}\t{{.Status}}'
podman logs notes-notes-worker | tail -3

Expected (abridged):

0cb01d682f13-infra  Up 13 seconds
notes-notes-db  Up 13 seconds
notes-notes-api Up 12 seconds (healthy)
notes-notes-web Up 12 seconds
notes-notes-worker  Up 12 seconds
worker started, api=http://127.0.0.1:8080, interval=5s
summary: 1 notes, 4 words
summary: 1 notes, 4 words

"5" must be in quotes: Kubernetes environment variables are strings. To finish the day, tear down the pod – the volume stays for Day 3:

podman kube down notes-pod-worker.yaml 2>/dev/null | tail -1
podman pod ps -q | wc -l

Expected:

Volumes removed:
0

Save your progress? Sign up for free

Enterprise Note

  • Assign subuid ranges centrally (LDAP/SSSD or configuration management), so that the ranges of different users never overlap.
  • Bind mounts with :U change owners on the host – dangerous on shared directories. For service data, named volumes or a separate directory per service are the better choice.
  • For ports 80/443 in rootless mode, prefer a rootful reverse proxy or a socket-activated service over opening the sysctl for all users.
  • Generated kube YAML is a starting point, not a finished manifest: secrets instead of plain text, resource limits, probes and a Deployment instead of a bare pod are added in “Kubernetes hands-on”.
  • podman kube play understands only part of the Kubernetes API (Pod, Deployment, PVC, ConfigMap, Secret). Some of what it does not know it ignores without a warning.