Day 14: Runners on Kubernetes with ARC
Goal: Install Actions Runner Controller (ARC) in k3s, create a runner scale set for your repository and watch how a fresh runner pod is created for every job and disappears afterwards. · Duration: approx. 75 min · Prerequisite: Day 13
Briefing
- The runner from Day 5 is long-lived: every job sees the traces of the previous one. ARC instead starts an ephemeral runner pod for every job, which is deleted after the job.
- Two Helm charts: the controller (once per cluster) and a runner scale set (one per repository, organization or runner type).
- A listener pod holds the connection to GitHub. When
a job comes in, the controller starts a runner pod – between
minRunnersandmaxRunners. - The name of the scale set is the label:
runs-on: arc-notes.
github.com ◄── listener (arc-systems) ── controller ──► runner pod (arc-runners)
job runs-on: arc-notes ─────────────────────────────► starts, 1 job, gone
Drill 1: Helm and the controller
mkdir -p ~/gha-lab/helm
cd ~/gha-lab/helm
curl -fsSLO https://get.helm.sh/helm-v4.3.1-linux-amd64.tar.gz
tar xzf helm-v4.3.1-linux-amd64.tar.gz
mv linux-amd64/helm ~/.local/bin/
helm version --shortExpected:
v4.3.1+gc0d08d0
The controller comes as an OCI chart from GHCR:
helm install arc --namespace arc-systems --create-namespace \
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller \
--version 0.15.1
kubectl -n arc-systems get podsExpected (after a few seconds):
NAME READY STATUS RESTARTS AGE
arc-gha-rs-controller-557d5846d4-bbnnn 1/1 Running 0 32s
Before that, Helm reports Pulled: … and
STATUS: deployed.
oci://ghcr.io/…: Helm fetches charts directly from an OCI registry, withouthelm repo add.- The controller brings the custom resources:
AutoscalingRunnerSet,EphemeralRunnerSet,EphemeralRunner.
Save your progress? Sign up for free
Drill 2: A runner scale set
The scale set needs access to GitHub. In the lab you use the token
from gh – it ends up as a secret in the cluster, not on the
screen:
kubectl create namespace arc-runners
kubectl -n arc-runners create secret generic gh-token \
--from-literal=github_token="$(gh auth token)"Expected:
namespace/arc-runners created
secret/gh-token created
The settings of the scale set go into a values file:
cat > ~/gha-lab/arc-notes.yaml <<'EOF'
githubConfigUrl: https://github.com/OWNER/notes-actions
githubConfigSecret: gh-token
minRunners: 0
maxRunners: 2
EOF
sed -i "s/OWNER/$(gh api user --jq .login)/" ~/gha-lab/arc-notes.yaml
helm install arc-notes --namespace arc-runners -f ~/gha-lab/arc-notes.yaml \
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \
--version 0.15.1
kubectl -n arc-systems get podsExpected (after a few seconds):
NAME READY STATUS RESTARTS AGE
arc-gha-rs-controller-557d5846d4-bbnnn 1/1 Running 0 65s
arc-notes-7b89bf7c-listener 1/1 Running 0 28s
- The release name
arc-notesbecomes the label forruns-on. minRunners=0: without jobs, not a single runner pod runs, only the listener.- The listener sits in the controller’s namespace, the runner pods go
into
arc-runners.
GitHub now knows the scale set as a runner – without individual runners:
cd ~/gha-lab/notes-actions
gh api repos/{owner}/{repo}/actions/runners --jq '.runners[] | .name + " " + .status'Expected:
gha-vm online
Only the runner from Day 5. A scale set only shows up with a runner when a job comes in.
Save your progress? Sign up for free
Drill 3: A job on ARC
cat > .github/workflows/arc.yml <<'EOF'
name: arc
on: workflow_dispatch
permissions:
contents: read
jobs:
wo:
runs-on: arc-notes
steps:
- name: Wo bin ich?
run: |
echo "Host: $(hostname)"
echo "Spur: $(cat /tmp/spur 2>/dev/null || echo keine)"
echo "$GITHUB_RUN_ID" > /tmp/spur
EOF
git add .github
git commit -qm "arc: Job auf dem Runner Scale Set"
git push -q
gh workflow run arc.yml
kubectl -n arc-runners get podsExpected (a few seconds after the start):
NAME READY STATUS RESTARTS AGE
arc-notes-runner-hn98k 0/1 ContainerCreating 0 13s
The first time, k3s downloads the runner image (around 1 GB) – that takes a while. As soon as the run is finished:
kubectl -n arc-runners get podsExpected:
No resources found in arc-runners namespace.
The pod is gone. What did the job see?
gh run view <ID> --log | grep -E 'Z (Host|Spur):'Expected (timestamps shortened):
wo Wo bin ich? ... Host: arc-notes-runner-hn98k
wo Wo bin ich? ... Spur: keine
A second run:
gh workflow run arc.ymlAs soon as it is finished, with the new ID:
gh run view <ID> --log | grep -E 'Z (Host|Spur):'Expected:
wo Wo bin ich? ... Host: arc-notes-runner-76kz5
wo Wo bin ich? ... Spur: keine
A different pod, no trace – unlike ~/gha-lab/laeufe.txt
on Day 5.
Save your progress? Sign up for free
Drill 4: Scaling
Three runs at once, but at most two runners:
gh workflow run arc.yml
gh workflow run arc.yml
gh workflow run arc.yml
sleep 20
kubectl -n arc-runners get ephemeralrunnersExpected (names shortened):
NAME GITHUB CONFIG URL RUNNERID PHASE ...
arc-notes-runner-8kjzm https://github.com/<du>/notes-actions 5 Running ...
arc-notes-runner-sc96k https://github.com/<du>/notes-actions 6 Running ...
gh run list --workflow arc.yml --limit 3 --json status --jq '.[].status'Expected (depending on timing):
queued
completed
completed
The newest run is at the top: it waits until a runner becomes free.
The other two were already finished here – short jobs are quick.
maxRunners limits how much of the cluster ARC occupies.
Save your progress? Sign up for free
Checkpoint
helm list -AExpected (shortened):
NAME NAMESPACE REVISION UPDATED ... STATUS CHART
arc arc-systems 1 2026-11-... deployed gha-runner-scale-set-cont...
arc-notes arc-runners 1 2026-11-... deployed gha-runner-scale-set-0.15.1
gateway-api-crd kube-system 1 2026-11-... deployed gateway-api-crd-1.6.103
k3s ships gateway-api-crd itself.
kubectl get autoscalingrunnersets -AExpected:
NAMESPACE NAME MINIMUM RUNNERS MAXIMUM RUNNERS PHASE
arc-runners arc-notes 0 2 Running
gh run list --workflow arc.yml --limit 5 --json conclusion --jq '.[].conclusion'Expected:
success
success
success
success
success
Challenge
The first job after a break waits until a pod starts. Keep one runner warm without reinstalling the scale set. Check that a runner pod is running without a job.
Hint: minRunners in
~/gha-lab/arc-notes.yaml, then helm upgrade
with the same file.
Solution
sed -i 's/^minRunners: 0/minRunners: 1/' ~/gha-lab/arc-notes.yaml
helm upgrade arc-notes --namespace arc-runners -f ~/gha-lab/arc-notes.yaml \
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \
--version 0.15.1 >/dev/null
sleep 30
kubectl -n arc-runners get podsExpected:
NAME READY STATUS RESTARTS AGE
arc-notes-runner-xk6g8 1/1 Running 0 35s
The warm runner accepts the next job immediately – afterwards ARC starts a new warm runner. Back to zero:
sed -i 's/^minRunners: 1/minRunners: 0/' ~/gha-lab/arc-notes.yaml
helm upgrade arc-notes --namespace arc-runners -f ~/gha-lab/arc-notes.yaml \
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \
--version 0.15.1 >/dev/null--reuse-values --set minRunners=1 would be shorter, but
fails with Helm 4 on this chart
(incompatible types for comparison). A values file is
better anyway: it belongs in Git, and everyone can see what is
configured.
Save your progress? Sign up for free
Enterprise Note
- GitHub App instead of a token: In production, ARC
authenticates with a GitHub App (
github_app_id,github_app_installation_id,github_app_private_key). Its permissions are narrower, it is not tied to a person, and its tokens expire after one hour. - Containers in jobs: Jobs that build containers
themselves need
containerModedind(Docker in the pod, privileged) orkubernetes(each step a pod of its own). Both need careful consideration – privileged pods are a risk. - Resources: A runner pod without limits takes
whatever it can get. Via
template.specin the chart, runners getresources,nodeSelectorandtolerations– e.g. dedicated nodes just for CI. - Many scale sets: One scale set per runner type
(
arc-klein,arc-gpu) or per organization; the names are the labels inruns-on.