Skip to content
Hands-on IT IT Drills

Day 14: Runners on Kubernetes with ARC

Goal: Install Actions Runner Controller (ARC) in k3s, create a runner scale set for your repository and watch how a fresh runner pod is created for every job and disappears afterwards. · Duration: approx. 75 min · Prerequisite: Day 13

Briefing

  • The runner from Day 5 is long-lived: every job sees the traces of the previous one. ARC instead starts an ephemeral runner pod for every job, which is deleted after the job.
  • Two Helm charts: the controller (once per cluster) and a runner scale set (one per repository, organization or runner type).
  • A listener pod holds the connection to GitHub. When a job comes in, the controller starts a runner pod – between minRunners and maxRunners.
  • The name of the scale set is the label: runs-on: arc-notes.
 github.com ◄── listener (arc-systems) ── controller ──► runner pod (arc-runners)
   job runs-on: arc-notes ─────────────────────────────► starts, 1 job, gone

Drill 1: Helm and the controller

mkdir -p ~/gha-lab/helm
cd ~/gha-lab/helm
curl -fsSLO https://get.helm.sh/helm-v4.3.1-linux-amd64.tar.gz
tar xzf helm-v4.3.1-linux-amd64.tar.gz
mv linux-amd64/helm ~/.local/bin/
helm version --short

Expected:

v4.3.1+gc0d08d0

The controller comes as an OCI chart from GHCR:

helm install arc --namespace arc-systems --create-namespace \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller \
  --version 0.15.1
kubectl -n arc-systems get pods

Expected (after a few seconds):

NAME                                     READY   STATUS    RESTARTS   AGE
arc-gha-rs-controller-557d5846d4-bbnnn   1/1     Running   0          32s

Before that, Helm reports Pulled: … and STATUS: deployed.

  • oci://ghcr.io/…: Helm fetches charts directly from an OCI registry, without helm repo add.
  • The controller brings the custom resources: AutoscalingRunnerSet, EphemeralRunnerSet, EphemeralRunner.

Save your progress? Sign up for free

Drill 2: A runner scale set

The scale set needs access to GitHub. In the lab you use the token from gh – it ends up as a secret in the cluster, not on the screen:

kubectl create namespace arc-runners
kubectl -n arc-runners create secret generic gh-token \
  --from-literal=github_token="$(gh auth token)"

Expected:

namespace/arc-runners created
secret/gh-token created

The settings of the scale set go into a values file:

cat > ~/gha-lab/arc-notes.yaml <<'EOF'
githubConfigUrl: https://github.com/OWNER/notes-actions
githubConfigSecret: gh-token
minRunners: 0
maxRunners: 2
EOF
sed -i "s/OWNER/$(gh api user --jq .login)/" ~/gha-lab/arc-notes.yaml
helm install arc-notes --namespace arc-runners -f ~/gha-lab/arc-notes.yaml \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \
  --version 0.15.1
kubectl -n arc-systems get pods

Expected (after a few seconds):

NAME                                     READY   STATUS    RESTARTS   AGE
arc-gha-rs-controller-557d5846d4-bbnnn   1/1     Running   0          65s
arc-notes-7b89bf7c-listener              1/1     Running   0          28s
  • The release name arc-notes becomes the label for runs-on.
  • minRunners=0: without jobs, not a single runner pod runs, only the listener.
  • The listener sits in the controller’s namespace, the runner pods go into arc-runners.

GitHub now knows the scale set as a runner – without individual runners:

cd ~/gha-lab/notes-actions
gh api repos/{owner}/{repo}/actions/runners --jq '.runners[] | .name + " " + .status'

Expected:

gha-vm online

Only the runner from Day 5. A scale set only shows up with a runner when a job comes in.

Save your progress? Sign up for free

Drill 3: A job on ARC

cat > .github/workflows/arc.yml <<'EOF'
name: arc
on: workflow_dispatch

permissions:
  contents: read

jobs:
  wo:
    runs-on: arc-notes
    steps:
      - name: Wo bin ich?
        run: |
          echo "Host: $(hostname)"
          echo "Spur: $(cat /tmp/spur 2>/dev/null || echo keine)"
          echo "$GITHUB_RUN_ID" > /tmp/spur
EOF
git add .github
git commit -qm "arc: Job auf dem Runner Scale Set"
git push -q
gh workflow run arc.yml
kubectl -n arc-runners get pods

Expected (a few seconds after the start):

NAME                     READY   STATUS              RESTARTS   AGE
arc-notes-runner-hn98k   0/1     ContainerCreating   0          13s

The first time, k3s downloads the runner image (around 1 GB) – that takes a while. As soon as the run is finished:

kubectl -n arc-runners get pods

Expected:

No resources found in arc-runners namespace.

The pod is gone. What did the job see?

gh run view <ID> --log | grep -E 'Z (Host|Spur):'

Expected (timestamps shortened):

wo  Wo bin ich? ... Host: arc-notes-runner-hn98k
wo  Wo bin ich? ... Spur: keine

A second run:

gh workflow run arc.yml

As soon as it is finished, with the new ID:

gh run view <ID> --log | grep -E 'Z (Host|Spur):'

Expected:

wo  Wo bin ich? ... Host: arc-notes-runner-76kz5
wo  Wo bin ich? ... Spur: keine

A different pod, no trace – unlike ~/gha-lab/laeufe.txt on Day 5.

Save your progress? Sign up for free

Drill 4: Scaling

Three runs at once, but at most two runners:

gh workflow run arc.yml
gh workflow run arc.yml
gh workflow run arc.yml
sleep 20
kubectl -n arc-runners get ephemeralrunners

Expected (names shortened):

NAME                     GITHUB CONFIG URL                      RUNNERID   PHASE     ...
arc-notes-runner-8kjzm   https://github.com/<du>/notes-actions  5          Running   ...
arc-notes-runner-sc96k   https://github.com/<du>/notes-actions  6          Running   ...
gh run list --workflow arc.yml --limit 3 --json status --jq '.[].status'

Expected (depending on timing):

queued
completed
completed

The newest run is at the top: it waits until a runner becomes free. The other two were already finished here – short jobs are quick. maxRunners limits how much of the cluster ARC occupies.

Save your progress? Sign up for free

Checkpoint

helm list -A

Expected (shortened):

NAME             NAMESPACE   REVISION  UPDATED ...  STATUS    CHART
arc              arc-systems 1         2026-11-...  deployed  gha-runner-scale-set-cont...
arc-notes        arc-runners 1         2026-11-...  deployed  gha-runner-scale-set-0.15.1
gateway-api-crd  kube-system 1         2026-11-...  deployed  gateway-api-crd-1.6.103

k3s ships gateway-api-crd itself.

kubectl get autoscalingrunnersets -A

Expected:

NAMESPACE     NAME        MINIMUM RUNNERS   MAXIMUM RUNNERS   PHASE
arc-runners   arc-notes   0                 2                 Running
gh run list --workflow arc.yml --limit 5 --json conclusion --jq '.[].conclusion'

Expected:

success
success
success
success
success

Challenge

The first job after a break waits until a pod starts. Keep one runner warm without reinstalling the scale set. Check that a runner pod is running without a job.

Hint: minRunners in ~/gha-lab/arc-notes.yaml, then helm upgrade with the same file.

Solution

sed -i 's/^minRunners: 0/minRunners: 1/' ~/gha-lab/arc-notes.yaml
helm upgrade arc-notes --namespace arc-runners -f ~/gha-lab/arc-notes.yaml \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \
  --version 0.15.1 >/dev/null
sleep 30
kubectl -n arc-runners get pods

Expected:

NAME                     READY   STATUS    RESTARTS   AGE
arc-notes-runner-xk6g8   1/1     Running   0          35s

The warm runner accepts the next job immediately – afterwards ARC starts a new warm runner. Back to zero:

sed -i 's/^minRunners: 1/minRunners: 0/' ~/gha-lab/arc-notes.yaml
helm upgrade arc-notes --namespace arc-runners -f ~/gha-lab/arc-notes.yaml \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \
  --version 0.15.1 >/dev/null

--reuse-values --set minRunners=1 would be shorter, but fails with Helm 4 on this chart (incompatible types for comparison). A values file is better anyway: it belongs in Git, and everyone can see what is configured.

Save your progress? Sign up for free

Enterprise Note

  • GitHub App instead of a token: In production, ARC authenticates with a GitHub App (github_app_id, github_app_installation_id, github_app_private_key). Its permissions are narrower, it is not tied to a person, and its tokens expire after one hour.
  • Containers in jobs: Jobs that build containers themselves need containerMode dind (Docker in the pod, privileged) or kubernetes (each step a pod of its own). Both need careful consideration – privileged pods are a risk.
  • Resources: A runner pod without limits takes whatever it can get. Via template.spec in the chart, runners get resources, nodeSelector and tolerations – e.g. dedicated nodes just for CI.
  • Many scale sets: One scale set per runner type (arc-klein, arc-gpu) or per organization; the names are the labels in runs-on.