Skip to content
Hands-on IT IT Drills

Day 11: Deployment to Kubernetes

Goal: k3s on the lab VM, notes-api as a deployment, and a workflow that rolls out a chosen version after approval, waits for the rollout and rolls back in case of failure. · Duration: approx. 90 min · Prerequisite: Day 10

Briefing

  • The cluster runs on your VM and is not reachable from outside. GitHub’s runners can’t get to it – your own runner from Day 5 can.
  • The workflow takes the version as an input, runs in the produktion environment and calls kubectl. GitHub records every deployment together with its result.
  • kubectl rollout status waits until the new pods are ready and fails after a deadline – then the next step rolls back.
 gh workflow run deploy.yml -f version=0.5.1
   ─► approval ─► runner gha-vm ─► kubectl set image ─► rollout status
                                                  └── failure ─► rollout undo
 k3s on the VM: namespace notes ─ deployment notes-api ─ service :30081

Drill 1: k3s

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.37.1+k3s1 \
  sh -s - --disable traefik --disable servicelb

Expected (shortened):

[INFO]  Using v1.37.1+k3s1 as release
...
[INFO]  systemd: Starting k3s
  • get.k3s.io is the official installation script. It downloads k3s, creates the service and links kubectl.
  • --disable traefik --disable servicelb: no ingress and no load balancer that grab ports 80 and 443 of the VM. A NodePort is enough for the lab.

The credentials are in /etc/rancher/k3s/k3s.yaml and belong to root. A copy for you:

mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown "$(id -u):$(id -g)" ~/.kube/config
chmod 600 ~/.kube/config
echo 'export KUBECONFIG=~/.kube/config' >> ~/.bashrc
export KUBECONFIG=~/.kube/config
kubectl get nodes

Expected:

NAME   STATUS   ROLES           AGE    VERSION
gha    Ready    control-plane   6m7s   v1.37.1+k3s1

KUBECONFIG is mandatory here: otherwise the k3s kubectl always reads /etc/rancher/k3s/k3s.yaml and fails with permission denied. The line in ~/.bashrc sets the variable on every new login.

Save your progress? Sign up for free

Drill 2: notes-api in the cluster

The manifests belong in the repository, next to the code:

cd ~/gha-lab/notes-actions
mkdir -p k8s
cat > k8s/notes-api.yaml <<'EOF'
apiVersion: v1
kind: Namespace
metadata:
  name: notes
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes-api
  namespace: notes
spec:
  replicas: 2
  selector:
    matchLabels: {app: notes-api}
  template:
    metadata:
      labels: {app: notes-api}
    spec:
      containers:
        - name: notes-api
          image: ghcr.io/OWNER/notes-api:0.5.0
          ports: [{containerPort: 8080}]
          env:
            - name: NOTES_API_KEY
              valueFrom:
                secretKeyRef: {name: notes-api, key: api-key}
          readinessProbe:
            httpGet: {path: /healthz, port: 8080}
            periodSeconds: 2
---
apiVersion: v1
kind: Service
metadata:
  name: notes-api
  namespace: notes
spec:
  type: NodePort
  selector: {app: notes-api}
  ports: [{port: 8080, nodePort: 30081}]
EOF
sed -i "s/OWNER/$(gh api user --jq .login)/" k8s/notes-api.yaml
grep image: k8s/notes-api.yaml

Expected:

          image: ghcr.io/<du>/notes-api:0.5.0
  • readinessProbe: a pod only gets traffic once /healthz responds. That is what rollout status waits for later.
  • {app: notes-api} is YAML in short form – the same as an indented line app: notes-api.

The secret is created by hand, once – it does not belong in the repository. First the namespace and the secret, then the rest:

kubectl create namespace notes
kubectl -n notes create secret generic notes-api --from-literal=api-key=lab-api-key-123
kubectl apply -f k8s/notes-api.yaml
kubectl -n notes rollout status deployment/notes-api --timeout=180s
curl -s localhost:30081/version

Expected:

namespace/notes created
secret/notes-api created
namespace/notes unchanged
deployment.apps/notes-api created
service/notes-api created
deployment "notes-api" successfully rolled out
0.5.0
  • The other way round, the pods would first come up without the secret (CreateContainerConfigError) and only recover after a back-off pause – rollout status could then easily run into the deadline.
  • The image is public (Day 6); k3s needs no pull secret.
git add k8s
git commit -qm "k8s: Manifeste für notes-api"
git push -q

Save your progress? Sign up for free

Drill 3: The deploy workflow

The runner runs as a service and does not read ~/.bashrc. Environment variables for all of its jobs go into the file .env in the runner folder:

echo "KUBECONFIG=$HOME/.kube/config" >> ~/gha-lab/runner/.env
cd ~/gha-lab/runner
sudo ./svc.sh stop
sudo ./svc.sh start
cd ~/gha-lab/notes-actions

Expected (shortened, at the end):

     Active: active (running) since ...

The runner reads .env only at startup – hence the restart.

deploy.yml gets an input and real work:

cat > .github/workflows/deploy.yml <<'EOF'
name: deploy
on:
  workflow_dispatch:
    inputs:
      version:
        description: Image-Tag, z. B. 0.5.1
        required: true

permissions:
  contents: read

concurrency:
  group: deploy-produktion
  cancel-in-progress: false

jobs:
  deploy:
    runs-on: [self-hosted, lab]
    environment:
      name: produktion
      url: http://localhost:30081/version
    env:
      IMAGE: ghcr.io/${{ github.repository_owner }}/notes-api:${{ inputs.version }}
    steps:
      - name: Version setzen
        run: kubectl -n notes set image deployment/notes-api notes-api="$IMAGE"
      - name: Rollout abwarten
        run: kubectl -n notes rollout status deployment/notes-api --timeout=60s
      - name: Zurückrollen
        if: failure()
        run: kubectl -n notes rollout undo deployment/notes-api
EOF
git commit -qam "deploy: Version auf k3s ausrollen"
git push -q
  • runs-on: [self-hosted, lab]: only your runner can reach the cluster. Via KUBECONFIG from .env, it uses the file ~/.kube/config of the user ubuntu.
  • concurrency with cancel-in-progress: false: two deployments run one after the other, neither cancels the other – nobody wants half-finished rollouts.
  • environment.url later appears next to the deployment in GitHub.
  • No checkout: set image only changes the image of the existing deployment. You apply the manifests themselves by hand when they change. In production you would rather write the version into the file and let the cluster fetch it (GitOps) – the sister book on Argo CD shows that.
gh workflow run deploy.yml -f version=0.5.1

Approve with the ID from gh run list --workflow deploy.yml --limit 1 – as on Day 10, with the ID of your environment:

gh api -X POST repos/{owner}/{repo}/actions/runs/<ID>/pending_deployments \
  -F 'environment_ids[]=26123220715' -f state=approved -f comment='0.5.1' \
  --jq '.[].environment'

As soon as the run is finished:

curl -s localhost:30081/version
kubectl -n notes get deployment notes-api -o wide

Expected:

0.5.1
NAME        READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES          ...
notes-api   2/2     2            2           2m59s   notes-api    ghcr.io/<du>/...

-o wide additionally shows containers and images – on the far right is ghcr.io/<du>/notes-api:0.5.1.

Save your progress? Sign up for free

Drill 4: A rollout that fails

A version that doesn’t exist:

gh workflow run deploy.yml -f version=9.9.9

Approve as before. After a little over a minute:

gh run view <ID> -v | grep -E 'Rollout|Zurück'

Expected:

  X Rollout abwarten
  ✓ Zurückrollen
gh run view <ID> --log | grep -E 'Z (error|deployment.apps)'
curl -s localhost:30081/version

Expected (shortened):

deploy  Version setzen  ... deployment.apps/notes-api image updated
deploy  Rollout abwarten    ... error: timed out waiting for the condition
deploy  Zurückrollen    ... deployment.apps/notes-api rolled back
0.5.1
  • The image 9.9.9 does not exist; the new pod stays in ImagePullBackOff, and rollout status gives up after 60 seconds.
  • The old pods kept running the whole time – a rolling update only replaces them when new ones are ready. rollout undo clears away the broken attempt.

GitHub has recorded both deployments:

First the deployments:

gh api repos/{owner}/{repo}/deployments --jq '.[0:2][] | "\(.id) \(.environment)"'

Expected (IDs differ):

7517792135 produktion
7517780729 produktion

Then the status of one of them:

gh api repos/{owner}/{repo}/deployments/7517792135/statuses --jq '.[0].state'

Expected:

failure

In the browser, Deployments shows the same list with the url from the workflow.

Save your progress? Sign up for free

Checkpoint

kubectl -n notes get pods

Expected:

NAME                         READY   STATUS    RESTARTS   AGE
notes-api-55dd946cfc-h67wt   1/1     Running   0          105s
notes-api-55dd946cfc-xm2s9   1/1     Running   0          109s
curl -s localhost:30081/version

Expected:

0.5.1
gh run list --workflow deploy.yml --limit 2 --json conclusion --jq '.[].conclusion'

Expected:

failure
success

Challenge

After every release, the deployment should start on its own (the approval stays). Add a job to release.yml that triggers deploy.yml with the new version, and test it with v0.5.2.

Hint: A release that a workflow creates with the GITHUB_TOKEN triggers no other workflows – protection against infinite loops. The exception: workflow_dispatch. gh workflow run needs actions: write.

Solution

cat >> .github/workflows/release.yml <<'EOF'

  deploy:
    needs: release
    if: ${{ !contains(github.ref_name, '-') }}
    runs-on: ubuntu-24.04
    permissions:
      actions: write
    steps:
      - name: Deployment anstoßen
        env:
          GH_TOKEN: ${{ github.token }}
        run: >-
          gh workflow run deploy.yml --repo "$GITHUB_REPOSITORY"
          -f version="${GITHUB_REF_NAME#v}"
EOF
git commit -qam "release: Deployment anstoßen"
git push -q
git tag v0.5.2
git push -q origin v0.5.2
  • !contains(…, '-'): pre-releases such as v0.6.0-rc.2 don’t go to production.
  • This job needs no checkout – only gh and the token.

After about two minutes (release and image must be finished):

gh run list --workflow deploy.yml --limit 1 --json event,status --jq '.[0]'

Expected:

{"event":"workflow_dispatch","status":"waiting"}

Approve as in Drill 3. Then:

curl -s localhost:30081/version

Expected:

0.5.2

Watch out, race condition: deploy only waits for release, not for image.yml – both run in parallel. Here the approval saves you, since you only give it once the image is there. Without approval, waiting for the image belongs in the deploy job.

Save your progress? Sign up for free

Enterprise Note

  • Push or pull: Here CI pushes into the cluster (push). With GitOps (Argo CD, Flux), CI only writes the new version into Git, and the cluster fetches it – no runner needs access to the cluster.
  • Access to the cluster: Instead of a kubeconfig with admin rights, the runner gets a ServiceAccount that may only change deployments in the notes namespace. With cloud clusters, it obtains access via OIDC (Day 10).
  • Approval before production: Many teams deploy to staging without approval and to produktion with it – two environments, the same workflow.
  • Verify signatures: An admission controller in the cluster can require that only images with a signature and provenance from your repository start (Day 8).