Skip to content
Hands-on IT IT Drills

Chapter 1: Running CI/CD chains yourself (GitLab CI, Jenkins, Argo CD/Flux)

In this chapter you will learn:

  • the difference between push deployment and pull deployment (GitOps)
  • how the runner or agent model works and why it is the biggest risk in the chain
  • when self-hosted CI/CD makes sense and how you harden it
  • how to set up GitLab CE with your own runner and build the demo-app without Docker-in-Docker
  • how Argo CD brings the app from Git into the cluster and repairs deviations on its own

What is it about?

Picture a bakery. The recipe is in the office (Git). In the bakehouse, bakers bake according to the recipe (runners). The finished loaves go to the warehouse (registry). A driver takes them to the branch stores (deployment).

Now the security question: who is allowed into the bakehouse? If every supplier has a master key, anyone can mix something into the dough. And the driver: does he get keys to all the branch stores, or do the stores pick up their bread themselves?

That is exactly what this chapter is about. A CI/CD chain is a factory for software. If you run it yourself, you decide on keys, doors and routes yourself. That is work, but it is also control.

   Developer                  CI (build zone)                 CD (target)
  +-----------+  git push  +------------------+  push   +--------------+
  | Laptop    |----------->| GitLab CE :8929  |-------->| reg-build    |
  +-----------+            |  .gitlab-ci.yml  | image   | :5001/:5000  |
                           +--------+---------+         +------+-------+
                                    | job                      |
                                    v                          | pull
                           +------------------+                |
                           | gitlab-runner    |                |
                           | (Docker executor)|                |
                           | Job: test, kaniko|                |
                           +------------------+                |
                                                               v
  Git repo (k8s/) <---------- watches (pull) --------- +---------------+
  "desired state"                                      | Argo CD in    |
                                                       | kind cluster  |
                                                       | -> namespace  |
                                                       |    demo       |
                                                       +---------------+

The CI side builds and tests. The CD side brings the result into the cluster. In our lab these are two separate systems with separate permissions. That is no accident but a security principle.

The concepts made simple

CI and CD

  • Continuous Integration (CI): Every commit is automatically built and tested. The result is an artifact, in our case a container image.
  • Continuous Delivery/Deployment (CD): The artifact is automatically rolled out to an environment. With “delivery”, a human approves; with “deployment”, it runs fully automatically.

Push vs. pull (GitOps)

In the push model, the pipeline actively pushes the result into the cluster. At the end of the CI job there is something like kubectl apply or helm upgrade. For this, the pipeline needs credentials for the cluster.

In the pull model, an agent runs in the cluster. It watches a Git repository with the desired state and continuously reconciles the cluster against it. This approach is called GitOps: Git is the single source of truth.

Aspect Push (CI deploys) Pull (GitOps)
Cluster access CI has credentials only the agent in the cluster
Firewall direction CI → cluster inbound cluster → Git outbound
Deviations (drift) go unnoticed are detected and repaired
Audit pipeline logs Git history

Remember: With the pull model, no path into the production zone has to be open. The agent fetches what it needs. For zone models in public authorities and critical infrastructure (KRITIS) environments, that is a big advantage.

GitOps is based on four principles (OpenGitOps): the state is described declaratively, stored versioned and immutable, pulled automatically and continuously reconciled.

The runner or agent model

GitLab itself does not run any jobs. It distributes them to runners. A runner is a program that regularly asks: “Is there work for me?” It then runs the job and reports the result back. In Jenkins, the executing parts are called agents (formerly “slaves”), and the central server is the controller.

The runner chooses how a job runs via the executor:

Executor How the job runs Isolation
shell directly on the runner host weak
docker in a fresh container medium
kubernetes as a pod in the cluster medium
docker-autoscaler / VM in a fresh VM per job strong

Security: The runner is the most dangerous part of the chain. It runs code that anyone who may change a .gitlab-ci.yml can write. It sees secrets and may write to registries. Whoever takes over a runner can manipulate any artifact without touching the source code.

Runners come in three scopes:

  • Instance runner (formerly “shared runner”): for all projects of the instance.
  • Group runner: for all projects of a group.
  • Project runner: for one project only.

The larger the scope, the greater the damage from a compromise. A runner shared by 300 projects can be attacked from any of those 300 projects.

The problem with Docker-in-Docker

Many guides build images in CI with “Docker-in-Docker” (DinD). For this, the job container needs privileged = true. A privileged container has practically all the rights of the host. A single malicious job can thus break out of the container and take over other jobs, their secrets and the runner itself.

The alternative is build tools that work without a Docker daemon and without privileges:

Tool Principle Note
Kaniko builds in the user space of a normal container Google project archived in 2025, forks exist
Buildah daemonless, OCI-native (Red Hat) rootless in a container needs fine-tuning
BuildKit rootless Docker build engine without root needs adjustments to seccomp/AppArmor

In the lab we use Kaniko. It runs without any extra configuration in a normal Docker executor. In Chapter 2 you assess the build processes in more detail.

Warning: Google archived the Kaniko repository in 2025. The image gcr.io/kaniko-project/executor is still available but no longer receives updates. For a new production environment, evaluate a maintained fork or Buildah/BuildKit. For the principle of “building without privileges”, Kaniko remains a good teaching example in the lab.

Self-hosted or SaaS?

Criterion SaaS (e.g. gitlab.com) Self-hosted
Operational effort low high (updates, backups)
Data sovereignty with the provider with you
Network internet required air-gapped also possible
Customizability limited full

Running it yourself pays off when data must not leave the building, when the production environment is separated from the internet, or when regulatory requirements (e.g. BSI IT-Grundschutz, the German KRITIS regulation) demand control over the build environment. A hybrid form is common: SaaS UI, but your own runners in your own network.

Hardening in five points

  1. Runner isolation: Separate runners per protection level. The runner that builds and signs production images is a project runner for protected branches only.
  2. No privileged shared runners: privileged = false, no mounted Docker socket in jobs, a list of allowed images (allowed_images).
  3. Ephemeral runners: Each job gets a fresh environment that is discarded afterwards. No job can leave anything behind for the next one.
  4. Backups: Database, repositories and key files. With GitLab, a backup without gitlab-secrets.json is only worth half as much.
  5. Updates: CI servers are popular targets. Apply security updates promptly and observe upgrade paths.

Argo CD and Flux

Both are GitOps agents for Kubernetes and CNCF projects with “graduated” status.

Feature Argo CD Flux
UI built-in web UI none (CLI, optional add-on UIs)
Configuration CRD Application CRDs GitRepository, Kustomization
Multi-tenancy AppProject namespaces + RBAC
Getting started very visual lean, very Kubernetes-native

We install Argo CD because the UI helps while learning. We look at Flux as a concept.

Hands-on Lab: Your own CI/CD chain for the demo-app

Prerequisite: Chapter 0 is done, the cluster seclab is running, reg-build is attached to the kind network. GitLab needs about 4 to 6 GB of RAM. Give Docker at least 12 GB, better 16 GB.

kubectl config current-context
docker network inspect kind --format '{{range .Containers}}{{.Name}} {{end}}'

Expected output:

kind-seclab
seclab-control-plane reg-build reg-ziel

Step 1: Start GitLab CE and the runner with docker compose

We attach GitLab and the runner directly to the Docker network kind. That way the CI jobs later reach GitLab as gitlab and the registry as reg-build:5000.

mkdir -p ~/seclab/kap01/gitlab
cat > ~/seclab/kap01/gitlab/compose.yaml <<'EOF'
services:
  gitlab:
    # Lab: latest. In real environments, pin a specific version!
    image: gitlab/gitlab-ce:latest
    container_name: gitlab
    hostname: gitlab
    restart: unless-stopped
    shm_size: "256m"
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'http://localhost:8929'
        # Save resources in the lab
        puma['worker_processes'] = 0
        sidekiq['concurrency'] = 10
        prometheus_monitoring['enable'] = false
    ports:
      - "127.0.0.1:8929:8929"
    volumes:
      - gitlab-config:/etc/gitlab
      - gitlab-logs:/var/log/gitlab
      - gitlab-data:/var/opt/gitlab

  gitlab-runner:
    image: gitlab/gitlab-runner:latest
    container_name: gitlab-runner
    restart: unless-stopped
    volumes:
      - runner-config:/etc/gitlab-runner
      # The runner starts job containers via the host's Docker daemon.
      - /var/run/docker.sock:/var/run/docker.sock

volumes:
  gitlab-config:
  gitlab-logs:
  gitlab-data:
  runner-config:

networks:
  default:
    name: kind
    external: true
EOF
cd ~/seclab/kap01/gitlab
docker compose up -d
docker compose ps

Expected output (shortened):

NAME            IMAGE                         STATUS
gitlab          gitlab/gitlab-ce:latest       Up 30 seconds (health: starting)
gitlab-runner   gitlab/gitlab-runner:latest   Up 30 seconds

Why external_url 'http://localhost:8929' and port 8929:8929? GitLab has its built-in web server listen on the port from external_url. So inside and outside it is the same port. That keeps the URLs in the UI correct.

Security: The runner container has access to the Docker socket. That effectively makes it root on your host. In the lab we accept that. In production, the runner runs on its own disposable VM or as a Kubernetes executor in a separate cluster.

Step 2: Wait and log in

The first start takes 3 to 6 minutes. GitLab sets up the database, Redis and all services.

until [ "$(curl -s -o /dev/null -w '%{http_code}' \
  http://localhost:8929/users/sign_in)" = "200" ]; do
  echo "GitLab startet noch ..."; sleep 15
done
echo "GitLab ist bereit."
docker exec gitlab grep 'Password:' /etc/gitlab/initial_root_password

Expected output (“GitLab is still starting …” until “GitLab is ready.”):

GitLab startet noch ...
...
GitLab ist bereit.
Password: AbC...xyz=

Open http://localhost:8929 and log in as root with this password. Change the password immediately (avatar at the top, “Edit profile”, “Password”).

Warning: The file initial_root_password is deleted automatically after 24 hours. If you missed the password, reset it with docker exec -it gitlab gitlab-rake "gitlab:password:reset[root]".

Step 3: Create an access token and push the demo-app

For Git over HTTP, you don’t use your password but a personal access token (PAT). It is revocable, has an expiration date and only the permissions you give it.

This is how you open the form:

  1. Click your avatar at the top right and then “Edit profile”.
  2. In the left sidebar, choose “Access”, “Personal access tokens”.
  3. Click “Generate token” and choose “Legacy token” in the menu.

Fill in the form like this:

Field Value Why
Token name seclab-git The name tells you later what the token is for, for example when you want to revoke it
Token description Git über HTTP im Lab (optional) Helps as soon as you have several tokens
Expiration date today + 30 days Without input, GitLab sets 365 days. Shorter is better: a forgotten token expires on its own
Scopes only read_repository and write_repository Cloning and pushing over HTTP, nothing else

Leave all other scopes unchecked, especially api. It would give the token full access to the GitLab API: deleting projects, changing settings, creating more tokens. Then click “Generate token” at the bottom.

GitLab shows the new token at the top of the page. It starts with glpat-. Copy it with the copy icon next to the field and store it in your password manager. As soon as you leave or reload the page, you can no longer view it. Then only a new token helps.

Warning: Since GitLab 19.2, “Generate token” offers two kinds. A fine-grained token grants permissions individually per resource and project. That is the more precise choice for production, but it takes considerably more clicks. In the lab you use the legacy token with the classic scopes. Older GitLab versions show the “Add new token” button with the same form instead of the menu.

Tip: In Chapter 11 you fetch build artifacts via the API and need the read_api scope for that. Create a second token then, instead of giving this one more permissions. One token per purpose can be revoked individually.

Now you push the demo-app to GitLab. GitLab creates the project automatically on the first push (“push to create”, private):

cd ~/seclab/demo-app
git remote add origin http://localhost:8929/root/demo-app.git
git push -u origin main

Enter root as the username and the token as the password.

Expected output (shortened):

remote: The private project root/demo-app was successfully created.
...
 * [new branch]      main -> main
branch 'main' set up to track 'origin/main'.

Check in the UI under http://localhost:8929/root/demo-app that the files are there.

Step 4: Create and register the runner

Since GitLab 16, registration works with a runner authentication token (starts with glrt-). You create the runner in the UI first. The old registration tokens are deprecated.

In GitLab: “Admin” (bottom left), “CI/CD”, “Runners”, “New instance runner”. Check “Run untagged jobs” and describe the runner as seclab-runner. Click “Create runner”. GitLab shows the token glrt-… exactly once.

Tip: For a single project, a project runner would be the better choice (project, “Settings”, “CI/CD”, “Runners”). In the lab we use an instance runner so that it is also available for other practice projects later.

Register:

read -rs RUNNER_TOKEN   # paste the token, Enter (not displayed)
docker exec gitlab-runner gitlab-runner register \
  --non-interactive \
  --url "http://gitlab:8929" \
  --token "$RUNNER_TOKEN" \
  --executor "docker" \
  --docker-image "alpine:3.20" \
  --docker-network-mode "kind" \
  --clone-url "http://gitlab:8929" \
  --description "seclab-runner"
unset RUNNER_TOKEN

Expected output (shortened):

Verifying runner... is valid      runner=xxxxxxx
Runner registered successfully. Feel free to start it, ...

The two important options for our lab:

  • --docker-network-mode kind: job containers start in the kind network. There they find gitlab and reg-build.
  • --clone-url http://gitlab:8929: jobs clone via the container name. The external_url (localhost) would point to the job itself inside the job container.

Step 5: Check the runner configuration

docker exec gitlab-runner cat /etc/gitlab-runner/config.toml

Expected output (excerpt):

[[runners]]
  name = "seclab-runner"
  url = "http://gitlab:8929"
  token = "glrt-..."
  executor = "docker"
  clone_url = "http://gitlab:8929"
  [runners.docker]
    image = "alpine:3.20"
    privileged = false
    network_mode = "kind"
    volumes = ["/cache"]

Watch for two lines: privileged = false and volumes = ["/cache"]. The Docker socket is not mounted into the jobs. That is exactly how it should be.

In the UI, the runner now appears under “Admin”, “CI/CD”, “Runners” with a green dot (“Online”).

Step 6: Write the pipeline

The pipeline has two stages. test runs the unit tests. build builds the image with Kaniko and pushes it to reg-build.

cd ~/seclab/demo-app
cat > .gitlab-ci.yml <<'EOF'
stages:
  - test
  - build

variables:
  # Cluster/network address of the build registry (Docker network "kind")
  IMAGE_NAME: "reg-build:5000/demo-app"

test:
  stage: test
  image: python:3.12-slim
  script:
    - pip install --no-cache-dir -r requirements.txt -r requirements-dev.txt
    - python -m pytest -q

build:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:v1.23.2-debug
    entrypoint: [""]
  script:
    - /kaniko/executor
      --context "$CI_PROJECT_DIR"
      --dockerfile "$CI_PROJECT_DIR/Dockerfile"
      --destination "$IMAGE_NAME:$CI_COMMIT_SHORT_SHA"
      --digest-file "$CI_PROJECT_DIR/image-digest.txt"
      --insecure
    - echo "Gebaut: $IMAGE_NAME@$(cat image-digest.txt)"
  artifacts:
    paths:
      - image-digest.txt
    expire_in: 1 week
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
EOF

What happens here:

Part Meaning
-debug image contains a shell, which GitLab needs for script
entrypoint: [""] overrides the Kaniko start command so that GitLab can take control
--insecure allows pushing to an http registry (lab only)
--digest-file writes the digest of the built image to a file
rules builds only on the default branch
tag $CI_COMMIT_SHORT_SHA each image is uniquely tied to a commit

Remember: The digest as a pipeline artifact is worth its weight in gold. Later steps such as signing (Chapter 5) or SBOM generation (Chapter 7) work with exactly this digest. That way you are guaranteed to sign the image you built, and not one that a tag happens to point to right now.

Commit and push:

git add .gitlab-ci.yml
git commit -m "ci: test- und build-Stufe mit Kaniko"
git push

Step 7: Watch the pipeline and check the result

Open http://localhost:8929/root/demo-app/-/pipelines. The pipeline runs with two jobs. The first run takes longer because the images are being pulled.

At the end of the build job’s log you see (Gebaut means “built”):

INFO[0042] Pushing image to reg-build:5000/demo-app:3f9a1c2b
INFO[0043] Pushed reg-build:5000/demo-app@sha256:7d4e...
$ echo "Gebaut: $IMAGE_NAME@$(cat image-digest.txt)"
Gebaut: reg-build:5000/demo-app@sha256:7d4e...
Job succeeded

From the host, check whether the image really arrived:

crane ls localhost:5001/demo-app
crane digest localhost:5001/demo-app:"$(git rev-parse --short HEAD)"

Expected output:

1.0.0
3f9a1c2b
sha256:7d4e...

The digest matches the one in the job log. The registry is the same, addressed once via the kind network and once via port 5001.

Tip: If the length of the short hash differs (git rev-parse --short sometimes returns 7 characters, GitLab 8), take the tag from crane ls.

Step 8: Start Jenkins for comparison

Jenkins is the classic among CI servers. It is extremely flexible, but that flexibility comes through plugins. Every plugin is another dependency that has to be maintained and updated.

docker run -d --name jenkins --restart=unless-stopped \
  --network kind \
  -p 127.0.0.1:8090:8080 \
  -v jenkins-home:/var/jenkins_home \
  jenkins/jenkins:lts-jdk21

sleep 30
docker exec jenkins cat /var/jenkins_home/secrets/initialAdminPassword

Expected output:

3c5d0b... (32 Zeichen)

Open http://localhost:8090, enter the password (32 characters), choose “Install suggested plugins” and create an admin user. The suggested plugins include “Pipeline” and “Git”.

Step 9: A declarative Jenkinsfile

Create a Jenkinsfile in the repo. It is deliberately small and runs on the Jenkins controller without Python or Docker. It is only meant to show the structure.

cd ~/seclab/demo-app
cat > Jenkinsfile <<'EOF'
pipeline {
  agent any
  options {
    timeout(time: 10, unit: 'MINUTES')
    disableConcurrentBuilds()
    buildDiscarder(logRotator(numToKeepStr: '20'))
  }
  stages {
    stage('Dockerfile-Check') {
      steps {
        sh 'test -f Dockerfile'
        // Mini rule: the image must run as a non-root user
        sh 'grep -q "^USER 10001" Dockerfile'
      }
    }
    stage('Info') {
      steps {
        sh 'git log -1 --oneline'
        sh 'sha256sum requirements.txt'
      }
    }
  }
  post {
    always {
      echo "Build ${env.BUILD_NUMBER}: ${currentBuild.currentResult}"
    }
  }
}
EOF
git add Jenkinsfile
git commit -m "ci: Jenkinsfile zum Vergleich"
git push

In Jenkins:

  1. “Manage Jenkins”, “Credentials”, “System”, “Global credentials”, “Add Credentials”. Type “Username with password”, user root, password = your glpat- token, ID gitlab-pat.
  2. “New Item”, name demo-app, type “Pipeline”.
  3. Under “Pipeline”: Definition “Pipeline script from SCM”, SCM “Git”, Repository URL http://gitlab:8929/root/demo-app.git, Credentials gitlab-pat, Branch */main, Script Path Jenkinsfile.
  4. Save, “Build Now”.

Expected console output (shortened):

[Pipeline] stage (Dockerfile-Check)
+ test -f Dockerfile
+ grep -q ^USER 10001 Dockerfile
[Pipeline] stage (Info)
+ git log -1 --oneline
b7e2f10 ci: Jenkinsfile zum Vergleich
...
Build 1: SUCCESS
Finished: SUCCESS

Security: agent any lets the build run on the controller here. That is acceptable only in the lab. In production you set the controller’s executors to 0 (“Manage Jenkins”, “Nodes”, “Built-In Node”) and let builds run only on agents, for example with the Kubernetes plugin as disposable pods (agent { kubernetes { ... } }).

GitLab CI and Jenkins compared:

Aspect GitLab CI Jenkins
Pipeline definition YAML (.gitlab-ci.yml) Groovy (Jenkinsfile)
Integration Git, CI, registry in one via plugins
Extension components, templates over 1,800 plugins
Hardening focus runners, variables, branches plugins, controller, credentials

Step 10: Install Argo CD in the cluster

Now the CD side. Argo CD runs in the cluster in the namespace argocd.

kubectl create namespace argocd
kubectl apply -n argocd --server-side --force-conflicts \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
kubectl -n argocd wait --for=condition=available deployment --all --timeout=300s
kubectl -n argocd get pods

Expected output (shortened):

deployment.apps/argocd-server condition met
...
NAME                                  READY   STATUS    RESTARTS   AGE
argocd-application-controller-0       1/1     Running   0          2m
argocd-repo-server-6c8f...            1/1     Running   0          2m
argocd-server-7b9d...                 1/1     Running   0          2m
...

Why --server-side? Some Argo CD CRDs are so large that the classic client-side apply fails at the size limit for annotations. The Argo CD docs therefore recommend server-side apply.

Tip: stable always points to the latest stable version. For reproducible installations, replace stable in the URL with a specific version, such as v3.1.0. That is exactly what you do in production, and from your mirror at that (see Chapter 3).

Step 11: Log in to Argo CD

kubectl -n argocd port-forward svc/argocd-server 8443:443 >/dev/null 2>&1 &
ARGO_PF=$!
sleep 3
ARGO_PW=$(argocd admin initial-password -n argocd | head -n1)
argocd login localhost:8443 --username admin --password "$ARGO_PW" --insecure

Expected output:

'admin:login' logged in successfully
Context 'localhost:8443' updated

You reach the UI at https://localhost:8443 (accept the certificate warning, it is self-signed).

Step 12: Give Argo CD access to the repository

Argo CD runs in a pod. The cluster DNS (CoreDNS) does not reliably resolve container names from the Docker network. That is why we address GitLab by its IP address in the kind network. Pods reach this address via the node.

GITLAB_IP=$(docker inspect -f \
  '{{(index .NetworkSettings.Networks "kind").IPAddress}}' gitlab)
echo "$GITLAB_IP"

Expected output (address may differ):

172.18.0.5

For read access you create a deploy token. It belongs to the project, not to a person, and may only read. In GitLab: project demo-app, “Settings”, “Repository”, “Deploy tokens”, “Add token”. Name argocd, scope only read_repository. Note down the username (something like gitlab+deploy-token-1) and the token.

read -r DT_USER    # e.g. gitlab+deploy-token-1
read -rs DT_PASS   # token, not displayed
argocd repo add "http://${GITLAB_IP}:8929/root/demo-app.git" \
  --username "$DT_USER" --password "$DT_PASS"
unset DT_PASS
argocd repo list

Expected output:

Repository 'http://172.18.0.5:8929/root/demo-app.git' added
TYPE  NAME  REPO                                       ...  STATUS      MESSAGE
git         http://172.18.0.5:8929/root/demo-app.git   ...  Successful

Security: Argo CD stores the credentials as a Kubernetes Secret in the namespace argocd. Never put such Secrets into the Git repo as a YAML file. Chapter 6 shows how to bring credentials cleanly from Vault into the cluster.

Warning: The IP address can change after a GitLab restart. Argo CD then reports a connection error. Determine the IP again and create the repository and application with the new address.

Step 13: Create the Application

An Application tells Argo CD: “Take the k8s directory from this repo and keep the namespace demo exactly like that.”

cat > ~/seclab/kap01/argocd-app.yaml <<EOF
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: demo-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: http://${GITLAB_IP}:8929/root/demo-app.git
    targetRevision: main
    path: k8s
  destination:
    server: https://kubernetes.default.svc
    namespace: demo
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
EOF
kubectl apply -f ~/seclab/kap01/argocd-app.yaml
sleep 20
argocd app get demo-app

Expected output (shortened):

Name:               argocd/demo-app
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          demo
Repo:               http://172.18.0.5:8929/root/demo-app.git
Target:             main
Path:               k8s
Sync Policy:        Automated (Prune)
Sync Status:        Synced to main (b7e2f10)
Health Status:      Healthy

GROUP  KIND        NAMESPACE  NAME      STATUS  HEALTH   HOOK  MESSAGE
       Service     demo       demo-app  Synced  Healthy        service/demo-app ...
apps   Deployment  demo       demo-app  Synced  Healthy        deployment.apps/...

The resources from Chapter 0 already existed. Argo CD adopted them because they match the desired state. From now on, Git is the truth.

Option Effect
automated changes in Git are rolled out automatically
prune: true resources deleted in Git are deleted in the cluster
selfHeal: true manual changes in the cluster are reverted
CreateNamespace=true the namespace is created if needed

Step 14: Experience GitOps

First a change via Git. Increase the replicas to 2:

cd ~/seclab/demo-app
yq -i '.spec.replicas = 2' k8s/deployment.yaml
git commit -am "deploy: 2 Replikate"
git push
argocd app sync demo-app   # otherwise Argo CD checks about every 3 minutes
kubectl -n demo get pods

Expected output:

NAME                        READY   STATUS    RESTARTS   AGE
demo-app-6d9f8b7c5d-x2k4q   1/1     Running   0          40m
demo-app-6d9f8b7c5d-p8m3z   1/1     Running   0          10s

Now the counter-test: someone changes the cluster by hand.

kubectl -n demo scale deployment demo-app --replicas=5
sleep 15
kubectl -n demo get deployment demo-app

Expected output:

NAME       READY   UP-TO-DATE   AVAILABLE   AGE
demo-app   2/2     2            2           41m

Argo CD detected the deviation (drift) and reset it to the state from Git. Under “History and Rollback” in the UI you see every sync with its commit hash.

Remember: Anyone who wants to change something in the cluster now has to make a commit. That automatically gives you the four-eyes principle (merge request), history and easy rollback with git revert.

The bridge between CI and CD is still missing: the pipeline builds demo-app:<sha>, but the deployment points to 1.0.0. In practice, a CI job writes the new digest into the deployment repo via a commit, or a tool such as Argo CD Image Updater or Flux Image Automation takes care of it. We build this step together with signature verification in the final project (Chapter 11).

Step 15: Flux as an alternative (concept)

Flux works similarly, but without a UI. The entry point is called bootstrap: the Flux CLI installs the controllers in the cluster, writes their own manifests into a Git repo and from then on lets Flux manage itself from that repo.

flux bootstrap git
   |
   +--> installs Flux controllers in the cluster (flux-system)
   +--> commits clusters/seclab/flux-system/*.yaml to the repo
   +--> from now on, Flux pulls itself AND your apps from Git

A bootstrap against our GitLab would look roughly like this (Flux CLI e.g. via brew install fluxcd/tap/flux):

# Illustration only - do not point it at the same app in parallel with Argo CD!
flux bootstrap git \
  --url="http://${GITLAB_IP}:8929/root/fleet.git" \
  --branch=main \
  --path=clusters/seclab \
  --allow-insecure-http=true \
  --token-auth=true \
  --username=root \
  --password="$GITLAB_PAT"

You then describe the actual app with two resources:

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: demo-app
  namespace: flux-system
spec:
  interval: 1m
  url: http://172.18.0.5:8929/root/demo-app.git
  ref:
    branch: main
  secretRef:
    name: demo-app-git
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: demo-app
  namespace: flux-system
spec:
  interval: 5m
  sourceRef:
    kind: GitRepository
    name: demo-app
  path: ./k8s
  prune: true
  targetNamespace: demo

GitRepository corresponds to the repo entry in Argo CD, Kustomization to the Application. prune: true has the same meaning. With Flux, self-healing is always built in: at every interval, the desired state is applied again.

Warning: Never let two GitOps agents manage the same resources. They would overwrite each other. If you want to try Flux, use a second kind cluster.

Step 16: Clean up

GitLab and Jenkins need a lot of RAM. Stop them when you don’t need them. The volumes are preserved; later chapters use GitLab again.

kill "$ARGO_PF" 2>/dev/null
cd ~/seclab/kap01/gitlab && docker compose stop
docker stop jenkins
docker ps --format '{{.Names}}'

Expected output:

seclab-control-plane
reg-ziel
reg-build

Argo CD stays in the cluster. You start things again with docker compose start in the directory ~/seclab/kap01/gitlab and docker start jenkins. To remove everything completely, you would use docker compose down -v and docker rm -f jenkins plus docker volume rm jenkins-home.

Save your progress? Sign up for free

Common pitfalls

Symptom Cause Solution
GitLab page loads with 502 GitLab is still starting wait 3–6 minutes, docker logs -f gitlab
GitLab container keeps restarting not enough RAM give Docker 12–16 GB, stop Jenkins
network kind declared as external, but could not be found cluster from Chapter 0 missing create the cluster, then compose up
Job hangs at “Waiting for runner” runner offline or only tagged jobs set “Run untagged jobs”, docker logs gitlab-runner
Job: Could not resolve host: localhost or similar when cloning clone_url missing clone_url = "http://gitlab:8929" in config.toml
Kaniko: http: server gave HTTP response to HTTPS client --insecure missing add the flag
Kaniko: no such host reg-build job not in the kind network network_mode = "kind" in config.toml
Argo CD repo “Failed”: connection error wrong or changed GitLab IP determine the IP again, add the repo again
Argo CD “authentication required” deploy token wrong or scope missing create a new token with read_repository
kubectl apply of Argo CD: “metadata.annotations: Too long” client-side apply with large CRDs use --server-side --force-conflicts
Jenkins: “Couldn’t find any revision to build” branch */master instead of */main adjust the branch specifier

Exercise: Harden the runner

Your instance runner accepts any image that appears in a .gitlab-ci.yml. A developer could thus run an arbitrary image from the internet, such as a manipulated “helper image” that reads out secrets.

Task: Restrict the runner to exactly the images the pipeline needs: alpine:3.20, python:3.12-slim and the Kaniko image. Also make sure that images are pulled fresh from the registry for every job and that no job runs in privileged mode. Prove with a test job on a separate branch that a disallowed image (e.g. docker:27-cli) is rejected while the normal pipeline stays green.

Hint 1: All settings are in the [runners.docker] section of the file /etc/gitlab-runner/config.toml in the gitlab-runner container. The container has no editor. Copy the file to the host with docker cp, edit it there and copy it back. The runner reloads changes on its own after a few seconds.

Hint 2: The keys you are looking for are allowed_images (a list, wildcard * allowed), pull_policy and privileged. You create the test job on a branch test-allowlist. Because of its rules, the build job does not even run there.

Solution

Fetch and edit the configuration:

docker cp gitlab-runner:/etc/gitlab-runner/config.toml ~/seclab/kap01/config.toml

Extend the [runners.docker] section like this (the other lines stay):

  [runners.docker]
    image = "alpine:3.20"
    privileged = false
    network_mode = "kind"
    volumes = ["/cache"]
    pull_policy = ["always"]
    allowed_images = [
      "alpine:3.20",
      "python:3.12-slim",
      "gcr.io/kaniko-project/executor:*",
    ]

Copy it back and check:

docker cp ~/seclab/kap01/config.toml gitlab-runner:/etc/gitlab-runner/config.toml
sleep 5
docker logs --tail 5 gitlab-runner

The log shows a message like Configuration loaded. If needed, docker restart gitlab-runner helps.

Test job on a separate branch (verboten means “forbidden”):

cd ~/seclab/demo-app
git switch -c test-allowlist
cat >> .gitlab-ci.yml <<'EOF'

verboten:
  stage: test
  image: docker:27-cli
  script:
    - docker info
EOF
git commit -am "test: nicht erlaubtes Image"
git push -u origin test-allowlist

Expected result in the branch’s pipeline: test is green, verboten fails immediately with a message like:

ERROR: The "docker:27-cli" image is not present on list of allowed images:
- alpine:3.20
- python:3.12-slim
- gcr.io/kaniko-project/executor:*

Afterwards, clean up the test branch:

git switch main
git branch -D test-allowlist
git push origin --delete test-allowlist

Why this works:

  • allowed_images prevents pipelines from smuggling arbitrary third-party code in as a job image. The list is a first form of “policy as code” at the runner level (covered in detail in Chapter 9).
  • pull_policy = ["always"] ensures that a job does not use a locally cached image that may have been manipulated by another job. Once the images come from your mirror (Chapter 3), this costs hardly any time.
  • privileged = false was already set. Writing it down explicitly makes the intent visible to auditors.

Tip: It gets even stricter if you put digests instead of tags in the list, such as python@sha256:…. Then the pipeline has to use digests too. Chapter 2 picks up this topic.

Save your progress? Sign up for free

From the field

Zones and runners: In regulated environments (public authorities, critical infrastructure/KRITIS) there are usually several network zones. A common setup: GitLab sits in the internal zone, runners for normal builds next to it. For release to production there is a separate, specially protected runner (project runner, only protected branches and tags, only specific people may trigger it). The production zone itself gets no inbound access. A GitOps agent in the target zone fetches the state from a mirrored repo and the images from the target registry (see Chapter 4).

Ephemeral runners: Large organizations run runners with the Docker Autoscaler or as a Kubernetes executor in a dedicated build cluster. Every job runs in a fresh VM or a fresh pod. After the job, everything is discarded. That rules out entire classes of attacks, such as poisoned caches or leftover background processes.

Protected variables and branches: Secrets for registry push or signing are marked as “protected” and “masked”. They are available only to pipelines on protected branches. That way a feature branch cannot grab the secrets. The CI_JOB_TOKEN is restricted via the “Job token allowlist” to the projects that really need it.

Backups: For GitLab, three things belong in the backup:

docker exec -t gitlab gitlab-backup create
docker exec gitlab ls /var/opt/gitlab/backups
docker cp gitlab:/etc/gitlab/gitlab-secrets.json ~/seclab/kap01/
docker cp gitlab:/etc/gitlab/gitlab.rb ~/seclab/kap01/

gitlab-backup backs up the database and repositories. gitlab-secrets.json contains the keys GitLab uses to encrypt CI variables and two-factor data. Without this file, that data is useless after a restore. Keep it separate and encrypted. And: a backup whose restore has never been practiced is only a hope.

Updates: GitLab releases monthly, with security patches often in between. For large version jumps there are mandatory intermediate stops (“upgrade path”). Plan regular maintenance windows. With Jenkins, the plugin inventory is the biggest risk. Keep it small, use Configuration as Code (JCasC) and check the Jenkins security advisories regularly.

Argo CD in operation: After setup, you disable the local admin user and log in via SSO (OIDC). AppProject objects limit from which repos and into which namespaces a team may deploy. The metrics of Argo CD and GitLab go into Prometheus. Alerts fire, for example, for apps that have been “OutOfSync” or “Degraded” for more than 15 minutes.

Audit: Auditors typically ask: Who may change pipelines? Who may deploy to production? How is it proven which image comes from which commit? With GitOps you answer the last two questions from the Git history. You answer the first with protected branches, CODEOWNERS for .gitlab-ci.yml and mandatory reviews.

Interview questions

Q: What is the difference between push-based and pull-based deployment?

A: In the push model, the CI pipeline performs the deployment and needs credentials for the cluster to do so. In the pull model (GitOps), an agent such as Argo CD or Flux runs in the cluster, watches a Git repo and reconciles the cluster. Advantages of pull: no cluster credentials in CI, no inbound network path needed, drift detection and Git as an audit log.

Q: Why do you avoid Docker-in-Docker on shared runners?

A: DinD needs privileged containers. With that, a job can break out of the container onto the host and take over other jobs, their secrets and the runner. Better are tools such as Kaniko, Buildah or rootless BuildKit that build without privileges, or a dedicated disposable VM per job.

Q: How do you harden a GitLab Runner?

A: Separate runners by protection level, sensitive runners only for protected branches. privileged = false, no Docker socket in jobs, allowed_images and pull_policy = always. Ephemeral environments per job wherever possible. Patch and monitor the runner host itself. Registration via runner authentication tokens, which are revocable.

Q: What is an ephemeral runner and what problem does it solve?

A: A runner or execution environment that exists for only one job and is discarded afterwards. It prevents a job from leaving traces for the next one, such as manipulated caches, images or running processes. That limits the damage of a compromised job to exactly that job.

Q: When would you run CI/CD yourself instead of using SaaS?

A: When source code or artifacts must not leave your own infrastructure, when target environments are separated from the internet (air-gapped), or when regulatory requirements demand control over the build environment. The price is operational effort: updates, backups, monitoring, hardening.

Q: What belongs in a GitLab backup?

A: The result of gitlab-backup create (database, repositories, uploads), plus gitlab-secrets.json and gitlab.rb. Without gitlab-secrets.json, encrypted data such as CI variables cannot be restored. Test the restore regularly.

Q: Argo CD or Flux – how do you decide?

A: Both are mature CNCF projects. Argo CD offers a strong web UI and AppProject for multi-tenancy, which makes it easier for teams to get started. Flux is leaner, purely CRD-based and good for platform teams that manage everything as code. What matters is existing know-how, the tenancy model and integration with SSO and monitoring.

Q: How do secrets get into a GitOps workflow without ending up in Git?

A: Git contains only a reference, not the secret itself. The External Secrets Operator fetches the value at runtime from Vault or OpenBao. Alternatively, encrypted secrets with Sealed Secrets or SOPS. Details in Chapter 6.

Summary

What you learned:

  • CI builds and tests, CD delivers the result. In the pull model (GitOps), an agent in the cluster fetches the desired state from Git; CI needs no cluster access.
  • The runner is the most critical part of the chain because it runs third-party code with access to secrets. Isolation, no privileges, an image allowlist and ephemeral environments are the most important protective measures.
  • Today you register GitLab runners with a runner authentication token (glrt-…) generated in the UI.
  • Kaniko builds images without a Docker daemon and without a privileged container. You save the digest of the result as a pipeline artifact.
  • Jenkins defines pipelines declaratively in the Jenkinsfile. Hardening there mainly means: a controller without builds, few plugins, agents as disposable pods.
  • Argo CD reconciles the cluster with the k8s/ directory in the repo, rolls out changes automatically and reverts manual changes via self-heal.
  • Flux pursues the same goal with GitRepository and Kustomization and manages itself from Git after the bootstrap.
  • Running it yourself means responsibility: backups including gitlab-secrets.json, regular updates, monitoring and traceable permissions.