Chapter 1: Running CI/CD chains yourself (GitLab CI, Jenkins, Argo CD/Flux)
In this chapter you will learn:
- the difference between push deployment and pull deployment (GitOps)
- how the runner or agent model works and why it is the biggest risk in the chain
- when self-hosted CI/CD makes sense and how you harden it
- how to set up GitLab CE with your own runner and build the demo-app without Docker-in-Docker
- how Argo CD brings the app from Git into the cluster and repairs deviations on its own
What is it about?
Picture a bakery. The recipe is in the office (Git). In the bakehouse, bakers bake according to the recipe (runners). The finished loaves go to the warehouse (registry). A driver takes them to the branch stores (deployment).
Now the security question: who is allowed into the bakehouse? If every supplier has a master key, anyone can mix something into the dough. And the driver: does he get keys to all the branch stores, or do the stores pick up their bread themselves?
That is exactly what this chapter is about. A CI/CD chain is a factory for software. If you run it yourself, you decide on keys, doors and routes yourself. That is work, but it is also control.
Developer CI (build zone) CD (target)
+-----------+ git push +------------------+ push +--------------+
| Laptop |----------->| GitLab CE :8929 |-------->| reg-build |
+-----------+ | .gitlab-ci.yml | image | :5001/:5000 |
+--------+---------+ +------+-------+
| job |
v | pull
+------------------+ |
| gitlab-runner | |
| (Docker executor)| |
| Job: test, kaniko| |
+------------------+ |
v
Git repo (k8s/) <---------- watches (pull) --------- +---------------+
"desired state" | Argo CD in |
| kind cluster |
| -> namespace |
| demo |
+---------------+
The CI side builds and tests. The CD side brings the result into the cluster. In our lab these are two separate systems with separate permissions. That is no accident but a security principle.
The concepts made simple
CI and CD
- Continuous Integration (CI): Every commit is automatically built and tested. The result is an artifact, in our case a container image.
- Continuous Delivery/Deployment (CD): The artifact is automatically rolled out to an environment. With “delivery”, a human approves; with “deployment”, it runs fully automatically.
Push vs. pull (GitOps)
In the push model, the pipeline actively pushes the
result into the cluster. At the end of the CI job there is something
like kubectl apply or helm upgrade. For this,
the pipeline needs credentials for the cluster.
In the pull model, an agent runs in the cluster. It watches a Git repository with the desired state and continuously reconciles the cluster against it. This approach is called GitOps: Git is the single source of truth.
| Aspect | Push (CI deploys) | Pull (GitOps) |
|---|---|---|
| Cluster access | CI has credentials | only the agent in the cluster |
| Firewall direction | CI → cluster inbound | cluster → Git outbound |
| Deviations (drift) | go unnoticed | are detected and repaired |
| Audit | pipeline logs | Git history |
Remember: With the pull model, no path into the production zone has to be open. The agent fetches what it needs. For zone models in public authorities and critical infrastructure (KRITIS) environments, that is a big advantage.
GitOps is based on four principles (OpenGitOps): the state is described declaratively, stored versioned and immutable, pulled automatically and continuously reconciled.
The runner or agent model
GitLab itself does not run any jobs. It distributes them to runners. A runner is a program that regularly asks: “Is there work for me?” It then runs the job and reports the result back. In Jenkins, the executing parts are called agents (formerly “slaves”), and the central server is the controller.
The runner chooses how a job runs via the executor:
| Executor | How the job runs | Isolation |
|---|---|---|
| shell | directly on the runner host | weak |
| docker | in a fresh container | medium |
| kubernetes | as a pod in the cluster | medium |
| docker-autoscaler / VM | in a fresh VM per job | strong |
Security: The runner is the most dangerous part of the chain. It runs code that anyone who may change a
.gitlab-ci.ymlcan write. It sees secrets and may write to registries. Whoever takes over a runner can manipulate any artifact without touching the source code.
Runners come in three scopes:
- Instance runner (formerly “shared runner”): for all projects of the instance.
- Group runner: for all projects of a group.
- Project runner: for one project only.
The larger the scope, the greater the damage from a compromise. A runner shared by 300 projects can be attacked from any of those 300 projects.
The problem with Docker-in-Docker
Many guides build images in CI with “Docker-in-Docker” (DinD). For
this, the job container needs privileged = true. A
privileged container has practically all the rights of the host. A
single malicious job can thus break out of the container and take over
other jobs, their secrets and the runner itself.
The alternative is build tools that work without a Docker daemon and without privileges:
| Tool | Principle | Note |
|---|---|---|
| Kaniko | builds in the user space of a normal container | Google project archived in 2025, forks exist |
| Buildah | daemonless, OCI-native (Red Hat) | rootless in a container needs fine-tuning |
| BuildKit rootless | Docker build engine without root | needs adjustments to seccomp/AppArmor |
In the lab we use Kaniko. It runs without any extra configuration in a normal Docker executor. In Chapter 2 you assess the build processes in more detail.
Warning: Google archived the Kaniko repository in 2025. The image
gcr.io/kaniko-project/executoris still available but no longer receives updates. For a new production environment, evaluate a maintained fork or Buildah/BuildKit. For the principle of “building without privileges”, Kaniko remains a good teaching example in the lab.
Self-hosted or SaaS?
| Criterion | SaaS (e.g. gitlab.com) | Self-hosted |
|---|---|---|
| Operational effort | low | high (updates, backups) |
| Data sovereignty | with the provider | with you |
| Network | internet required | air-gapped also possible |
| Customizability | limited | full |
Running it yourself pays off when data must not leave the building, when the production environment is separated from the internet, or when regulatory requirements (e.g. BSI IT-Grundschutz, the German KRITIS regulation) demand control over the build environment. A hybrid form is common: SaaS UI, but your own runners in your own network.
Hardening in five points
- Runner isolation: Separate runners per protection level. The runner that builds and signs production images is a project runner for protected branches only.
- No privileged shared runners:
privileged = false, no mounted Docker socket in jobs, a list of allowed images (allowed_images). - Ephemeral runners: Each job gets a fresh environment that is discarded afterwards. No job can leave anything behind for the next one.
- Backups: Database, repositories and key
files. With GitLab, a backup without
gitlab-secrets.jsonis only worth half as much. - Updates: CI servers are popular targets. Apply security updates promptly and observe upgrade paths.
Argo CD and Flux
Both are GitOps agents for Kubernetes and CNCF projects with “graduated” status.
| Feature | Argo CD | Flux |
|---|---|---|
| UI | built-in web UI | none (CLI, optional add-on UIs) |
| Configuration | CRD Application |
CRDs GitRepository, Kustomization |
| Multi-tenancy | AppProject |
namespaces + RBAC |
| Getting started | very visual | lean, very Kubernetes-native |
We install Argo CD because the UI helps while learning. We look at Flux as a concept.
Hands-on Lab: Your own CI/CD chain for the demo-app
Prerequisite: Chapter 0 is done, the cluster seclab is
running, reg-build is attached to the kind
network. GitLab needs about 4 to 6 GB of RAM. Give Docker at least 12
GB, better 16 GB.
kubectl config current-context
docker network inspect kind --format '{{range .Containers}}{{.Name}} {{end}}'Expected output:
kind-seclab
seclab-control-plane reg-build reg-ziel
Step 1: Start GitLab CE and the runner with docker compose
We attach GitLab and the runner directly to the Docker network
kind. That way the CI jobs later reach GitLab as
gitlab and the registry as reg-build:5000.
mkdir -p ~/seclab/kap01/gitlab
cat > ~/seclab/kap01/gitlab/compose.yaml <<'EOF'
services:
gitlab:
# Lab: latest. In real environments, pin a specific version!
image: gitlab/gitlab-ce:latest
container_name: gitlab
hostname: gitlab
restart: unless-stopped
shm_size: "256m"
environment:
GITLAB_OMNIBUS_CONFIG: |
external_url 'http://localhost:8929'
# Save resources in the lab
puma['worker_processes'] = 0
sidekiq['concurrency'] = 10
prometheus_monitoring['enable'] = false
ports:
- "127.0.0.1:8929:8929"
volumes:
- gitlab-config:/etc/gitlab
- gitlab-logs:/var/log/gitlab
- gitlab-data:/var/opt/gitlab
gitlab-runner:
image: gitlab/gitlab-runner:latest
container_name: gitlab-runner
restart: unless-stopped
volumes:
- runner-config:/etc/gitlab-runner
# The runner starts job containers via the host's Docker daemon.
- /var/run/docker.sock:/var/run/docker.sock
volumes:
gitlab-config:
gitlab-logs:
gitlab-data:
runner-config:
networks:
default:
name: kind
external: true
EOF
cd ~/seclab/kap01/gitlab
docker compose up -d
docker compose psExpected output (shortened):
NAME IMAGE STATUS
gitlab gitlab/gitlab-ce:latest Up 30 seconds (health: starting)
gitlab-runner gitlab/gitlab-runner:latest Up 30 seconds
Why external_url 'http://localhost:8929' and port
8929:8929? GitLab has its built-in web server listen on the
port from external_url. So inside and outside it is the
same port. That keeps the URLs in the UI correct.
Security: The runner container has access to the Docker socket. That effectively makes it root on your host. In the lab we accept that. In production, the runner runs on its own disposable VM or as a Kubernetes executor in a separate cluster.
Step 2: Wait and log in
The first start takes 3 to 6 minutes. GitLab sets up the database, Redis and all services.
until [ "$(curl -s -o /dev/null -w '%{http_code}' \
http://localhost:8929/users/sign_in)" = "200" ]; do
echo "GitLab startet noch ..."; sleep 15
done
echo "GitLab ist bereit."
docker exec gitlab grep 'Password:' /etc/gitlab/initial_root_passwordExpected output (“GitLab is still starting …” until “GitLab is ready.”):
GitLab startet noch ...
...
GitLab ist bereit.
Password: AbC...xyz=
Open http://localhost:8929 and log in as
root with this password. Change the password immediately
(avatar at the top, “Edit profile”, “Password”).
Warning: The file
initial_root_passwordis deleted automatically after 24 hours. If you missed the password, reset it withdocker exec -it gitlab gitlab-rake "gitlab:password:reset[root]".
Step 3: Create an access token and push the demo-app
For Git over HTTP, you don’t use your password but a personal access token (PAT). It is revocable, has an expiration date and only the permissions you give it.
This is how you open the form:
- Click your avatar at the top right and then “Edit profile”.
- In the left sidebar, choose “Access”, “Personal access tokens”.
- Click “Generate token” and choose “Legacy token” in the menu.
Fill in the form like this:
| Field | Value | Why |
|---|---|---|
| Token name | seclab-git |
The name tells you later what the token is for, for example when you want to revoke it |
| Token description | Git über HTTP im Lab (optional) |
Helps as soon as you have several tokens |
| Expiration date | today + 30 days | Without input, GitLab sets 365 days. Shorter is better: a forgotten token expires on its own |
| Scopes | only read_repository and
write_repository |
Cloning and pushing over HTTP, nothing else |
Leave all other scopes unchecked, especially api. It
would give the token full access to the GitLab API: deleting projects,
changing settings, creating more tokens. Then click “Generate token” at
the bottom.
GitLab shows the new token at the top of the page. It starts with
glpat-. Copy it with the copy icon next to the field and
store it in your password manager. As soon as you leave or reload the
page, you can no longer view it. Then only a new token helps.
Warning: Since GitLab 19.2, “Generate token” offers two kinds. A fine-grained token grants permissions individually per resource and project. That is the more precise choice for production, but it takes considerably more clicks. In the lab you use the legacy token with the classic scopes. Older GitLab versions show the “Add new token” button with the same form instead of the menu.
Tip: In Chapter 11 you fetch build artifacts via the API and need the
read_apiscope for that. Create a second token then, instead of giving this one more permissions. One token per purpose can be revoked individually.
Now you push the demo-app to GitLab. GitLab creates the project automatically on the first push (“push to create”, private):
cd ~/seclab/demo-app
git remote add origin http://localhost:8929/root/demo-app.git
git push -u origin mainEnter root as the username and the token as the
password.
Expected output (shortened):
remote: The private project root/demo-app was successfully created.
...
* [new branch] main -> main
branch 'main' set up to track 'origin/main'.
Check in the UI under
http://localhost:8929/root/demo-app that the files are
there.
Step 4: Create and register the runner
Since GitLab 16, registration works with a runner
authentication token (starts with glrt-). You
create the runner in the UI first. The old registration tokens are
deprecated.
In GitLab: “Admin” (bottom left), “CI/CD”, “Runners”, “New instance
runner”. Check “Run untagged jobs” and describe the runner as
seclab-runner. Click “Create runner”. GitLab shows the
token glrt-… exactly once.
Tip: For a single project, a project runner would be the better choice (project, “Settings”, “CI/CD”, “Runners”). In the lab we use an instance runner so that it is also available for other practice projects later.
Register:
read -rs RUNNER_TOKEN # paste the token, Enter (not displayed)
docker exec gitlab-runner gitlab-runner register \
--non-interactive \
--url "http://gitlab:8929" \
--token "$RUNNER_TOKEN" \
--executor "docker" \
--docker-image "alpine:3.20" \
--docker-network-mode "kind" \
--clone-url "http://gitlab:8929" \
--description "seclab-runner"
unset RUNNER_TOKENExpected output (shortened):
Verifying runner... is valid runner=xxxxxxx
Runner registered successfully. Feel free to start it, ...
The two important options for our lab:
--docker-network-mode kind: job containers start in thekindnetwork. There they findgitlabandreg-build.--clone-url http://gitlab:8929: jobs clone via the container name. Theexternal_url(localhost) would point to the job itself inside the job container.
Step 5: Check the runner configuration
docker exec gitlab-runner cat /etc/gitlab-runner/config.tomlExpected output (excerpt):
[[runners]]
name = "seclab-runner"
url = "http://gitlab:8929"
token = "glrt-..."
executor = "docker"
clone_url = "http://gitlab:8929"
[runners.docker]
image = "alpine:3.20"
privileged = false
network_mode = "kind"
volumes = ["/cache"]Watch for two lines: privileged = false and
volumes = ["/cache"]. The Docker socket is
not mounted into the jobs. That is exactly how it
should be.
In the UI, the runner now appears under “Admin”, “CI/CD”, “Runners” with a green dot (“Online”).
Step 6: Write the pipeline
The pipeline has two stages. test runs the unit tests.
build builds the image with Kaniko and pushes it to
reg-build.
cd ~/seclab/demo-app
cat > .gitlab-ci.yml <<'EOF'
stages:
- test
- build
variables:
# Cluster/network address of the build registry (Docker network "kind")
IMAGE_NAME: "reg-build:5000/demo-app"
test:
stage: test
image: python:3.12-slim
script:
- pip install --no-cache-dir -r requirements.txt -r requirements-dev.txt
- python -m pytest -q
build:
stage: build
image:
name: gcr.io/kaniko-project/executor:v1.23.2-debug
entrypoint: [""]
script:
- /kaniko/executor
--context "$CI_PROJECT_DIR"
--dockerfile "$CI_PROJECT_DIR/Dockerfile"
--destination "$IMAGE_NAME:$CI_COMMIT_SHORT_SHA"
--digest-file "$CI_PROJECT_DIR/image-digest.txt"
--insecure
- echo "Gebaut: $IMAGE_NAME@$(cat image-digest.txt)"
artifacts:
paths:
- image-digest.txt
expire_in: 1 week
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
EOFWhat happens here:
| Part | Meaning |
|---|---|
-debug image |
contains a shell, which GitLab needs for script |
entrypoint: [""] |
overrides the Kaniko start command so that GitLab can take control |
--insecure |
allows pushing to an http registry (lab only) |
--digest-file |
writes the digest of the built image to a file |
rules |
builds only on the default branch |
tag $CI_COMMIT_SHORT_SHA |
each image is uniquely tied to a commit |
Remember: The digest as a pipeline artifact is worth its weight in gold. Later steps such as signing (Chapter 5) or SBOM generation (Chapter 7) work with exactly this digest. That way you are guaranteed to sign the image you built, and not one that a tag happens to point to right now.
Commit and push:
git add .gitlab-ci.yml
git commit -m "ci: test- und build-Stufe mit Kaniko"
git pushStep 7: Watch the pipeline and check the result
Open http://localhost:8929/root/demo-app/-/pipelines.
The pipeline runs with two jobs. The first run takes longer because the
images are being pulled.
At the end of the build job’s log you see
(Gebaut means “built”):
INFO[0042] Pushing image to reg-build:5000/demo-app:3f9a1c2b
INFO[0043] Pushed reg-build:5000/demo-app@sha256:7d4e...
$ echo "Gebaut: $IMAGE_NAME@$(cat image-digest.txt)"
Gebaut: reg-build:5000/demo-app@sha256:7d4e...
Job succeeded
From the host, check whether the image really arrived:
crane ls localhost:5001/demo-app
crane digest localhost:5001/demo-app:"$(git rev-parse --short HEAD)"Expected output:
1.0.0
3f9a1c2b
sha256:7d4e...
The digest matches the one in the job log. The registry is the same,
addressed once via the kind network and once via port
5001.
Tip: If the length of the short hash differs (
git rev-parse --shortsometimes returns 7 characters, GitLab 8), take the tag fromcrane ls.
Step 8: Start Jenkins for comparison
Jenkins is the classic among CI servers. It is extremely flexible, but that flexibility comes through plugins. Every plugin is another dependency that has to be maintained and updated.
docker run -d --name jenkins --restart=unless-stopped \
--network kind \
-p 127.0.0.1:8090:8080 \
-v jenkins-home:/var/jenkins_home \
jenkins/jenkins:lts-jdk21
sleep 30
docker exec jenkins cat /var/jenkins_home/secrets/initialAdminPasswordExpected output:
3c5d0b... (32 Zeichen)
Open http://localhost:8090, enter the password (32
characters), choose “Install suggested plugins” and create an admin
user. The suggested plugins include “Pipeline” and “Git”.
Step 9: A declarative Jenkinsfile
Create a Jenkinsfile in the repo. It is deliberately
small and runs on the Jenkins controller without Python or Docker. It is
only meant to show the structure.
cd ~/seclab/demo-app
cat > Jenkinsfile <<'EOF'
pipeline {
agent any
options {
timeout(time: 10, unit: 'MINUTES')
disableConcurrentBuilds()
buildDiscarder(logRotator(numToKeepStr: '20'))
}
stages {
stage('Dockerfile-Check') {
steps {
sh 'test -f Dockerfile'
// Mini rule: the image must run as a non-root user
sh 'grep -q "^USER 10001" Dockerfile'
}
}
stage('Info') {
steps {
sh 'git log -1 --oneline'
sh 'sha256sum requirements.txt'
}
}
}
post {
always {
echo "Build ${env.BUILD_NUMBER}: ${currentBuild.currentResult}"
}
}
}
EOF
git add Jenkinsfile
git commit -m "ci: Jenkinsfile zum Vergleich"
git pushIn Jenkins:
- “Manage Jenkins”, “Credentials”, “System”, “Global credentials”,
“Add Credentials”. Type “Username with password”, user
root, password = yourglpat-token, IDgitlab-pat. - “New Item”, name
demo-app, type “Pipeline”. - Under “Pipeline”: Definition “Pipeline script from SCM”, SCM “Git”,
Repository URL
http://gitlab:8929/root/demo-app.git, Credentialsgitlab-pat, Branch*/main, Script PathJenkinsfile. - Save, “Build Now”.
Expected console output (shortened):
[Pipeline] stage (Dockerfile-Check)
+ test -f Dockerfile
+ grep -q ^USER 10001 Dockerfile
[Pipeline] stage (Info)
+ git log -1 --oneline
b7e2f10 ci: Jenkinsfile zum Vergleich
...
Build 1: SUCCESS
Finished: SUCCESS
Security:
agent anylets the build run on the controller here. That is acceptable only in the lab. In production you set the controller’s executors to 0 (“Manage Jenkins”, “Nodes”, “Built-In Node”) and let builds run only on agents, for example with the Kubernetes plugin as disposable pods (agent { kubernetes { ... } }).
GitLab CI and Jenkins compared:
| Aspect | GitLab CI | Jenkins |
|---|---|---|
| Pipeline definition | YAML (.gitlab-ci.yml) |
Groovy (Jenkinsfile) |
| Integration | Git, CI, registry in one | via plugins |
| Extension | components, templates | over 1,800 plugins |
| Hardening focus | runners, variables, branches | plugins, controller, credentials |
Step 10: Install Argo CD in the cluster
Now the CD side. Argo CD runs in the cluster in the namespace
argocd.
kubectl create namespace argocd
kubectl apply -n argocd --server-side --force-conflicts \
-f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
kubectl -n argocd wait --for=condition=available deployment --all --timeout=300s
kubectl -n argocd get podsExpected output (shortened):
deployment.apps/argocd-server condition met
...
NAME READY STATUS RESTARTS AGE
argocd-application-controller-0 1/1 Running 0 2m
argocd-repo-server-6c8f... 1/1 Running 0 2m
argocd-server-7b9d... 1/1 Running 0 2m
...
Why --server-side? Some Argo CD CRDs are so large that
the classic client-side apply fails at the size limit for annotations.
The Argo CD docs therefore recommend server-side apply.
Tip:
stablealways points to the latest stable version. For reproducible installations, replacestablein the URL with a specific version, such asv3.1.0. That is exactly what you do in production, and from your mirror at that (see Chapter 3).
Step 11: Log in to Argo CD
kubectl -n argocd port-forward svc/argocd-server 8443:443 >/dev/null 2>&1 &
ARGO_PF=$!
sleep 3
ARGO_PW=$(argocd admin initial-password -n argocd | head -n1)
argocd login localhost:8443 --username admin --password "$ARGO_PW" --insecureExpected output:
'admin:login' logged in successfully
Context 'localhost:8443' updated
You reach the UI at https://localhost:8443 (accept the
certificate warning, it is self-signed).
Step 12: Give Argo CD access to the repository
Argo CD runs in a pod. The cluster DNS (CoreDNS) does not reliably
resolve container names from the Docker network. That is why we address
GitLab by its IP address in the kind network. Pods reach
this address via the node.
GITLAB_IP=$(docker inspect -f \
'{{(index .NetworkSettings.Networks "kind").IPAddress}}' gitlab)
echo "$GITLAB_IP"Expected output (address may differ):
172.18.0.5
For read access you create a deploy token. It
belongs to the project, not to a person, and may only read. In GitLab:
project demo-app, “Settings”, “Repository”, “Deploy
tokens”, “Add token”. Name argocd, scope only
read_repository. Note down the username (something like
gitlab+deploy-token-1) and the token.
read -r DT_USER # e.g. gitlab+deploy-token-1
read -rs DT_PASS # token, not displayed
argocd repo add "http://${GITLAB_IP}:8929/root/demo-app.git" \
--username "$DT_USER" --password "$DT_PASS"
unset DT_PASS
argocd repo listExpected output:
Repository 'http://172.18.0.5:8929/root/demo-app.git' added
TYPE NAME REPO ... STATUS MESSAGE
git http://172.18.0.5:8929/root/demo-app.git ... Successful
Security: Argo CD stores the credentials as a Kubernetes Secret in the namespace
argocd. Never put such Secrets into the Git repo as a YAML file. Chapter 6 shows how to bring credentials cleanly from Vault into the cluster.
Warning: The IP address can change after a GitLab restart. Argo CD then reports a connection error. Determine the IP again and create the repository and application with the new address.
Step 13: Create the Application
An Application tells Argo CD: “Take the k8s
directory from this repo and keep the namespace demo
exactly like that.”
cat > ~/seclab/kap01/argocd-app.yaml <<EOF
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: demo-app
namespace: argocd
spec:
project: default
source:
repoURL: http://${GITLAB_IP}:8929/root/demo-app.git
targetRevision: main
path: k8s
destination:
server: https://kubernetes.default.svc
namespace: demo
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
EOF
kubectl apply -f ~/seclab/kap01/argocd-app.yaml
sleep 20
argocd app get demo-appExpected output (shortened):
Name: argocd/demo-app
Project: default
Server: https://kubernetes.default.svc
Namespace: demo
Repo: http://172.18.0.5:8929/root/demo-app.git
Target: main
Path: k8s
Sync Policy: Automated (Prune)
Sync Status: Synced to main (b7e2f10)
Health Status: Healthy
GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE
Service demo demo-app Synced Healthy service/demo-app ...
apps Deployment demo demo-app Synced Healthy deployment.apps/...
The resources from Chapter 0 already existed. Argo CD adopted them because they match the desired state. From now on, Git is the truth.
| Option | Effect |
|---|---|
automated |
changes in Git are rolled out automatically |
prune: true |
resources deleted in Git are deleted in the cluster |
selfHeal: true |
manual changes in the cluster are reverted |
CreateNamespace=true |
the namespace is created if needed |
Step 14: Experience GitOps
First a change via Git. Increase the replicas to 2:
cd ~/seclab/demo-app
yq -i '.spec.replicas = 2' k8s/deployment.yaml
git commit -am "deploy: 2 Replikate"
git push
argocd app sync demo-app # otherwise Argo CD checks about every 3 minutes
kubectl -n demo get podsExpected output:
NAME READY STATUS RESTARTS AGE
demo-app-6d9f8b7c5d-x2k4q 1/1 Running 0 40m
demo-app-6d9f8b7c5d-p8m3z 1/1 Running 0 10s
Now the counter-test: someone changes the cluster by hand.
kubectl -n demo scale deployment demo-app --replicas=5
sleep 15
kubectl -n demo get deployment demo-appExpected output:
NAME READY UP-TO-DATE AVAILABLE AGE
demo-app 2/2 2 2 41m
Argo CD detected the deviation (drift) and reset it to the state from Git. Under “History and Rollback” in the UI you see every sync with its commit hash.
Remember: Anyone who wants to change something in the cluster now has to make a commit. That automatically gives you the four-eyes principle (merge request), history and easy rollback with
git revert.
The bridge between CI and CD is still missing: the pipeline builds
demo-app:<sha>, but the deployment points to
1.0.0. In practice, a CI job writes the new digest into the
deployment repo via a commit, or a tool such as Argo CD Image Updater or
Flux Image Automation takes care of it. We build this step together with
signature verification in the final project (Chapter 11).
Step 15: Flux as an alternative (concept)
Flux works similarly, but without a UI. The entry point is called bootstrap: the Flux CLI installs the controllers in the cluster, writes their own manifests into a Git repo and from then on lets Flux manage itself from that repo.
flux bootstrap git
|
+--> installs Flux controllers in the cluster (flux-system)
+--> commits clusters/seclab/flux-system/*.yaml to the repo
+--> from now on, Flux pulls itself AND your apps from Git
A bootstrap against our GitLab would look roughly like this (Flux CLI
e.g. via brew install fluxcd/tap/flux):
# Illustration only - do not point it at the same app in parallel with Argo CD!
flux bootstrap git \
--url="http://${GITLAB_IP}:8929/root/fleet.git" \
--branch=main \
--path=clusters/seclab \
--allow-insecure-http=true \
--token-auth=true \
--username=root \
--password="$GITLAB_PAT"You then describe the actual app with two resources:
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: demo-app
namespace: flux-system
spec:
interval: 1m
url: http://172.18.0.5:8929/root/demo-app.git
ref:
branch: main
secretRef:
name: demo-app-git
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: demo-app
namespace: flux-system
spec:
interval: 5m
sourceRef:
kind: GitRepository
name: demo-app
path: ./k8s
prune: true
targetNamespace: demoGitRepository corresponds to the repo entry in Argo CD,
Kustomization to the Application.
prune: true has the same meaning. With Flux, self-healing
is always built in: at every interval, the desired state is applied
again.
Warning: Never let two GitOps agents manage the same resources. They would overwrite each other. If you want to try Flux, use a second kind cluster.
Step 16: Clean up
GitLab and Jenkins need a lot of RAM. Stop them when you don’t need them. The volumes are preserved; later chapters use GitLab again.
kill "$ARGO_PF" 2>/dev/null
cd ~/seclab/kap01/gitlab && docker compose stop
docker stop jenkins
docker ps --format '{{.Names}}'Expected output:
seclab-control-plane
reg-ziel
reg-build
Argo CD stays in the cluster. You start things again with
docker compose start in the directory
~/seclab/kap01/gitlab and
docker start jenkins. To remove everything completely, you
would use docker compose down -v and
docker rm -f jenkins plus
docker volume rm jenkins-home.
Save your progress? Sign up for free
Common pitfalls
| Symptom | Cause | Solution |
|---|---|---|
| GitLab page loads with 502 | GitLab is still starting | wait 3–6 minutes, docker logs -f gitlab |
| GitLab container keeps restarting | not enough RAM | give Docker 12–16 GB, stop Jenkins |
network kind declared as external, but could not be found |
cluster from Chapter 0 missing | create the cluster, then compose up |
| Job hangs at “Waiting for runner” | runner offline or only tagged jobs | set “Run untagged jobs”, docker logs gitlab-runner |
Job: Could not resolve host: localhost or similar when
cloning |
clone_url missing |
clone_url = "http://gitlab:8929" in
config.toml |
Kaniko:
http: server gave HTTP response to HTTPS client |
--insecure missing |
add the flag |
Kaniko: no such host reg-build |
job not in the kind network |
network_mode = "kind" in config.toml |
| Argo CD repo “Failed”: connection error | wrong or changed GitLab IP | determine the IP again, add the repo again |
| Argo CD “authentication required” | deploy token wrong or scope missing | create a new token with read_repository |
kubectl apply of Argo CD: “metadata.annotations: Too
long” |
client-side apply with large CRDs | use --server-side --force-conflicts |
| Jenkins: “Couldn’t find any revision to build” | branch */master instead of */main |
adjust the branch specifier |
Exercise: Harden the runner
Your instance runner accepts any image that appears in a
.gitlab-ci.yml. A developer could thus run an arbitrary
image from the internet, such as a manipulated “helper image” that reads
out secrets.
Task: Restrict the runner to exactly the images the
pipeline needs: alpine:3.20, python:3.12-slim
and the Kaniko image. Also make sure that images are pulled fresh from
the registry for every job and that no job runs in privileged mode.
Prove with a test job on a separate branch that a disallowed image
(e.g. docker:27-cli) is rejected while the normal pipeline
stays green.
Hint 1: All settings are in the
[runners.docker] section of the file
/etc/gitlab-runner/config.toml in the
gitlab-runner container. The container has no editor. Copy
the file to the host with docker cp, edit it there and copy
it back. The runner reloads changes on its own after a few seconds.
Hint 2: The keys you are looking for are
allowed_images (a list, wildcard * allowed),
pull_policy and privileged. You create the
test job on a branch test-allowlist. Because of its
rules, the build job does not even run
there.
Solution
Fetch and edit the configuration:
docker cp gitlab-runner:/etc/gitlab-runner/config.toml ~/seclab/kap01/config.tomlExtend the [runners.docker] section like this (the other
lines stay):
[runners.docker]
image = "alpine:3.20"
privileged = false
network_mode = "kind"
volumes = ["/cache"]
pull_policy = ["always"]
allowed_images = [
"alpine:3.20",
"python:3.12-slim",
"gcr.io/kaniko-project/executor:*",
]Copy it back and check:
docker cp ~/seclab/kap01/config.toml gitlab-runner:/etc/gitlab-runner/config.toml
sleep 5
docker logs --tail 5 gitlab-runnerThe log shows a message like Configuration loaded. If
needed, docker restart gitlab-runner helps.
Test job on a separate branch (verboten means
“forbidden”):
cd ~/seclab/demo-app
git switch -c test-allowlist
cat >> .gitlab-ci.yml <<'EOF'
verboten:
stage: test
image: docker:27-cli
script:
- docker info
EOF
git commit -am "test: nicht erlaubtes Image"
git push -u origin test-allowlistExpected result in the branch’s pipeline: test is green,
verboten fails immediately with a message like:
ERROR: The "docker:27-cli" image is not present on list of allowed images:
- alpine:3.20
- python:3.12-slim
- gcr.io/kaniko-project/executor:*
Afterwards, clean up the test branch:
git switch main
git branch -D test-allowlist
git push origin --delete test-allowlistWhy this works:
allowed_imagesprevents pipelines from smuggling arbitrary third-party code in as a job image. The list is a first form of “policy as code” at the runner level (covered in detail in Chapter 9).pull_policy = ["always"]ensures that a job does not use a locally cached image that may have been manipulated by another job. Once the images come from your mirror (Chapter 3), this costs hardly any time.privileged = falsewas already set. Writing it down explicitly makes the intent visible to auditors.
Tip: It gets even stricter if you put digests instead of tags in the list, such as
python@sha256:…. Then the pipeline has to use digests too. Chapter 2 picks up this topic.
Save your progress? Sign up for free
From the field
Zones and runners: In regulated environments (public authorities, critical infrastructure/KRITIS) there are usually several network zones. A common setup: GitLab sits in the internal zone, runners for normal builds next to it. For release to production there is a separate, specially protected runner (project runner, only protected branches and tags, only specific people may trigger it). The production zone itself gets no inbound access. A GitOps agent in the target zone fetches the state from a mirrored repo and the images from the target registry (see Chapter 4).
Ephemeral runners: Large organizations run runners with the Docker Autoscaler or as a Kubernetes executor in a dedicated build cluster. Every job runs in a fresh VM or a fresh pod. After the job, everything is discarded. That rules out entire classes of attacks, such as poisoned caches or leftover background processes.
Protected variables and branches: Secrets for
registry push or signing are marked as “protected” and “masked”. They
are available only to pipelines on protected branches. That way a
feature branch cannot grab the secrets. The CI_JOB_TOKEN is
restricted via the “Job token allowlist” to the projects that really
need it.
Backups: For GitLab, three things belong in the backup:
docker exec -t gitlab gitlab-backup create
docker exec gitlab ls /var/opt/gitlab/backups
docker cp gitlab:/etc/gitlab/gitlab-secrets.json ~/seclab/kap01/
docker cp gitlab:/etc/gitlab/gitlab.rb ~/seclab/kap01/gitlab-backup backs up the database and repositories.
gitlab-secrets.json contains the keys GitLab uses to
encrypt CI variables and two-factor data. Without this file, that data
is useless after a restore. Keep it separate and encrypted. And: a
backup whose restore has never been practiced is only a hope.
Updates: GitLab releases monthly, with security patches often in between. For large version jumps there are mandatory intermediate stops (“upgrade path”). Plan regular maintenance windows. With Jenkins, the plugin inventory is the biggest risk. Keep it small, use Configuration as Code (JCasC) and check the Jenkins security advisories regularly.
Argo CD in operation: After setup, you disable the
local admin user and log in via SSO (OIDC).
AppProject objects limit from which repos and into which
namespaces a team may deploy. The metrics of Argo CD and GitLab go into
Prometheus. Alerts fire, for example, for apps that have been
“OutOfSync” or “Degraded” for more than 15 minutes.
Audit: Auditors typically ask: Who may change
pipelines? Who may deploy to production? How is it proven which image
comes from which commit? With GitOps you answer the last two questions
from the Git history. You answer the first with protected branches,
CODEOWNERS for .gitlab-ci.yml and mandatory reviews.
Interview questions
Q: What is the difference between push-based and pull-based deployment?
A: In the push model, the CI pipeline performs the deployment and needs credentials for the cluster to do so. In the pull model (GitOps), an agent such as Argo CD or Flux runs in the cluster, watches a Git repo and reconciles the cluster. Advantages of pull: no cluster credentials in CI, no inbound network path needed, drift detection and Git as an audit log.
Q: Why do you avoid Docker-in-Docker on shared runners?
A: DinD needs privileged containers. With that, a job can break out of the container onto the host and take over other jobs, their secrets and the runner. Better are tools such as Kaniko, Buildah or rootless BuildKit that build without privileges, or a dedicated disposable VM per job.
Q: How do you harden a GitLab Runner?
A: Separate runners by protection level, sensitive
runners only for protected branches. privileged = false, no
Docker socket in jobs, allowed_images and
pull_policy = always. Ephemeral environments per job
wherever possible. Patch and monitor the runner host itself.
Registration via runner authentication tokens, which are revocable.
Q: What is an ephemeral runner and what problem does it solve?
A: A runner or execution environment that exists for only one job and is discarded afterwards. It prevents a job from leaving traces for the next one, such as manipulated caches, images or running processes. That limits the damage of a compromised job to exactly that job.
Q: When would you run CI/CD yourself instead of using SaaS?
A: When source code or artifacts must not leave your own infrastructure, when target environments are separated from the internet (air-gapped), or when regulatory requirements demand control over the build environment. The price is operational effort: updates, backups, monitoring, hardening.
Q: What belongs in a GitLab backup?
A: The result of gitlab-backup create
(database, repositories, uploads), plus gitlab-secrets.json
and gitlab.rb. Without gitlab-secrets.json,
encrypted data such as CI variables cannot be restored. Test the restore
regularly.
Q: Argo CD or Flux – how do you decide?
A: Both are mature CNCF projects. Argo CD offers a
strong web UI and AppProject for multi-tenancy, which makes
it easier for teams to get started. Flux is leaner, purely CRD-based and
good for platform teams that manage everything as code. What matters is
existing know-how, the tenancy model and integration with SSO and
monitoring.
Q: How do secrets get into a GitOps workflow without ending up in Git?
A: Git contains only a reference, not the secret itself. The External Secrets Operator fetches the value at runtime from Vault or OpenBao. Alternatively, encrypted secrets with Sealed Secrets or SOPS. Details in Chapter 6.
Summary
What you learned:
- CI builds and tests, CD delivers the result. In the pull model (GitOps), an agent in the cluster fetches the desired state from Git; CI needs no cluster access.
- The runner is the most critical part of the chain because it runs third-party code with access to secrets. Isolation, no privileges, an image allowlist and ephemeral environments are the most important protective measures.
- Today you register GitLab runners with a runner authentication token
(
glrt-…) generated in the UI. - Kaniko builds images without a Docker daemon and without a privileged container. You save the digest of the result as a pipeline artifact.
- Jenkins defines pipelines declaratively in the
Jenkinsfile. Hardening there mainly means: a controller without builds, few plugins, agents as disposable pods. - Argo CD reconciles the cluster with the
k8s/directory in the repo, rolls out changes automatically and reverts manual changes via self-heal. - Flux pursues the same goal with
GitRepositoryandKustomizationand manages itself from Git after the bootstrap. - Running it yourself means responsibility: backups including
gitlab-secrets.json, regular updates, monitoring and traceable permissions.