Skip to content
Hands-on IT IT Drills

Day 0: The Training Ground

Goal: A VM with single-node k3s, the tools sops, age, helm and yq, and a private Git repo on your own Forgejo – the place where encrypted secrets land from tomorrow on. · Duration: approx. 60 min · Prerequisite: Ubuntu 24.04 VM with 4 vCPU and 8 GB RAM

Briefing

  • The whole book runs on one VM argo (172.16.0.182), user ubuntu. It is workstation and cluster at the same time. Working folder: ~/sops/tagNN/.
  • The Git repo lab/secrets lives on your own Forgejo in the cluster, not on GitHub. That way your lab sees exactly what an attacker with read access to the repo would see – and Flux and Argo CD (Days 10 and 11) clone it from inside the cluster later.
  • Every tool comes as a binary for your architecture, with a checksum, never via curl | sh from a third-party source (k3s is the one documented exception).
                 VM argo 172.16.0.182 (4 vCPU, 7.7 GiB)
 ┌─────────────────────────────────────────────────────────────────┐
 │ Workstation: sops, age, helm, yq, git   ~/sops/repo             │
 │ k3s v1.37.2  (:6443)   local-path (default StorageClass)        │
 │   forgejo  NodePort :30300   repo lab/secrets (private)         │
 └─────────────────────────────────────────────────────────────────┘

Drill 1: Check the VM

Before you install anything, you look at what is already there.

# on argo
free -m
nproc
df -h / | tail -1
ss -tlnp | grep -E ':(80|443|6443|30300|30820) ' ; echo "Treffer: $?"

Expected:

               total        used        free      shared  buff/cache   available
Mem:            7941         477        7025           2         708        7463
Swap:              0           0           0
4
/dev/sda1        77G  4.5G   72G   6% /
Treffer: 1

Treffer: 1 (“hits: 1”) means: grep found nothing, the ports for k3s (6443), Traefik (80, 443), Forgejo (30300) and OpenBao (30820, Day 7) are free.

Tip: The book’s lab VM holds leftovers from other books (a stopped PostgreSQL, ~/debug). The book does not touch them. You do not need them on your VM.

Save your progress? Sign up for free

Drill 2: Install a pinned k3s

# on argo
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.37.2+k3s1 sh -s - \
  --write-kubeconfig-mode 600 --disable metrics-server
sleep 60
sudo k3s kubectl get nodes
sudo k3s kubectl get pods -A

Expected (abridged):

[INFO]  Using v1.37.2+k3s1 as release
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  systemd: Starting k3s
NAME   STATUS   ROLES           AGE   VERSION
argo   Ready    control-plane   61s   v1.37.2+k3s1
NAMESPACE     NAME                                      READY   STATUS      RESTARTS
kube-system   coredns-577d995dff-5q7ss                  1/1     Running     0
kube-system   helm-install-gateway-api-crd-7m484        0/1     Completed   0
kube-system   helm-install-traefik-crd-4p7q8            0/1     Completed   0
kube-system   helm-install-traefik-qxln5                0/1     Completed   2
kube-system   local-path-provisioner-56c6584955-f8nz9   1/1     Running     0
kube-system   svclb-traefik-e9cd5754-h8xwp              2/2     Running     0
kube-system   traefik-68df944ff7-ws7ds                  0/1     Running     0

--write-kubeconfig-mode 600 makes the admin kubeconfig readable for root only – it grants full access to every Secret in the cluster. The book does not need metrics-server.

Save your progress? Sign up for free

Drill 3: kubeconfig and the context sops-lab

# on argo
mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown ubuntu: ~/.kube/config
chmod 600 ~/.kube/config
echo 'export KUBECONFIG=$HOME/.kube/config' >> ~/.bashrc
export KUBECONFIG=$HOME/.kube/config
kubectl config rename-context default sops-lab
kubectl config get-contexts
kubectl get nodes

Expected:

Context "default" renamed to "sops-lab".
CURRENT   NAME       CLUSTER   AUTHINFO   NAMESPACE
*         sops-lab   default   default
NAME   STATUS   ROLES           AGE   VERSION
argo   Ready    control-plane   78s   v1.37.2+k3s1

Without KUBECONFIG, the k3s kubectl always reads /etc/rancher/k3s/k3s.yaml – and as ubuntu it is not allowed to. With the variable, kubectl and Helm read the same file.

Save your progress? Sign up for free

Drill 4: sops, age, helm and yq

sops and Helm publish checksums, yq a file with several hash types. age provides no checksum file but transparency proofs (.proof, Sigsum) – here the download over HTTPS from the official release page is enough.

# on argo
mkdir -p ~/sops/tag00 && cd ~/sops/tag00
ARCH=$(dpkg --print-architecture)
S=https://github.com/getsops/sops/releases/download/v3.13.3
curl -fsSLO $S/sops-v3.13.3.linux.$ARCH
curl -fsSLO $S/sops-v3.13.3.checksums.txt
grep " sops-v3.13.3.linux.$ARCH$" sops-v3.13.3.checksums.txt | sha256sum -c
sudo install -m 0755 sops-v3.13.3.linux.$ARCH /usr/local/bin/sops
A=https://github.com/FiloSottile/age/releases/download/v1.3.2
curl -fsSLO $A/age-v1.3.2-linux-$ARCH.tar.gz
tar -xzf age-v1.3.2-linux-$ARCH.tar.gz
ls age
sudo install -m 0755 age/age age/age-keygen age/age-inspect /usr/local/bin/

Expected:

sops-v3.13.3.linux.amd64: OK
LICENSE
age
age-inspect
age-keygen
age-plugin-batchpass
age-plugin-pq
age-plugin-tag
age-plugin-tagpq

Besides age and age-keygen, age 1.3 ships a tool age-inspect and several plugins. Ubuntu 24.04 only has age 1.1.1 via apt – without age-inspect and without post-quantum keys (Day 1).

# on argo
cd ~/sops/tag00
ARCH=$(dpkg --print-architecture)
H=v4.3.2
curl -fsSLO https://get.helm.sh/helm-$H-linux-$ARCH.tar.gz
curl -fsSLO https://get.helm.sh/helm-$H-linux-$ARCH.tar.gz.sha256sum
sha256sum -c helm-$H-linux-$ARCH.tar.gz.sha256sum
tar -xzf helm-$H-linux-$ARCH.tar.gz
sudo install -m 0755 linux-$ARCH/helm /usr/local/bin/helm
Y=https://github.com/mikefarah/yq/releases/download/v4.54.3
curl -fsSLo yq $Y/yq_linux_$ARCH
curl -fsSLo yq-checksums $Y/checksums
grep "^yq_linux_$ARCH " yq-checksums | grep -c "$(sha256sum yq | cut -d' ' -f1)"
sudo install -m 0755 yq /usr/local/bin/yq
sudo apt-get install -y git jq >/dev/null
sops --version --disable-version-check
age --version
helm version --short
yq --version

Expected:

helm-v4.3.2-linux-amd64.tar.gz: OK
1
sops 3.13.3
v1.3.2
v4.3.2+g8346aaa
yq (https://github.com/mikefarah/yq/) version v4.54.3

The 1 means: the SHA-256 sum of yq is in the line for yq_linux_amd64.

Save your progress? Sign up for free

Drill 5: Forgejo in the cluster

Forgejo is a lean Git server. The rootless image exists for amd64 and arm64; SQLite as the database, data on a local-path volume, the web UI and Git over HTTP on NodePort 30300.

# on argo
cd ~/sops/tag00
cat > forgejo.yaml <<'EOF'
apiVersion: v1
kind: Namespace
metadata: {name: forgejo}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: forgejo-data, namespace: forgejo}
spec:
  accessModes: [ReadWriteOnce]
  resources: {requests: {storage: 2Gi}}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: forgejo, namespace: forgejo}
spec:
  replicas: 1
  strategy: {type: Recreate}
  selector: {matchLabels: {app: forgejo}}
  template:
    metadata: {labels: {app: forgejo}}
    spec:
      securityContext: {fsGroup: 1000}
      containers:
      - name: forgejo
        image: codeberg.org/forgejo/forgejo:17.0.2-rootless
        env:
        - {name: FORGEJO__security__INSTALL_LOCK, value: "true"}
        - {name: FORGEJO__server__ROOT_URL, value: "http://172.16.0.182:30300/"}
        - {name: FORGEJO__database__DB_TYPE, value: sqlite3}
        - {name: FORGEJO__service__DISABLE_REGISTRATION, value: "true"}
        ports: [{containerPort: 3000, name: http}]
        readinessProbe:
          httpGet: {path: /api/healthz, port: http}
        volumeMounts:
        - {name: data, mountPath: /var/lib/gitea}
        - {name: data, mountPath: /etc/gitea, subPath: conf}
      volumes:
      - name: data
        persistentVolumeClaim: {claimName: forgejo-data}
---
apiVersion: v1
kind: Service
metadata: {name: forgejo-http, namespace: forgejo}
spec:
  type: NodePort
  selector: {app: forgejo}
  ports: [{port: 3000, targetPort: http, nodePort: 30300, name: http}]
EOF
kubectl apply -f forgejo.yaml
kubectl -n forgejo rollout status deploy/forgejo --timeout=180s
kubectl -n forgejo get pods,svc

Expected:

namespace/forgejo created
persistentvolumeclaim/forgejo-data created
deployment.apps/forgejo created
service/forgejo-http created
Waiting for deployment "forgejo" rollout to finish: 0 of 1 updated replicas are ...
deployment "forgejo" successfully rolled out
NAME                           READY   STATUS    RESTARTS   AGE
pod/forgejo-7478d756d7-l4f7t   1/1     Running   0          32s

NAME                   TYPE       CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE
service/forgejo-http   NodePort   10.43.52.65   <none>        3000:30300/TCP   32s

INSTALL_LOCK=true skips the setup wizard in the browser, the FORGEJO__section__KEY variables end up in app.ini. Self-registration is off: only the admin creates users.

Save your progress? Sign up for free

Drill 6: User and private repo

# on argo
kubectl -n forgejo exec deploy/forgejo -- forgejo admin user create \
  --username lab --password lab-forgejo-123 --email [email protected] --admin \
  --must-change-password=false
curl -s -u lab:lab-forgejo-123 -H 'Content-Type: application/json' \
  -d '{"name":"secrets","private":true,"default_branch":"main"}' \
  http://172.16.0.182:30300/api/v1/user/repos | jq -r '.full_name, .clone_url'

Expected:

New user 'lab' has been successfully created!
lab/secrets
http://172.16.0.182:30300/lab/secrets.git

The password lab-forgejo-123 is a lab value. Private means: without logging in, the repo is invisible. That still does not protect secrets – everyone with read access, every fork, every backup and every CI runner has the full history.

Save your progress? Sign up for free

Drill 7: Clone and the first push

# on argo
git config --global user.name "Lab Admin"
git config --global user.email [email protected]
git config --global init.defaultBranch main
git config --global credential.helper store
printf 'http://lab:[email protected]:30300\n' > ~/.git-credentials
chmod 600 ~/.git-credentials
cd ~/sops
git clone http://172.16.0.182:30300/lab/secrets.git repo
cd repo
printf '# lab/secrets\n\nÜbungsrepo für SOPS zum Anfassen.\n' > README.md
git add README.md
git commit -m "tag00: README"
git push -u origin main

Expected (abridged):

Cloning into 'repo'...
warning: You appear to have cloned an empty repository.
[main (root-commit) cbb84ec] tag00: README
 1 file changed, 3 insertions(+)
 create mode 100644 README.md
To http://172.16.0.182:30300/lab/secrets.git
 * [new branch]      main -> main
branch 'main' set up to track 'origin/main'.

Warning: credential.helper store stores the password in plaintext in ~/.git-credentials. In the lab with a lab value that is fine; on your workstation use a credential manager or SSH keys – this book’s problem starts exactly here: secrets in plaintext files.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Count all pods in the cluster. Target time: 15 s
  2. Show Forgejo’s volume with its StorageClass. Target time: 15 s
  3. Show the Forgejo version from the running pod. Target time: 20 s

Solution

# on argo
kubectl get pods -A --no-headers | wc -l
kubectl -n forgejo get pvc -o custom-columns=NAME:.metadata.name,\
STATUS:.status.phase,SC:.spec.storageClassName
kubectl -n forgejo exec deploy/forgejo -- forgejo --version | cut -c1-60

Expected:

8
NAME           STATUS   SC
forgejo-data   Bound    local-path
forgejo version 17.0.2+gitea-1.22.0 (release name 17.0.2) bu

Save your progress? Sign up for free

Checkpoint

# on argo
kubectl get nodes
kubectl -n forgejo get deploy
curl -s -o /dev/null -w '%{http_code}\n' http://172.16.0.182:30300/api/healthz

Expected:

NAME   STATUS   ROLES           AGE    VERSION
argo   Ready    control-plane   3m1s   v1.37.2+k3s1
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
forgejo   1/1     1            1           63s
200
# on argo
cd ~/sops/repo && git status -sb
which sops age age-keygen age-inspect helm yq kubectl

Expected:

## main...origin/main
/usr/local/bin/sops
/usr/local/bin/age
/usr/local/bin/age-keygen
/usr/local/bin/age-inspect
/usr/local/bin/helm
/usr/local/bin/yq
/usr/local/bin/kubectl

Challenge

Prove without a browser that the repo really is private: a Git access and an API call without logging in must fail, with login the API reports private: true.

Hint: Otherwise the credential helper logs you in silently – switch it off for one command with git -c.

Solution

# on argo
GIT_TERMINAL_PROMPT=0 git -c credential.helper= ls-remote \
  http://172.16.0.182:30300/lab/secrets.git 2>&1 | head -1
curl -s -o /dev/null -w '%{http_code}\n' \
  http://172.16.0.182:30300/api/v1/repos/lab/secrets
curl -s -u lab:lab-forgejo-123 \
  http://172.16.0.182:30300/api/v1/repos/lab/secrets | jq .private

Expected (line wrapped):

fatal: could not read Username for 'http://172.16.0.182:30300': terminal prompts
  disabled
404
true

Forgejo answers anonymous requests with 404, not 403 – it does not even reveal that the repo exists.

Save your progress? Sign up for free

Interview Note

  • Why is a private repo not enough for secrets? Read access is widely spread (team, CI, forks, backups, mirrors), and Git forgets nothing: plaintext committed once stays in the history, even after it is deleted.
  • Why install tools with a checksum? An encryption tool sees every plaintext – a tampered binary would be the perfect attack.

Enterprise Note

  • Git server with SSO, enforced two-factor login and an audit log; access for CI and GitOps as deploy keys or tokens with read-only rights per repo.
  • Pin tool versions centrally (container image for CI, package mirror) so everyone works with the same sops version.
  • A cluster’s admin kubeconfig is a secret itself: never in the repo, never in tickets.