Day 0: The Training Ground
Goal: A VM with single-node k3s, the tools sops, age, helm and yq, and a private Git repo on your own Forgejo – the place where encrypted secrets land from tomorrow on. · Duration: approx. 60 min · Prerequisite: Ubuntu 24.04 VM with 4 vCPU and 8 GB RAM
Briefing
- The whole book runs on one VM
argo(172.16.0.182), userubuntu. It is workstation and cluster at the same time. Working folder:~/sops/tagNN/. - The Git repo
lab/secretslives on your own Forgejo in the cluster, not on GitHub. That way your lab sees exactly what an attacker with read access to the repo would see – and Flux and Argo CD (Days 10 and 11) clone it from inside the cluster later. - Every tool comes as a binary for your architecture, with a checksum,
never via
curl | shfrom a third-party source (k3s is the one documented exception).
VM argo 172.16.0.182 (4 vCPU, 7.7 GiB)
┌─────────────────────────────────────────────────────────────────┐
│ Workstation: sops, age, helm, yq, git ~/sops/repo │
│ k3s v1.37.2 (:6443) local-path (default StorageClass) │
│ forgejo NodePort :30300 repo lab/secrets (private) │
└─────────────────────────────────────────────────────────────────┘
Drill 1: Check the VM
Before you install anything, you look at what is already there.
# on argo
free -m
nproc
df -h / | tail -1
ss -tlnp | grep -E ':(80|443|6443|30300|30820) ' ; echo "Treffer: $?"Expected:
total used free shared buff/cache available
Mem: 7941 477 7025 2 708 7463
Swap: 0 0 0
4
/dev/sda1 77G 4.5G 72G 6% /
Treffer: 1
Treffer: 1 (“hits: 1”) means: grep found
nothing, the ports for k3s (6443), Traefik (80, 443), Forgejo (30300)
and OpenBao (30820, Day 7) are free.
Tip: The book’s lab VM holds leftovers from other books (a stopped PostgreSQL,
~/debug). The book does not touch them. You do not need them on your VM.
Save your progress? Sign up for free
Drill 2: Install a pinned k3s
# on argo
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.37.2+k3s1 sh -s - \
--write-kubeconfig-mode 600 --disable metrics-server
sleep 60
sudo k3s kubectl get nodes
sudo k3s kubectl get pods -AExpected (abridged):
[INFO] Using v1.37.2+k3s1 as release
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] systemd: Starting k3s
NAME STATUS ROLES AGE VERSION
argo Ready control-plane 61s v1.37.2+k3s1
NAMESPACE NAME READY STATUS RESTARTS
kube-system coredns-577d995dff-5q7ss 1/1 Running 0
kube-system helm-install-gateway-api-crd-7m484 0/1 Completed 0
kube-system helm-install-traefik-crd-4p7q8 0/1 Completed 0
kube-system helm-install-traefik-qxln5 0/1 Completed 2
kube-system local-path-provisioner-56c6584955-f8nz9 1/1 Running 0
kube-system svclb-traefik-e9cd5754-h8xwp 2/2 Running 0
kube-system traefik-68df944ff7-ws7ds 0/1 Running 0
--write-kubeconfig-mode 600 makes the admin kubeconfig
readable for root only – it grants full access to every Secret in the
cluster. The book does not need metrics-server.
Save your progress? Sign up for free
Drill 3: kubeconfig and the context sops-lab
# on argo
mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown ubuntu: ~/.kube/config
chmod 600 ~/.kube/config
echo 'export KUBECONFIG=$HOME/.kube/config' >> ~/.bashrc
export KUBECONFIG=$HOME/.kube/config
kubectl config rename-context default sops-lab
kubectl config get-contexts
kubectl get nodesExpected:
Context "default" renamed to "sops-lab".
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* sops-lab default default
NAME STATUS ROLES AGE VERSION
argo Ready control-plane 78s v1.37.2+k3s1
Without KUBECONFIG, the k3s kubectl always
reads /etc/rancher/k3s/k3s.yaml – and as
ubuntu it is not allowed to. With the variable, kubectl and
Helm read the same file.
Save your progress? Sign up for free
Drill 4: sops, age, helm and yq
sops and Helm publish checksums, yq a file with several hash types.
age provides no checksum file but transparency proofs
(.proof, Sigsum) – here the download over HTTPS from the
official release page is enough.
# on argo
mkdir -p ~/sops/tag00 && cd ~/sops/tag00
ARCH=$(dpkg --print-architecture)
S=https://github.com/getsops/sops/releases/download/v3.13.3
curl -fsSLO $S/sops-v3.13.3.linux.$ARCH
curl -fsSLO $S/sops-v3.13.3.checksums.txt
grep " sops-v3.13.3.linux.$ARCH$" sops-v3.13.3.checksums.txt | sha256sum -c
sudo install -m 0755 sops-v3.13.3.linux.$ARCH /usr/local/bin/sops
A=https://github.com/FiloSottile/age/releases/download/v1.3.2
curl -fsSLO $A/age-v1.3.2-linux-$ARCH.tar.gz
tar -xzf age-v1.3.2-linux-$ARCH.tar.gz
ls age
sudo install -m 0755 age/age age/age-keygen age/age-inspect /usr/local/bin/Expected:
sops-v3.13.3.linux.amd64: OK
LICENSE
age
age-inspect
age-keygen
age-plugin-batchpass
age-plugin-pq
age-plugin-tag
age-plugin-tagpq
Besides age and age-keygen, age 1.3 ships a
tool age-inspect and several plugins. Ubuntu 24.04 only has
age 1.1.1 via apt – without age-inspect and
without post-quantum keys (Day 1).
# on argo
cd ~/sops/tag00
ARCH=$(dpkg --print-architecture)
H=v4.3.2
curl -fsSLO https://get.helm.sh/helm-$H-linux-$ARCH.tar.gz
curl -fsSLO https://get.helm.sh/helm-$H-linux-$ARCH.tar.gz.sha256sum
sha256sum -c helm-$H-linux-$ARCH.tar.gz.sha256sum
tar -xzf helm-$H-linux-$ARCH.tar.gz
sudo install -m 0755 linux-$ARCH/helm /usr/local/bin/helm
Y=https://github.com/mikefarah/yq/releases/download/v4.54.3
curl -fsSLo yq $Y/yq_linux_$ARCH
curl -fsSLo yq-checksums $Y/checksums
grep "^yq_linux_$ARCH " yq-checksums | grep -c "$(sha256sum yq | cut -d' ' -f1)"
sudo install -m 0755 yq /usr/local/bin/yq
sudo apt-get install -y git jq >/dev/null
sops --version --disable-version-check
age --version
helm version --short
yq --versionExpected:
helm-v4.3.2-linux-amd64.tar.gz: OK
1
sops 3.13.3
v1.3.2
v4.3.2+g8346aaa
yq (https://github.com/mikefarah/yq/) version v4.54.3
The 1 means: the SHA-256 sum of yq is in
the line for yq_linux_amd64.
Save your progress? Sign up for free
Drill 5: Forgejo in the cluster
Forgejo is a lean Git server. The rootless image exists for amd64 and arm64; SQLite as the database, data on a local-path volume, the web UI and Git over HTTP on NodePort 30300.
# on argo
cd ~/sops/tag00
cat > forgejo.yaml <<'EOF'
apiVersion: v1
kind: Namespace
metadata: {name: forgejo}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: forgejo-data, namespace: forgejo}
spec:
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 2Gi}}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: forgejo, namespace: forgejo}
spec:
replicas: 1
strategy: {type: Recreate}
selector: {matchLabels: {app: forgejo}}
template:
metadata: {labels: {app: forgejo}}
spec:
securityContext: {fsGroup: 1000}
containers:
- name: forgejo
image: codeberg.org/forgejo/forgejo:17.0.2-rootless
env:
- {name: FORGEJO__security__INSTALL_LOCK, value: "true"}
- {name: FORGEJO__server__ROOT_URL, value: "http://172.16.0.182:30300/"}
- {name: FORGEJO__database__DB_TYPE, value: sqlite3}
- {name: FORGEJO__service__DISABLE_REGISTRATION, value: "true"}
ports: [{containerPort: 3000, name: http}]
readinessProbe:
httpGet: {path: /api/healthz, port: http}
volumeMounts:
- {name: data, mountPath: /var/lib/gitea}
- {name: data, mountPath: /etc/gitea, subPath: conf}
volumes:
- name: data
persistentVolumeClaim: {claimName: forgejo-data}
---
apiVersion: v1
kind: Service
metadata: {name: forgejo-http, namespace: forgejo}
spec:
type: NodePort
selector: {app: forgejo}
ports: [{port: 3000, targetPort: http, nodePort: 30300, name: http}]
EOF
kubectl apply -f forgejo.yaml
kubectl -n forgejo rollout status deploy/forgejo --timeout=180s
kubectl -n forgejo get pods,svcExpected:
namespace/forgejo created
persistentvolumeclaim/forgejo-data created
deployment.apps/forgejo created
service/forgejo-http created
Waiting for deployment "forgejo" rollout to finish: 0 of 1 updated replicas are ...
deployment "forgejo" successfully rolled out
NAME READY STATUS RESTARTS AGE
pod/forgejo-7478d756d7-l4f7t 1/1 Running 0 32s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/forgejo-http NodePort 10.43.52.65 <none> 3000:30300/TCP 32s
INSTALL_LOCK=true skips the setup wizard in the browser,
the FORGEJO__section__KEY variables end up in
app.ini. Self-registration is off: only the admin creates
users.
Save your progress? Sign up for free
Drill 6: User and private repo
# on argo
kubectl -n forgejo exec deploy/forgejo -- forgejo admin user create \
--username lab --password lab-forgejo-123 --email [email protected] --admin \
--must-change-password=false
curl -s -u lab:lab-forgejo-123 -H 'Content-Type: application/json' \
-d '{"name":"secrets","private":true,"default_branch":"main"}' \
http://172.16.0.182:30300/api/v1/user/repos | jq -r '.full_name, .clone_url'Expected:
New user 'lab' has been successfully created!
lab/secrets
http://172.16.0.182:30300/lab/secrets.git
The password lab-forgejo-123 is a lab value. Private
means: without logging in, the repo is invisible. That still does not
protect secrets – everyone with read access, every fork, every backup
and every CI runner has the full history.
Save your progress? Sign up for free
Drill 7: Clone and the first push
# on argo
git config --global user.name "Lab Admin"
git config --global user.email [email protected]
git config --global init.defaultBranch main
git config --global credential.helper store
printf 'http://lab:[email protected]:30300\n' > ~/.git-credentials
chmod 600 ~/.git-credentials
cd ~/sops
git clone http://172.16.0.182:30300/lab/secrets.git repo
cd repo
printf '# lab/secrets\n\nÜbungsrepo für SOPS zum Anfassen.\n' > README.md
git add README.md
git commit -m "tag00: README"
git push -u origin mainExpected (abridged):
Cloning into 'repo'...
warning: You appear to have cloned an empty repository.
[main (root-commit) cbb84ec] tag00: README
1 file changed, 3 insertions(+)
create mode 100644 README.md
To http://172.16.0.182:30300/lab/secrets.git
* [new branch] main -> main
branch 'main' set up to track 'origin/main'.
Warning:
credential.helper storestores the password in plaintext in~/.git-credentials. In the lab with a lab value that is fine; on your workstation use a credential manager or SSH keys – this book’s problem starts exactly here: secrets in plaintext files.
Save your progress? Sign up for free
Exercise: Speed Drills
- Count all pods in the cluster. Target time: 15 s
- Show Forgejo’s volume with its StorageClass. Target time: 15 s
- Show the Forgejo version from the running pod. Target time: 20 s
Solution
# on argo
kubectl get pods -A --no-headers | wc -l
kubectl -n forgejo get pvc -o custom-columns=NAME:.metadata.name,\
STATUS:.status.phase,SC:.spec.storageClassName
kubectl -n forgejo exec deploy/forgejo -- forgejo --version | cut -c1-60Expected:
8
NAME STATUS SC
forgejo-data Bound local-path
forgejo version 17.0.2+gitea-1.22.0 (release name 17.0.2) bu
Save your progress? Sign up for free
Checkpoint
# on argo
kubectl get nodes
kubectl -n forgejo get deploy
curl -s -o /dev/null -w '%{http_code}\n' http://172.16.0.182:30300/api/healthzExpected:
NAME STATUS ROLES AGE VERSION
argo Ready control-plane 3m1s v1.37.2+k3s1
NAME READY UP-TO-DATE AVAILABLE AGE
forgejo 1/1 1 1 63s
200
# on argo
cd ~/sops/repo && git status -sb
which sops age age-keygen age-inspect helm yq kubectlExpected:
## main...origin/main
/usr/local/bin/sops
/usr/local/bin/age
/usr/local/bin/age-keygen
/usr/local/bin/age-inspect
/usr/local/bin/helm
/usr/local/bin/yq
/usr/local/bin/kubectl
Challenge
Prove without a browser that the repo really is private: a Git access
and an API call without logging in must fail, with login the API reports
private: true.
Hint: Otherwise the credential helper logs you in
silently – switch it off for one command with git -c.
Solution
# on argo
GIT_TERMINAL_PROMPT=0 git -c credential.helper= ls-remote \
http://172.16.0.182:30300/lab/secrets.git 2>&1 | head -1
curl -s -o /dev/null -w '%{http_code}\n' \
http://172.16.0.182:30300/api/v1/repos/lab/secrets
curl -s -u lab:lab-forgejo-123 \
http://172.16.0.182:30300/api/v1/repos/lab/secrets | jq .privateExpected (line wrapped):
fatal: could not read Username for 'http://172.16.0.182:30300': terminal prompts
disabled
404
true
Forgejo answers anonymous requests with 404, not 403 – it does not even reveal that the repo exists.
Save your progress? Sign up for free
Interview Note
- Why is a private repo not enough for secrets? Read access is widely spread (team, CI, forks, backups, mirrors), and Git forgets nothing: plaintext committed once stays in the history, even after it is deleted.
- Why install tools with a checksum? An encryption tool sees every plaintext – a tampered binary would be the perfect attack.
Enterprise Note
- Git server with SSO, enforced two-factor login and an audit log; access for CI and GitOps as deploy keys or tokens with read-only rights per repo.
- Pin tool versions centrally (container image for CI, package mirror) so everyone works with the same sops version.
- A cluster’s admin kubeconfig is a secret itself: never in the repo, never in tickets.