Skip to content
Hands-on IT IT Drills

Day 5: Environments and Repo Layout

Goal: Split notes-api into a shared base and overlays per environment without disturbing running operations, add an environment dev and promote a version from dev to prod. · Duration: approx. 75 min · Prerequisite: Day 4

Briefing

  • An environment here is a separate namespace with its own database: notes is prod, notes-dev is dev. In practice they are often separate clusters. The principle stays the same.
  • Base and overlays are Kustomize tools: the base contains what is the same everywhere. An overlay takes the base and changes only what differs, such as namespace and port.
  • Flux recommends a fixed layout for the fleet repo:
 flux-fleet/
 ├── apps/                    what teams run
 │   ├── notes/
 │   │   ├── base/            the same for all environments
 │   │   └── overlays/
 │   │       ├── dev/         only the differences
 │   │       └── prod/
 │   └── notes-db/            same pattern
 ├── infrastructure/          what the platform runs for everyone (registry, later kro)
 └── clusters/
     └── lab/                 per cluster: which folders run here, with which values
  • clusters/<name> contains only Flux objects (sources, Kustomizations, variables). The actual manifests live in apps/ and infrastructure/. A new cluster is then a new folder under clusters/ that points to the same overlays.

Drill 1: Restructuring without a restart

The files move into base/. For prod there is an overlay that simply takes over the base:

cd ~/flux-lab/fleet
for a in notes notes-db; do
  mkdir -p apps/$a/base apps/$a/overlays/prod
  git mv apps/$a/*.yaml apps/$a/base/
  printf 'apiVersion: kustomize.config.k8s.io/v1beta1\nkind: Kustomization\nresources:\n  - ../../base\n' \
    > apps/$a/overlays/prod/kustomization.yaml
done
sed -i 's#path: ./apps/notes$#path: ./apps/notes/overlays/prod#' clusters/lab/notes.yaml
sed -i 's#path: ./apps/notes-db$#path: ./apps/notes-db/overlays/prod#' clusters/lab/notes-db.yaml
grep path: clusters/lab/notes*.yaml

Expected:

clusters/lab/notes-db.yaml:  path: ./apps/notes-db/overlays/prod
clusters/lab/notes.yaml:  path: ./apps/notes/overlays/prod

After Day 4 you are careful. Before you push, you check that nothing changes in the cluster:

flux diff kustomization notes --path ./apps/notes/overlays/prod \
  --kustomization-file clusters/lab/notes.yaml; echo "Exit-Code $?"
flux diff kustomization notes-db --path ./apps/notes-db/overlays/prod \
  --kustomization-file clusters/lab/notes-db.yaml; echo "Exit-Code $?"

Expected:

Exit-Code 0
Exit-Code 0

No difference: the same Kustomizations apply the same objects, just from different folders. No object changes owner, so prune removes nothing.

git add -A
git commit -m "notes: base und overlays"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes-db
flux reconcile kustomization notes
kubectl -n notes get pods

Expected (shortened): the pods of notes and notes-db are older than the commit.

Save your progress? Sign up for free

Drill 2: The dev environment

The dev overlay sets a different namespace. With that, Kustomize also changes the name of the namespace object from the base: notes becomes notes-dev:

mkdir -p apps/notes-db/overlays/dev apps/notes/overlays/dev
cat > apps/notes-db/overlays/dev/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: notes-dev
resources:
  - ../../base
EOF

notes-api in dev also needs a different NodePort. A patch changes it:

cat > apps/notes/overlays/dev/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: notes-dev
resources:
  - ../../base
patches:
  - target:
      kind: Service
      name: notes
    patch: |
      - op: replace
        path: /spec/ports/0/nodePort
        value: 30081
EOF
kubectl kustomize apps/notes/overlays/dev | grep -E 'namespace:|nodePort'

Expected:

  namespace: notes-dev
  namespace: notes-dev
  - nodePort: 30081
  namespace: notes-dev

kubectl kustomize builds an overlay locally, without Flux and without variables. The two Kustomizations for dev go into one file. dev gets a newer version of notes-api than prod:

flux create kustomization notes-db-dev --source=GitRepository/flux-system \
  --path=./apps/notes-db/overlays/dev --prune=true --interval=10m --wait=true \
  --export > clusters/lab/notes-dev.yaml
flux create kustomization notes-dev --source=GitRepository/flux-system \
  --path=./apps/notes/overlays/dev --prune=true --interval=5m --wait=true \
  --timeout=2m --depends-on=notes-db-dev --export >> clusters/lab/notes-dev.yaml
cat >> clusters/lab/notes-dev.yaml <<'EOF'
  postBuild:
    substitute:
      NOTES_VERSION: 0.7.1
    substituteFrom:
      - kind: ConfigMap
        name: cluster-vars
EOF

NOTES_VERSION now appears twice: 0.5.3 in the ConfigMap, 0.7.1 under substitute. The values under substitute take precedence.

git add -A
git commit -m "notes: Umgebung dev"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes-db-dev
flux reconcile kustomization notes-dev
flux get kustomizations

Expected (shortened):

NAME            REVISION                SUSPENDED   READY   MESSAGE
flux-system     main@sha1:048e0670      False       True    Applied revision: main@sha1:048e0670
notes           main@sha1:048e0670      False       True    Applied revision: main@sha1:048e0670
notes-db        main@sha1:048e0670      False       True    Applied revision: main@sha1:048e0670
notes-db-dev    main@sha1:048e0670      False       True    Applied revision: main@sha1:048e0670
notes-dev       main@sha1:048e0670      False       True    Applied revision: main@sha1:048e0670
podinfo         6.17.0@sha256:3a5e82b1  False       True    Applied revision: 6.17.0@sha256:3a5e82b1
registry        main@sha1:048e0670      False       True    Applied revision: main@sha1:048e0670
curl -s localhost:30081/version; echo
curl -s localhost:30080/version; echo
kubectl -n notes-dev get deploy

Expected:

0.7.1
0.5.3
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
notes      1/1     1            1           10s
notes-db   1/1     1            1           34s

dev runs with one replica: NOTES_REPLICAS is not set there, so the default value from ${NOTES_REPLICAS:=1} applies.

Save your progress? Sign up for free

Drill 3: Promoting

0.7.1 has proven itself in dev. Prod gets the version through the cluster’s ConfigMap. The override in dev is now superfluous, so it goes:

sed -i 's/NOTES_VERSION: 0.5.3/NOTES_VERSION: 0.7.1/' clusters/lab/cluster-vars.yaml
sed -i '/substitute:$/,/NOTES_VERSION: 0.7.1/d' clusters/lab/notes-dev.yaml
tail -5 clusters/lab/notes-dev.yaml

Expected:

  postBuild:
    substituteFrom:
      - kind: ConfigMap
        name: cluster-vars
git commit -am "notes 0.7.1 nach prod"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes
curl -s localhost:30080/version; echo
curl -s localhost:30080/notes

Expected (shortened):

✔ applied revision main@sha1:9d1fa13881ed44265c91e36f1f6fa8b399e97d1d
0.7.1
[{"id":"ec17fa30-3215-4493-841d-8e7dbb16dae7","text":"Hallo Flux"}]
  • The upgrade took the data along. The init container migrate adjusts the schema before the new version starts.
  • As with every commit, all Kustomizations that read from flux-system reconcile. The second reconcile only makes sure that notes reads the already changed ConfigMap and not, by chance, the old one.

Save your progress? Sign up for free

Checkpoint

flux get kustomizations

Expected: seven Kustomizations, all READY True.

curl -s localhost:30081/version; echo

Expected: 0.7.1.

diff -r ~/flux-lab/fleet/apps ~/flux-lab/code/loesung/tag05/apps

Expected: no output.

Challenge

The QA team wants a third environment test: its own namespace notes-test, its own database, notes-api on port 30082. Build it, check it and then tear it down again. Is anything left over after the teardown?

Hint: The dev overlays are a good template. And remember the annotation on the namespace from Day 4.

Solution

for a in notes notes-db; do
  cp -r apps/$a/overlays/dev apps/$a/overlays/test
  sed -i 's/notes-dev/notes-test/; s/30081/30082/' apps/$a/overlays/test/kustomization.yaml
done
sed 's/-dev/-test/g; s#overlays/dev#overlays/test#' clusters/lab/notes-dev.yaml   > clusters/lab/notes-test.yaml
git add -A
git commit -m "notes: Umgebung test"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes-db-test
flux reconcile kustomization notes-test
curl -s localhost:30082/version; echo

Expected (shortened):

...
✔ applied revision main@sha1:1b213d7067ca133c438ccc74d84a11caffb28743
0.7.1

Tearing down means: deleting the files from Git.

git rm -q -r apps/notes/overlays/test apps/notes-db/overlays/test \
  clusters/lab/notes-test.yaml
git commit -m "notes: Umgebung test weg"
git push
flux reconcile kustomization flux-system --with-source
kubectl get ns notes-test
kubectl -n notes-test get all,pvc,secret

Expected:

NAME         STATUS   AGE
notes-test   Active   68s
No resources found in notes-test namespace.
  • flux-system deleted the two Kustomizations. Because they have prune: true, they remove everything they created when they are deleted: Deployments, Services, Secrets, even the volume.
  • Only the namespace remains; the annotation prune: disabled protects it. An empty namespace does no harm, but you deliberately remove it by hand:
kubectl delete ns notes-test

Save your progress? Sign up for free

Enterprise Note

  • Monorepo or one repo per environment? One repo for everything is the simplest and shows every promotion as a diff. Many companies still split it up: teams maintain apps/ in their own repos, the platform team owns clusters/ and infrastructure/. Flux connects this with one GitRepository per team repo (Day 12).
  • Promoting via pull request: a promotion is a commit like any other. In operations it goes through a review as a pull request, and prod may read a protected branch or a released OCI artifact (Day 3).
  • Variables from outside: if a ConfigMap for substituteFrom does not come from the same repo but, for example, from a cloud tool, it gets the label reconcile.fluxcd.io/watch: Enabled. Then the Kustomizations reconcile immediately when it changes.
  • Clusters instead of namespaces: dev and prod in the same cluster share nodes, network and admins. For real separation, each environment gets its own cluster with its own folder clusters/dev and clusters/prod.