Day 5: Environments and Repo Layout
Goal: Split notes-api into a shared base and overlays per environment without disturbing running operations, add an environment
devand promote a version from dev to prod. · Duration: approx. 75 min · Prerequisite: Day 4
Briefing
- An environment here is a separate namespace with
its own database:
notesis prod,notes-devis dev. In practice they are often separate clusters. The principle stays the same. - Base and overlays are Kustomize tools: the base contains what is the same everywhere. An overlay takes the base and changes only what differs, such as namespace and port.
- Flux recommends a fixed layout for the fleet repo:
flux-fleet/
├── apps/ what teams run
│ ├── notes/
│ │ ├── base/ the same for all environments
│ │ └── overlays/
│ │ ├── dev/ only the differences
│ │ └── prod/
│ └── notes-db/ same pattern
├── infrastructure/ what the platform runs for everyone (registry, later kro)
└── clusters/
└── lab/ per cluster: which folders run here, with which values
clusters/<name>contains only Flux objects (sources, Kustomizations, variables). The actual manifests live inapps/andinfrastructure/. A new cluster is then a new folder underclusters/that points to the same overlays.
Drill 1: Restructuring without a restart
The files move into base/. For prod there is an overlay
that simply takes over the base:
cd ~/flux-lab/fleet
for a in notes notes-db; do
mkdir -p apps/$a/base apps/$a/overlays/prod
git mv apps/$a/*.yaml apps/$a/base/
printf 'apiVersion: kustomize.config.k8s.io/v1beta1\nkind: Kustomization\nresources:\n - ../../base\n' \
> apps/$a/overlays/prod/kustomization.yaml
done
sed -i 's#path: ./apps/notes$#path: ./apps/notes/overlays/prod#' clusters/lab/notes.yaml
sed -i 's#path: ./apps/notes-db$#path: ./apps/notes-db/overlays/prod#' clusters/lab/notes-db.yaml
grep path: clusters/lab/notes*.yamlExpected:
clusters/lab/notes-db.yaml: path: ./apps/notes-db/overlays/prod
clusters/lab/notes.yaml: path: ./apps/notes/overlays/prod
After Day 4 you are careful. Before you push, you check that nothing changes in the cluster:
flux diff kustomization notes --path ./apps/notes/overlays/prod \
--kustomization-file clusters/lab/notes.yaml; echo "Exit-Code $?"
flux diff kustomization notes-db --path ./apps/notes-db/overlays/prod \
--kustomization-file clusters/lab/notes-db.yaml; echo "Exit-Code $?"Expected:
Exit-Code 0
Exit-Code 0
No difference: the same Kustomizations apply the same objects, just
from different folders. No object changes owner, so prune
removes nothing.
git add -A
git commit -m "notes: base und overlays"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes-db
flux reconcile kustomization notes
kubectl -n notes get podsExpected (shortened): the pods of notes and
notes-db are older than the commit.
Save your progress? Sign up for free
Drill 2: The dev environment
The dev overlay sets a different namespace. With that, Kustomize also
changes the name of the namespace object from the base:
notes becomes notes-dev:
mkdir -p apps/notes-db/overlays/dev apps/notes/overlays/dev
cat > apps/notes-db/overlays/dev/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: notes-dev
resources:
- ../../base
EOFnotes-api in dev also needs a different NodePort. A patch changes it:
cat > apps/notes/overlays/dev/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: notes-dev
resources:
- ../../base
patches:
- target:
kind: Service
name: notes
patch: |
- op: replace
path: /spec/ports/0/nodePort
value: 30081
EOF
kubectl kustomize apps/notes/overlays/dev | grep -E 'namespace:|nodePort'Expected:
namespace: notes-dev
namespace: notes-dev
- nodePort: 30081
namespace: notes-dev
kubectl kustomize builds an overlay locally, without
Flux and without variables. The two Kustomizations for dev go into one
file. dev gets a newer version of notes-api than prod:
flux create kustomization notes-db-dev --source=GitRepository/flux-system \
--path=./apps/notes-db/overlays/dev --prune=true --interval=10m --wait=true \
--export > clusters/lab/notes-dev.yaml
flux create kustomization notes-dev --source=GitRepository/flux-system \
--path=./apps/notes/overlays/dev --prune=true --interval=5m --wait=true \
--timeout=2m --depends-on=notes-db-dev --export >> clusters/lab/notes-dev.yaml
cat >> clusters/lab/notes-dev.yaml <<'EOF'
postBuild:
substitute:
NOTES_VERSION: 0.7.1
substituteFrom:
- kind: ConfigMap
name: cluster-vars
EOFNOTES_VERSION now appears twice: 0.5.3 in
the ConfigMap, 0.7.1 under substitute. The
values under substitute take precedence.
git add -A
git commit -m "notes: Umgebung dev"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes-db-dev
flux reconcile kustomization notes-dev
flux get kustomizationsExpected (shortened):
NAME REVISION SUSPENDED READY MESSAGE
flux-system main@sha1:048e0670 False True Applied revision: main@sha1:048e0670
notes main@sha1:048e0670 False True Applied revision: main@sha1:048e0670
notes-db main@sha1:048e0670 False True Applied revision: main@sha1:048e0670
notes-db-dev main@sha1:048e0670 False True Applied revision: main@sha1:048e0670
notes-dev main@sha1:048e0670 False True Applied revision: main@sha1:048e0670
podinfo 6.17.0@sha256:3a5e82b1 False True Applied revision: 6.17.0@sha256:3a5e82b1
registry main@sha1:048e0670 False True Applied revision: main@sha1:048e0670
curl -s localhost:30081/version; echo
curl -s localhost:30080/version; echo
kubectl -n notes-dev get deployExpected:
0.7.1
0.5.3
NAME READY UP-TO-DATE AVAILABLE AGE
notes 1/1 1 1 10s
notes-db 1/1 1 1 34s
dev runs with one replica: NOTES_REPLICAS is not set
there, so the default value from ${NOTES_REPLICAS:=1}
applies.
Save your progress? Sign up for free
Drill 3: Promoting
0.7.1 has proven itself in dev. Prod gets the version through the cluster’s ConfigMap. The override in dev is now superfluous, so it goes:
sed -i 's/NOTES_VERSION: 0.5.3/NOTES_VERSION: 0.7.1/' clusters/lab/cluster-vars.yaml
sed -i '/substitute:$/,/NOTES_VERSION: 0.7.1/d' clusters/lab/notes-dev.yaml
tail -5 clusters/lab/notes-dev.yamlExpected:
postBuild:
substituteFrom:
- kind: ConfigMap
name: cluster-vars
git commit -am "notes 0.7.1 nach prod"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes
curl -s localhost:30080/version; echo
curl -s localhost:30080/notesExpected (shortened):
✔ applied revision main@sha1:9d1fa13881ed44265c91e36f1f6fa8b399e97d1d
0.7.1
[{"id":"ec17fa30-3215-4493-841d-8e7dbb16dae7","text":"Hallo Flux"}]
- The upgrade took the data along. The init container
migrateadjusts the schema before the new version starts. - As with every commit, all Kustomizations that read from
flux-systemreconcile. The secondreconcileonly makes sure thatnotesreads the already changed ConfigMap and not, by chance, the old one.
Save your progress? Sign up for free
Checkpoint
flux get kustomizationsExpected: seven Kustomizations, all READY
True.
curl -s localhost:30081/version; echoExpected: 0.7.1.
diff -r ~/flux-lab/fleet/apps ~/flux-lab/code/loesung/tag05/appsExpected: no output.
Challenge
The QA team wants a third environment test: its own
namespace notes-test, its own database, notes-api on port
30082. Build it, check it and then tear it down again. Is anything left
over after the teardown?
Hint: The dev overlays are a good template. And remember the annotation on the namespace from Day 4.
Solution
for a in notes notes-db; do
cp -r apps/$a/overlays/dev apps/$a/overlays/test
sed -i 's/notes-dev/notes-test/; s/30081/30082/' apps/$a/overlays/test/kustomization.yaml
done
sed 's/-dev/-test/g; s#overlays/dev#overlays/test#' clusters/lab/notes-dev.yaml > clusters/lab/notes-test.yaml
git add -A
git commit -m "notes: Umgebung test"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes-db-test
flux reconcile kustomization notes-test
curl -s localhost:30082/version; echoExpected (shortened):
...
✔ applied revision main@sha1:1b213d7067ca133c438ccc74d84a11caffb28743
0.7.1
Tearing down means: deleting the files from Git.
git rm -q -r apps/notes/overlays/test apps/notes-db/overlays/test \
clusters/lab/notes-test.yaml
git commit -m "notes: Umgebung test weg"
git push
flux reconcile kustomization flux-system --with-source
kubectl get ns notes-test
kubectl -n notes-test get all,pvc,secretExpected:
NAME STATUS AGE
notes-test Active 68s
No resources found in notes-test namespace.
flux-systemdeleted the two Kustomizations. Because they haveprune: true, they remove everything they created when they are deleted: Deployments, Services, Secrets, even the volume.- Only the namespace remains; the annotation
prune: disabledprotects it. An empty namespace does no harm, but you deliberately remove it by hand:
kubectl delete ns notes-testSave your progress? Sign up for free
Enterprise Note
- Monorepo or one repo per environment? One repo for
everything is the simplest and shows every promotion as a diff. Many
companies still split it up: teams maintain
apps/in their own repos, the platform team ownsclusters/andinfrastructure/. Flux connects this with oneGitRepositoryper team repo (Day 12). - Promoting via pull request: a promotion is a commit like any other. In operations it goes through a review as a pull request, and prod may read a protected branch or a released OCI artifact (Day 3).
- Variables from outside: if a ConfigMap for
substituteFromdoes not come from the same repo but, for example, from a cloud tool, it gets the labelreconcile.fluxcd.io/watch: Enabled. Then the Kustomizations reconcile immediately when it changes. - Clusters instead of namespaces: dev and prod in the
same cluster share nodes, network and admins. For real separation, each
environment gets its own cluster with its own folder
clusters/devandclusters/prod.