Day 9: API Versions and Conversion
Goal: The StaticSite gets a stable version
v1(the fieldcontentFromis calledsourcethere), which becomes the storage version, and a conversion webhook translates losslessly betweenv1alpha1andv1– including migration of the stored objects · Duration: approx. 150 min · Prerequisite: Day 8
Briefing
- A CRD can serve several versions
(
served), but store in only one (storage). Everything else is converted on read and write. - Without a webhook (
strategy: None), the API server only swapsapiVersion– renamed fields get lost along the way. - Hub & spoke: one version is the hub
(
Hub()), every other one is a spoke withConvertTo/ConvertFrom. With n versions you need n-1 pairs, not n². status.storedVersionsremembers every version that was ever in etcd. You remove an old version only after the migration.
kubectl get ...v1alpha1... kubectl get ...v1...
| |
v1alpha1 (spoke) <-- ConvertFrom -- v1 (hub, storage) <--> etcd
--- ConvertTo -->
POST /convert (conversion webhook in the operator)
Drill 1: Create version v1
source ~/op-lab/env.sh
cp -r ~/op-lab/tag08-site-operator ~/op-lab/tag09-site-operator
cd ~/op-lab/tag09-site-operator
kubebuilder create api --group web --version v1 --kind StaticSite \
--resource --controller=falseExpected (shortened):
Next: implement your new API and generate the manifests (e.g. CRDs,CRs) with:
$ make manifests
Kubebuilder created api/v1/ and added
webv1.AddToScheme to cmd/main.go.
--controller=false: there is still one
controller – from Drill 5 on, it works with v1.
Save your progress? Sign up for free
Drill 2: The v1 types
v1 is v1alpha1 with one renamed field and
the storageversion marker. Instead of typing 100 lines, you
derive the file and let the tooling show you the difference – exactly
this diff is your API change:
sed -e 's/^package v1alpha1/package v1/' \
-e 's/ExactlyOneOf=content;contentFrom/ExactlyOneOf=content;source/' \
-e 's/ContentFrom \*ContentSource/Source *ContentSource/' \
-e 's/json:"contentFrom,omitempty"/json:"source,omitempty"/' \
-e 's/zu contentFrom)/zu source)/; s|// contentFrom verweist|// source verweist|' \
-e 's|^// +kubebuilder:subresource:status|&\n// +kubebuilder:storageversion|' \
api/v1alpha1/staticsite_types.go > api/v1/staticsite_types.go
sed 's/^package v1alpha1/package v1/' api/v1alpha1/conditions.go > api/v1/conditions.go
gofmt -l api/
diff api/v1alpha1/staticsite_types.go api/v1/staticsite_types.goExpected:
1c1
< package v1alpha1
---
> package v1
24c24
< // +kubebuilder:validation:ExactlyOneOf=content;contentFrom
---
> // +kubebuilder:validation:ExactlyOneOf=content;source
26c26
< // content ist der HTML-Inhalt der Seite (alternativ zu contentFrom).
---
> // content ist der HTML-Inhalt der Seite (alternativ zu source).
31c31
< // contentFrom verweist auf eine vorhandene ConfigMap (alternativ zu content).
---
> // source verweist auf eine vorhandene ConfigMap (alternativ zu content).
33c33
< ContentFrom *ContentSource `json:"contentFrom,omitempty"`
---
> Source *ContentSource `json:"source,omitempty"`
77a78
> // +kubebuilder:storageversion
The old version gets a deprecation notice. In
api/v1alpha1/staticsite_types.go, directly below
// +kubebuilder:subresource:status:
// +kubebuilder:deprecatedversion:warning="v1alpha1 ist veraltet, nutze v1"make manifests generate
grep -E '^ (name|served|storage|deprecat)' \
config/crd/bases/web.lab.example_staticsites.yamlExpected (shortened):
name: v1
served: true
storage: true
deprecated: true
deprecationWarning: v1alpha1 ist veraltet, nutze v1
name: v1alpha1
served: true
storage: false
Save your progress? Sign up for free
Drill 3: Without conversion, something gets lost
Simulate a cluster that already holds StaticSites from yesterday. Terminal 1 with the CRD from Day 8:
source ~/op-lab/env.sh
envtest-up --crds ~/op-lab/tag08-site-operator/config/crd/basesTerminal 2: create two old objects, then apply the new CRD – still without a webhook:
kubectl apply -f lab/demo.yaml -f lab/extern.yaml
kubectl replace -f config/crd/bases/
kubectl get crd staticsites.web.lab.example \
-o jsonpath='{.status.storedVersions}{"\n"}'
kubectl get staticsites.v1.web.lab.example extern -o jsonpath='{.spec}{"\n"}'Expected:
staticsite.web.lab.example/demo created
staticsite.web.lab.example/extern created
customresourcedefinition.apiextensions.k8s.io/staticsites.web.lab.example replaced
["v1alpha1","v1"]
{"image":"nginxinc/nginx-unprivileged:1.31-alpine","path":"/","replicas":1}
Read as v1, extern shows
no source anymore: contentFrom is unknown
in the v1 schema and gets cut away (pruning). It is still in etcd – but
the first write through v1 would delete it for good. That
is exactly what the conversion webhook is for.
Save your progress? Sign up for free
Drill 4: Hub and spoke
kubebuilder create webhook --group web --version v1 --kind StaticSite \
--conversion --spoke v1alpha1
cat api/v1/staticsite_conversion.go | grep -v '^//' | grep .Expected (shortened):
package v1
func (*StaticSite) Hub() {}
The hub is done. Kubebuilder also created
internal/webhook/v1/ (webhook server for
/convert), the call in cmd/main.go and the
patch config/crd/patches/webhook_in_staticsites.yaml
(strategy Webhook). The actual work lives in the spoke:
cat > api/v1alpha1/staticsite_conversion.go <<'EOF'
package v1alpha1
import (
"sigs.k8s.io/controller-runtime/pkg/conversion"
webv1 "example.com/site-operator/api/v1"
)
// ConvertTo converts this spoke version (v1alpha1) to the hub (v1).
func (src *StaticSite) ConvertTo(dstRaw conversion.Hub) error {
dst := dstRaw.(*webv1.StaticSite)
dst.ObjectMeta = src.ObjectMeta
dst.Spec = webv1.StaticSiteSpec{
Content: src.Spec.Content,
Replicas: src.Spec.Replicas,
Image: src.Spec.Image,
Path: src.Spec.Path,
}
if from := src.Spec.ContentFrom; from != nil { // contentFrom -> source
dst.Spec.Source = &webv1.ContentSource{}
if from.ConfigMapRef != nil {
dst.Spec.Source.ConfigMapRef = &webv1.ConfigMapReference{
Name: from.ConfigMapRef.Name}
}
}
dst.Status = webv1.StaticSiteStatus(src.Status) // identische Felder
return nil
}
// ConvertFrom converts the hub (v1) back to v1alpha1.
func (dst *StaticSite) ConvertFrom(srcRaw conversion.Hub) error {
src := srcRaw.(*webv1.StaticSite)
dst.ObjectMeta = src.ObjectMeta
dst.Spec = StaticSiteSpec{
Content: src.Spec.Content,
Replicas: src.Spec.Replicas,
Image: src.Spec.Image,
Path: src.Spec.Path,
}
if s := src.Spec.Source; s != nil { // source -> contentFrom
dst.Spec.ContentFrom = &ContentSource{}
if s.ConfigMapRef != nil {
dst.Spec.ContentFrom.ConfigMapRef = &ConfigMapReference{
Name: s.ConfigMapRef.Name}
}
}
dst.Status = StaticSiteStatus(src.Status)
return nil
}
EOFwebv1.StaticSiteStatus(src.Status) is a Go type
conversion: it works as long as both structs have exactly the same
fields – if the status changes in one version, the compiler reports it
immediately.
The most important test for any conversion is the round trip spoke → hub → spoke:
cat > api/v1alpha1/staticsite_conversion_test.go <<'EOF'
package v1alpha1
import (
"testing"
"k8s.io/apimachinery/pkg/api/equality"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
webv1 "example.com/site-operator/api/v1"
)
// Spoke -> hub -> spoke must be lossless, otherwise every
// read through the other version loses data.
func TestRoundTrip(t *testing.T) {
ref := &ConfigMapReference{Name: "html"}
tests := map[string]StaticSiteSpec{
"content": {Content: "<h1>Hallo</h1>", Replicas: 2, Path: "/docs/"},
"contentFrom": {ContentFrom: &ContentSource{ConfigMapRef: ref}},
"leere Quelle": {ContentFrom: &ContentSource{}},
}
for name, spec := range tests {
t.Run(name, func(t *testing.T) {
in := &StaticSite{ObjectMeta: metav1.ObjectMeta{Name: "rt"}, Spec: spec}
in.Status.URL = "http://rt.default.svc"
hub := &webv1.StaticSite{}
if err := in.ConvertTo(hub); err != nil {
t.Fatal(err)
}
out := &StaticSite{}
if err := out.ConvertFrom(hub); err != nil {
t.Fatal(err)
}
if !equality.Semantic.DeepEqual(in, out) {
t.Errorf("Round-Trip verändert:\n in %+v\nout %+v", in.Spec, out.Spec)
}
})
}
}
EOF
go test ./api/... -run RoundTrip -v 2>&1 | grep -E '^(---| ---|ok)'Expected:
--- PASS: TestRoundTrip (0.00s)
--- PASS: TestRoundTrip/content (0.00s)
--- PASS: TestRoundTrip/contentFrom (0.00s)
--- PASS: TestRoundTrip/leere_Quelle (0.00s)
ok example.com/site-operator/api/v1alpha1 0.153s
Why the case “leere Quelle” (empty source)? The obvious shorthand
if from != nil && from.ConfigMapRef != nil turns
contentFrom: {} into nil on the way back:
--- FAIL: TestRoundTrip (0.00s)
--- FAIL: TestRoundTrip/leere_Quelle (0.00s)
staticsite_conversion_test.go:34: Round-Trip verändert:
in {Content: ContentFrom:0x5482827ac3b8 Replicas:0 Image: Path:}
out {Content: ContentFrom:<nil> Replicas:0 Image: Path:}
In the cluster, small details like this lead to endless diffs in GitOps tools.
Save your progress? Sign up for free
Drill 5: The controller works with the hub
The operator should know only one version: the
storage version v1. Old clients may keep sending
v1alpha1 – the API server converts before your informer
sees the object.
OLD='webv1alpha1 "example.com/site-operator/api/v1alpha1"'
NEW='webv1 "example.com/site-operator/api/v1"'
sed -i '' -e "s#$OLD#$NEW#" \
-e 's/webv1alpha1\./webv1./g; s/ContentFrom/Source/g' \
-e 's/spec\.contentFrom\./spec.source./g; s/contentFrom/source/g' \
internal/controller/*.go
grep -l 'v1alpha1' internal/controller/*.go
go vet ./... && go test ./internal/controller/ -count=1 2>&1 | tail -1Expected:
ok example.com/site-operator/internal/controller 13.032s
grep -l finds nothing anymore: the controller, status,
finalizer and all controller tests (including the CEL table from Day 8)
now speak v1. The admission webhooks from Day 8 stay on
v1alpha1 – the API server converts for them too
(matchPolicy: Equivalent).
Warning:
sed -i ''is the macOS syntax. On Linux it issed -iwithout the empty quotes.
Save your progress? Sign up for free
Drill 6: Conversion in envtest
Envtest sets up the conversion webhook itself – but only for types it
can recognize as convertible in the scheme, that is, when
both versions are registered. And because all webhook
configurations are loaded from config/webhook/, every suite
must also run all handlers.
In internal/webhook/v1/webhook_suite_test.go (add the
imports webv1alpha1 ".../api/v1alpha1" and
webhookv1alpha1 ".../internal/webhook/v1alpha1"):
err = webv1.AddToScheme(scheme.Scheme)
Expect(err).NotTo(HaveOccurred())
Expect(webv1alpha1.AddToScheme(scheme.Scheme)).To(Succeed())
// ...
err = SetupStaticSiteWebhookWithManager(mgr)
Expect(err).NotTo(HaveOccurred())
Expect(webhookv1alpha1.SetupStaticSiteWebhookWithManager(mgr)).To(Succeed())The mirror image in
internal/webhook/v1alpha1/webhook_suite_test.go (imports
webv1 ".../api/v1" and
webhookv1 ".../internal/webhook/v1"):
Expect(webv1.AddToScheme(scheme.Scheme)).To(Succeed())
// ...
Expect(webhookv1.SetupStaticSiteWebhookWithManager(mgr)).To(Succeed())Three specs: both directions and the deprecation warning.
cat > internal/webhook/v1/staticsite_webhook_test.go <<'EOF'
package v1
import (
"context"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/rest"
"sigs.k8s.io/controller-runtime/pkg/client"
webv1 "example.com/site-operator/api/v1"
webv1alpha1 "example.com/site-operator/api/v1alpha1"
)
var _ = Describe("Conversion im API-Server", func() {
meta := func(name string) metav1.ObjectMeta {
return metav1.ObjectMeta{Name: name, Namespace: "default"}
}
It("liest ein v1alpha1-Objekt als v1", func() {
old := &webv1alpha1.StaticSite{ObjectMeta: meta("alt"),
Spec: webv1alpha1.StaticSiteSpec{ContentFrom: &webv1alpha1.ContentSource{
ConfigMapRef: &webv1alpha1.ConfigMapReference{Name: "html"}}}}
Expect(k8sClient.Create(ctx, old)).To(Succeed())
hub := &webv1.StaticSite{}
Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(old), hub)).To(Succeed())
Expect(hub.Spec.Source.ConfigMapRef.Name).To(Equal("html"))
})
It("liest ein v1-Objekt als v1alpha1", func() {
neu := &webv1.StaticSite{ObjectMeta: meta("neu"),
Spec: webv1.StaticSiteSpec{Source: &webv1.ContentSource{
ConfigMapRef: &webv1.ConfigMapReference{Name: "html"}}}}
Expect(k8sClient.Create(ctx, neu)).To(Succeed())
spoke := &webv1alpha1.StaticSite{}
Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(neu), spoke)).
To(Succeed())
Expect(spoke.Spec.ContentFrom.ConfigMapRef.Name).To(Equal("html"))
Expect(spoke.Spec.Path).To(Equal("/")) // Default aus dem v1-Schema
})
It("warnt bei Nutzung von v1alpha1", func() {
var got warnings
cfgW := rest.CopyConfig(cfg)
cfgW.WarningHandlerWithContext = &got
c, err := client.New(cfgW, client.Options{Scheme: k8sClient.Scheme()})
Expect(err).NotTo(HaveOccurred())
Expect(c.Get(ctx, client.ObjectKey{Namespace: "default", Name: "neu"},
&webv1alpha1.StaticSite{})).To(Succeed())
Expect(got).To(ContainElement("v1alpha1 ist veraltet, nutze v1"))
})
})
// warnings collects warning headers from the API server.
type warnings []string
func (w *warnings) HandleWarningHeaderWithContext(_ context.Context, _ int,
_ string, text string) {
*w = append(*w, text)
}
EOF
go test ./internal/webhook/... -count=1 -v -ginkgo.no-color \
2>&1 | grep -E '^(Ran|SUCCESS|FAIL|ok)'Expected:
Ran 3 of 3 Specs in 5.315 seconds
SUCCESS! -- 3 Passed | 0 Failed | 0 Pending | 0 Skipped
ok example.com/site-operator/internal/webhook/v1 5.687s
Ran 7 of 7 Specs in 5.006 seconds
SUCCESS! -- 7 Passed | 0 Failed | 0 Pending | 0 Skipped
ok example.com/site-operator/internal/webhook/v1alpha1 5.690s
The second spec also shows in passing: the default
path: / comes from the v1 schema and survives the way back
to v1alpha1.
Save your progress? Sign up for free
Drill 7: Live – turn on the webhook and migrate
Back to the old objects from Drill 3 (Terminal 1 is still running).
In a cluster, kustomize would apply the patch from
config/crd/patches/ and cert-manager would fill in the CA
bundle. Against envtest-up you do both by hand – with the
lab certificate from Day 8 (lab/certs came along with
cp -r):
kubectl apply -f lab/webhooks-local.yaml
CA=$(base64 < lab/certs/tls.crt | tr -d '\n')
kubectl patch crd staticsites.web.lab.example --type=merge -p '{"spec":{"conversion":{
"strategy":"Webhook","webhook":{"conversionReviewVersions":["v1"],
"clientConfig":{"url":"https://127.0.0.1:9443/convert","caBundle":"'"$CA"'"}}}}}'Expected (shortened):
customresourcedefinition.apiextensions.k8s.io/staticsites.web.lab.example patched
Terminal 3: start the operator – it now also serves
/convert:
cd ~/op-lab/tag09-site-operator
go run ./cmd/main.go -health-probe-bind-address=0 --webhook-cert-path=lab/certs \
2>&1 | tee run.logTerminal 2: read the same object in both versions:
kubectl get staticsites.v1.web.lab.example extern -o jsonpath='{.spec.source}{"\n"}'
kubectl get staticsites.v1alpha1.web.lab.example extern \
-o jsonpath='{.spec.contentFrom}{"\n"}'
kubectl get siteExpected:
{"configMapRef":{"name":"extern-html"}}
Warning: v1alpha1 ist veraltet, nutze v1
{"configMapRef":{"name":"extern-html"}}
NAME READY AVAILABLE URL AGE
demo False http://demo.default.svc 11m
extern False http://extern.default.svc 11m
The source is back – it was in etcd as v1alpha1 the
whole time, and the webhook translates on read. Now the storage
migration: read every object once and write it back unchanged,
and it ends up in etcd as v1. Only then may
v1alpha1 disappear from storedVersions:
kubectl get staticsites.v1.web.lab.example -A -o json | kubectl replace -f -
kubectl patch crd staticsites.web.lab.example --subresource=status --type=merge \
-p '{"status":{"storedVersions":["v1"]}}'
kubectl get crd staticsites.web.lab.example \
-o jsonpath='{.status.storedVersions}{"\n"}'Expected:
staticsite.web.lab.example/demo replaced
Warning: ConfigMap extern-html fehlt (noch)
staticsite.web.lab.example/extern replaced
customresourcedefinition.apiextensions.k8s.io/staticsites.web.lab.example patched
["v1"]
The warning comes from your validating webhook from Day 8 – a
replace is an update too and runs through the whole
admission chain.
Warning: Order is everything. Trimming
storedVersionsbefore all objects have been rewritten leaves objects behind that nobody can read anymore as soon asv1alpha1is removed from the CRD.
Save your progress? Sign up for free
Checkpoint
cd ~/op-lab/tag09-site-operator
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
CRD=config/crd/bases/web.lab.example_staticsites.yaml
go test ./... -count=1 2>&1 | grep -q FAIL && nok "Tests" || ok "Tests grün"
grep -A1 'name: v1$' -m1 $CRD >/dev/null && grep -q 'storage: true' $CRD \
&& ok "v1 in der CRD" || nok "v1 fehlt"
grep -q 'func (\*StaticSite) Hub()' api/v1/staticsite_conversion.go \
&& ok "v1 ist Hub" || nok "Hub fehlt"
kubectl get staticsites.v1.web.lab.example extern -o jsonpath='{.spec.source}' \
| grep -q extern-html && ok "Conversion live" || nok "Conversion (Operator?)"
kubectl get crd staticsites.web.lab.example -o jsonpath='{.status.storedVersions}' \
| grep -qx '\["v1"\]' && ok "Migration abgeschlossen" || nok "storedVersions"Expected:
✔ Tests grün
✔ v1 in der CRD
✔ v1 ist Hub
✔ Conversion live
✔ Migration abgeschlossen
Then stop the operator and envtest-up.
Challenge
v1 gets a new, optional field spec.title
(max. 200 characters) that v1alpha1 does not know. Make
sure a title survives the trip hub → spoke → hub, and prove it with a
test.
Hint: Whatever the spoke does not know as a field,
it parks in an annotation (web.lab.example/v1-title) – and
ConvertTo fetches it back and removes the annotation
again.
Solution
In api/v1/staticsite_types.go at the end of
StaticSiteSpec:
// title is the page title (new in v1).
// +kubebuilder:validation:MaxLength=200
// +optional
Title string `json:"title,omitempty"`In api/v1alpha1/staticsite_conversion.go (import
"maps"): one constant and one block each at the end of
ConvertTo and ConvertFrom:
// titleAnnotation parks v1 fields that v1alpha1 does not know.
const titleAnnotation = "web.lab.example/v1-title"
// ConvertTo, before return nil:
if t, ok := src.Annotations[titleAnnotation]; ok { // v1-Feld zurückholen
dst.Spec.Title = t
dst.Annotations = maps.Clone(src.Annotations)
delete(dst.Annotations, titleAnnotation)
}
// ConvertFrom, before return nil:
if src.Spec.Title != "" { // v1alpha1 kennt title nicht: parken
dst.Annotations = maps.Clone(src.Annotations)
if dst.Annotations == nil {
dst.Annotations = map[string]string{}
}
dst.Annotations[titleAnnotation] = src.Spec.Title
}maps.Clone matters:
dst.ObjectMeta = src.ObjectMeta shares the map with the
source object – without a copy, you would modify the hub object as a
side effect.
cat >> api/v1alpha1/staticsite_conversion_test.go <<'EOF'
// Hub -> spoke -> hub: the new field title survives in an annotation.
func TestRoundTripHub(t *testing.T) {
in := &webv1.StaticSite{ObjectMeta: metav1.ObjectMeta{Name: "rt"},
Spec: webv1.StaticSiteSpec{Content: "<h1>x</h1>", Title: "Willkommen"}}
spoke := &StaticSite{}
if err := spoke.ConvertFrom(in); err != nil {
t.Fatal(err)
}
out := &webv1.StaticSite{}
if err := spoke.ConvertTo(out); err != nil {
t.Fatal(err)
}
if !equality.Semantic.DeepEqual(in, out) {
t.Errorf("Round-Trip verändert:\n in %+v\nout %+v", in, out)
}
if in.Annotations != nil {
t.Errorf("Hub-Objekt verändert: %v", in.Annotations)
}
}
EOF
make manifests generate
go test ./api/... -run RoundTrip -v 2>&1 | grep -E '^(---|ok)'Expected:
--- PASS: TestRoundTrip (0.00s)
--- PASS: TestRoundTripHub (0.00s)
ok example.com/site-operator/api/v1alpha1 0.163s
Save your progress? Sign up for free
Enterprise Note
- Versions are promises:
v1alpha1may break,v1may not. New fields inv1only as optional, renames only through a new version with conversion. - The conversion webhook sits in the read path of
every access to older versions – including
kubectl getand other controllers. It needs HA (≥ 2 replicas, PDB) like an admission webhook, otherwise the CRD’s entire API is gone. - Migration in the cluster: as in Drill 7, paginated for many objects
– or with the kube-storage-version-migrator. Only then trim
storedVersions, thenserved: false, and remove the old version only in the next release. - Helm, Argo CD and Flux keep using the old version as long as manifests still contain it: switch charts and repos before turning it off (the deprecation warning helps you find them).
- Round-trip tests for every spoke version belong in
CI; fuzzing-based tests
(
k8s.io/apimachinery/pkg/api/apitesting/roundtrip) find lost fields systematically.