Skip to content
Hands-on IT IT Drills

Day 9: API Versions and Conversion

Goal: The StaticSite gets a stable version v1 (the field contentFrom is called source there), which becomes the storage version, and a conversion webhook translates losslessly between v1alpha1 and v1 – including migration of the stored objects · Duration: approx. 150 min · Prerequisite: Day 8

Briefing

  • A CRD can serve several versions (served), but store in only one (storage). Everything else is converted on read and write.
  • Without a webhook (strategy: None), the API server only swaps apiVersion – renamed fields get lost along the way.
  • Hub & spoke: one version is the hub (Hub()), every other one is a spoke with ConvertTo/ConvertFrom. With n versions you need n-1 pairs, not n².
  • status.storedVersions remembers every version that was ever in etcd. You remove an old version only after the migration.
kubectl get ...v1alpha1...           kubectl get ...v1...
        |                                   |
   v1alpha1 (spoke)  <-- ConvertFrom --  v1 (hub, storage)  <--> etcd
                     --- ConvertTo  -->
             POST /convert  (conversion webhook in the operator)

Drill 1: Create version v1

source ~/op-lab/env.sh
cp -r ~/op-lab/tag08-site-operator ~/op-lab/tag09-site-operator
cd ~/op-lab/tag09-site-operator
kubebuilder create api --group web --version v1 --kind StaticSite \
  --resource --controller=false

Expected (shortened):

Next: implement your new API and generate the manifests (e.g. CRDs,CRs) with:
$ make manifests

Kubebuilder created api/v1/ and added webv1.AddToScheme to cmd/main.go. --controller=false: there is still one controller – from Drill 5 on, it works with v1.

Save your progress? Sign up for free

Drill 2: The v1 types

v1 is v1alpha1 with one renamed field and the storageversion marker. Instead of typing 100 lines, you derive the file and let the tooling show you the difference – exactly this diff is your API change:

sed -e 's/^package v1alpha1/package v1/' \
    -e 's/ExactlyOneOf=content;contentFrom/ExactlyOneOf=content;source/' \
    -e 's/ContentFrom \*ContentSource/Source *ContentSource/' \
    -e 's/json:"contentFrom,omitempty"/json:"source,omitempty"/' \
    -e 's/zu contentFrom)/zu source)/; s|// contentFrom verweist|// source verweist|' \
    -e 's|^// +kubebuilder:subresource:status|&\n// +kubebuilder:storageversion|' \
    api/v1alpha1/staticsite_types.go > api/v1/staticsite_types.go
sed 's/^package v1alpha1/package v1/' api/v1alpha1/conditions.go > api/v1/conditions.go
gofmt -l api/
diff api/v1alpha1/staticsite_types.go api/v1/staticsite_types.go

Expected:

1c1
< package v1alpha1
---
> package v1
24c24
< // +kubebuilder:validation:ExactlyOneOf=content;contentFrom
---
> // +kubebuilder:validation:ExactlyOneOf=content;source
26c26
<   // content ist der HTML-Inhalt der Seite (alternativ zu contentFrom).
---
>   // content ist der HTML-Inhalt der Seite (alternativ zu source).
31c31
<   // contentFrom verweist auf eine vorhandene ConfigMap (alternativ zu content).
---
>   // source verweist auf eine vorhandene ConfigMap (alternativ zu content).
33c33
<   ContentFrom *ContentSource `json:"contentFrom,omitempty"`
---
>   Source *ContentSource `json:"source,omitempty"`
77a78
> // +kubebuilder:storageversion

The old version gets a deprecation notice. In api/v1alpha1/staticsite_types.go, directly below // +kubebuilder:subresource:status:

// +kubebuilder:deprecatedversion:warning="v1alpha1 ist veraltet, nutze v1"
make manifests generate
grep -E '^    (name|served|storage|deprecat)' \
  config/crd/bases/web.lab.example_staticsites.yaml

Expected (shortened):

    name: v1
    served: true
    storage: true
    deprecated: true
    deprecationWarning: v1alpha1 ist veraltet, nutze v1
    name: v1alpha1
    served: true
    storage: false

Save your progress? Sign up for free

Drill 3: Without conversion, something gets lost

Simulate a cluster that already holds StaticSites from yesterday. Terminal 1 with the CRD from Day 8:

source ~/op-lab/env.sh
envtest-up --crds ~/op-lab/tag08-site-operator/config/crd/bases

Terminal 2: create two old objects, then apply the new CRD – still without a webhook:

kubectl apply -f lab/demo.yaml -f lab/extern.yaml
kubectl replace -f config/crd/bases/
kubectl get crd staticsites.web.lab.example \
  -o jsonpath='{.status.storedVersions}{"\n"}'
kubectl get staticsites.v1.web.lab.example extern -o jsonpath='{.spec}{"\n"}'

Expected:

staticsite.web.lab.example/demo created
staticsite.web.lab.example/extern created
customresourcedefinition.apiextensions.k8s.io/staticsites.web.lab.example replaced
["v1alpha1","v1"]
{"image":"nginxinc/nginx-unprivileged:1.31-alpine","path":"/","replicas":1}

Read as v1, extern shows no source anymore: contentFrom is unknown in the v1 schema and gets cut away (pruning). It is still in etcd – but the first write through v1 would delete it for good. That is exactly what the conversion webhook is for.

Save your progress? Sign up for free

Drill 4: Hub and spoke

kubebuilder create webhook --group web --version v1 --kind StaticSite \
  --conversion --spoke v1alpha1
cat api/v1/staticsite_conversion.go | grep -v '^//' | grep .

Expected (shortened):

package v1
func (*StaticSite) Hub() {}

The hub is done. Kubebuilder also created internal/webhook/v1/ (webhook server for /convert), the call in cmd/main.go and the patch config/crd/patches/webhook_in_staticsites.yaml (strategy Webhook). The actual work lives in the spoke:

cat > api/v1alpha1/staticsite_conversion.go <<'EOF'
package v1alpha1

import (
    "sigs.k8s.io/controller-runtime/pkg/conversion"

    webv1 "example.com/site-operator/api/v1"
)

// ConvertTo converts this spoke version (v1alpha1) to the hub (v1).
func (src *StaticSite) ConvertTo(dstRaw conversion.Hub) error {
    dst := dstRaw.(*webv1.StaticSite)
    dst.ObjectMeta = src.ObjectMeta
    dst.Spec = webv1.StaticSiteSpec{
        Content:  src.Spec.Content,
        Replicas: src.Spec.Replicas,
        Image:    src.Spec.Image,
        Path:     src.Spec.Path,
    }
    if from := src.Spec.ContentFrom; from != nil { // contentFrom -> source
        dst.Spec.Source = &webv1.ContentSource{}
        if from.ConfigMapRef != nil {
            dst.Spec.Source.ConfigMapRef = &webv1.ConfigMapReference{
                Name: from.ConfigMapRef.Name}
        }
    }
    dst.Status = webv1.StaticSiteStatus(src.Status) // identische Felder
    return nil
}

// ConvertFrom converts the hub (v1) back to v1alpha1.
func (dst *StaticSite) ConvertFrom(srcRaw conversion.Hub) error {
    src := srcRaw.(*webv1.StaticSite)
    dst.ObjectMeta = src.ObjectMeta
    dst.Spec = StaticSiteSpec{
        Content:  src.Spec.Content,
        Replicas: src.Spec.Replicas,
        Image:    src.Spec.Image,
        Path:     src.Spec.Path,
    }
    if s := src.Spec.Source; s != nil { // source -> contentFrom
        dst.Spec.ContentFrom = &ContentSource{}
        if s.ConfigMapRef != nil {
            dst.Spec.ContentFrom.ConfigMapRef = &ConfigMapReference{
                Name: s.ConfigMapRef.Name}
        }
    }
    dst.Status = StaticSiteStatus(src.Status)
    return nil
}
EOF

webv1.StaticSiteStatus(src.Status) is a Go type conversion: it works as long as both structs have exactly the same fields – if the status changes in one version, the compiler reports it immediately.

The most important test for any conversion is the round trip spoke → hub → spoke:

cat > api/v1alpha1/staticsite_conversion_test.go <<'EOF'
package v1alpha1

import (
    "testing"

    "k8s.io/apimachinery/pkg/api/equality"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

    webv1 "example.com/site-operator/api/v1"
)

// Spoke -> hub -> spoke must be lossless, otherwise every
// read through the other version loses data.
func TestRoundTrip(t *testing.T) {
    ref := &ConfigMapReference{Name: "html"}
    tests := map[string]StaticSiteSpec{
        "content":      {Content: "<h1>Hallo</h1>", Replicas: 2, Path: "/docs/"},
        "contentFrom":  {ContentFrom: &ContentSource{ConfigMapRef: ref}},
        "leere Quelle": {ContentFrom: &ContentSource{}},
    }
    for name, spec := range tests {
        t.Run(name, func(t *testing.T) {
            in := &StaticSite{ObjectMeta: metav1.ObjectMeta{Name: "rt"}, Spec: spec}
            in.Status.URL = "http://rt.default.svc"
            hub := &webv1.StaticSite{}
            if err := in.ConvertTo(hub); err != nil {
                t.Fatal(err)
            }
            out := &StaticSite{}
            if err := out.ConvertFrom(hub); err != nil {
                t.Fatal(err)
            }
            if !equality.Semantic.DeepEqual(in, out) {
                t.Errorf("Round-Trip verändert:\n in %+v\nout %+v", in.Spec, out.Spec)
            }
        })
    }
}
EOF
go test ./api/... -run RoundTrip -v 2>&1 | grep -E '^(---|    ---|ok)'

Expected:

--- PASS: TestRoundTrip (0.00s)
    --- PASS: TestRoundTrip/content (0.00s)
    --- PASS: TestRoundTrip/contentFrom (0.00s)
    --- PASS: TestRoundTrip/leere_Quelle (0.00s)
ok      example.com/site-operator/api/v1alpha1  0.153s

Why the case “leere Quelle” (empty source)? The obvious shorthand if from != nil && from.ConfigMapRef != nil turns contentFrom: {} into nil on the way back:

--- FAIL: TestRoundTrip (0.00s)
    --- FAIL: TestRoundTrip/leere_Quelle (0.00s)
        staticsite_conversion_test.go:34: Round-Trip verändert:
             in {Content: ContentFrom:0x5482827ac3b8 Replicas:0 Image: Path:}
            out {Content: ContentFrom:<nil> Replicas:0 Image: Path:}

In the cluster, small details like this lead to endless diffs in GitOps tools.

Save your progress? Sign up for free

Drill 5: The controller works with the hub

The operator should know only one version: the storage version v1. Old clients may keep sending v1alpha1 – the API server converts before your informer sees the object.

OLD='webv1alpha1 "example.com/site-operator/api/v1alpha1"'
NEW='webv1 "example.com/site-operator/api/v1"'
sed -i '' -e "s#$OLD#$NEW#" \
  -e 's/webv1alpha1\./webv1./g; s/ContentFrom/Source/g' \
  -e 's/spec\.contentFrom\./spec.source./g; s/contentFrom/source/g' \
  internal/controller/*.go
grep -l 'v1alpha1' internal/controller/*.go
go vet ./... && go test ./internal/controller/ -count=1 2>&1 | tail -1

Expected:

ok      example.com/site-operator/internal/controller   13.032s

grep -l finds nothing anymore: the controller, status, finalizer and all controller tests (including the CEL table from Day 8) now speak v1. The admission webhooks from Day 8 stay on v1alpha1 – the API server converts for them too (matchPolicy: Equivalent).

Warning: sed -i '' is the macOS syntax. On Linux it is sed -i without the empty quotes.

Save your progress? Sign up for free

Drill 6: Conversion in envtest

Envtest sets up the conversion webhook itself – but only for types it can recognize as convertible in the scheme, that is, when both versions are registered. And because all webhook configurations are loaded from config/webhook/, every suite must also run all handlers.

In internal/webhook/v1/webhook_suite_test.go (add the imports webv1alpha1 ".../api/v1alpha1" and webhookv1alpha1 ".../internal/webhook/v1alpha1"):

    err = webv1.AddToScheme(scheme.Scheme)
    Expect(err).NotTo(HaveOccurred())
    Expect(webv1alpha1.AddToScheme(scheme.Scheme)).To(Succeed())
    // ...
    err = SetupStaticSiteWebhookWithManager(mgr)
    Expect(err).NotTo(HaveOccurred())
    Expect(webhookv1alpha1.SetupStaticSiteWebhookWithManager(mgr)).To(Succeed())

The mirror image in internal/webhook/v1alpha1/webhook_suite_test.go (imports webv1 ".../api/v1" and webhookv1 ".../internal/webhook/v1"):

    Expect(webv1.AddToScheme(scheme.Scheme)).To(Succeed())
    // ...
    Expect(webhookv1.SetupStaticSiteWebhookWithManager(mgr)).To(Succeed())

Three specs: both directions and the deprecation warning.

cat > internal/webhook/v1/staticsite_webhook_test.go <<'EOF'
package v1

import (
    "context"

    . "github.com/onsi/ginkgo/v2"
    . "github.com/onsi/gomega"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/rest"
    "sigs.k8s.io/controller-runtime/pkg/client"

    webv1 "example.com/site-operator/api/v1"
    webv1alpha1 "example.com/site-operator/api/v1alpha1"
)

var _ = Describe("Conversion im API-Server", func() {
    meta := func(name string) metav1.ObjectMeta {
        return metav1.ObjectMeta{Name: name, Namespace: "default"}
    }

    It("liest ein v1alpha1-Objekt als v1", func() {
        old := &webv1alpha1.StaticSite{ObjectMeta: meta("alt"),
            Spec: webv1alpha1.StaticSiteSpec{ContentFrom: &webv1alpha1.ContentSource{
                ConfigMapRef: &webv1alpha1.ConfigMapReference{Name: "html"}}}}
        Expect(k8sClient.Create(ctx, old)).To(Succeed())

        hub := &webv1.StaticSite{}
        Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(old), hub)).To(Succeed())
        Expect(hub.Spec.Source.ConfigMapRef.Name).To(Equal("html"))
    })

    It("liest ein v1-Objekt als v1alpha1", func() {
        neu := &webv1.StaticSite{ObjectMeta: meta("neu"),
            Spec: webv1.StaticSiteSpec{Source: &webv1.ContentSource{
                ConfigMapRef: &webv1.ConfigMapReference{Name: "html"}}}}
        Expect(k8sClient.Create(ctx, neu)).To(Succeed())

        spoke := &webv1alpha1.StaticSite{}
        Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(neu), spoke)).
            To(Succeed())
        Expect(spoke.Spec.ContentFrom.ConfigMapRef.Name).To(Equal("html"))
        Expect(spoke.Spec.Path).To(Equal("/")) // Default aus dem v1-Schema
    })

    It("warnt bei Nutzung von v1alpha1", func() {
        var got warnings
        cfgW := rest.CopyConfig(cfg)
        cfgW.WarningHandlerWithContext = &got
        c, err := client.New(cfgW, client.Options{Scheme: k8sClient.Scheme()})
        Expect(err).NotTo(HaveOccurred())
        Expect(c.Get(ctx, client.ObjectKey{Namespace: "default", Name: "neu"},
            &webv1alpha1.StaticSite{})).To(Succeed())
        Expect(got).To(ContainElement("v1alpha1 ist veraltet, nutze v1"))
    })
})

// warnings collects warning headers from the API server.
type warnings []string

func (w *warnings) HandleWarningHeaderWithContext(_ context.Context, _ int,
    _ string, text string) {
    *w = append(*w, text)
}
EOF
go test ./internal/webhook/... -count=1 -v -ginkgo.no-color \
  2>&1 | grep -E '^(Ran|SUCCESS|FAIL|ok)'

Expected:

Ran 3 of 3 Specs in 5.315 seconds
SUCCESS! -- 3 Passed | 0 Failed | 0 Pending | 0 Skipped
ok      example.com/site-operator/internal/webhook/v1   5.687s
Ran 7 of 7 Specs in 5.006 seconds
SUCCESS! -- 7 Passed | 0 Failed | 0 Pending | 0 Skipped
ok      example.com/site-operator/internal/webhook/v1alpha1 5.690s

The second spec also shows in passing: the default path: / comes from the v1 schema and survives the way back to v1alpha1.

Save your progress? Sign up for free

Drill 7: Live – turn on the webhook and migrate

Back to the old objects from Drill 3 (Terminal 1 is still running). In a cluster, kustomize would apply the patch from config/crd/patches/ and cert-manager would fill in the CA bundle. Against envtest-up you do both by hand – with the lab certificate from Day 8 (lab/certs came along with cp -r):

kubectl apply -f lab/webhooks-local.yaml
CA=$(base64 < lab/certs/tls.crt | tr -d '\n')
kubectl patch crd staticsites.web.lab.example --type=merge -p '{"spec":{"conversion":{
  "strategy":"Webhook","webhook":{"conversionReviewVersions":["v1"],
  "clientConfig":{"url":"https://127.0.0.1:9443/convert","caBundle":"'"$CA"'"}}}}}'

Expected (shortened):

customresourcedefinition.apiextensions.k8s.io/staticsites.web.lab.example patched

Terminal 3: start the operator – it now also serves /convert:

cd ~/op-lab/tag09-site-operator
go run ./cmd/main.go -health-probe-bind-address=0 --webhook-cert-path=lab/certs \
  2>&1 | tee run.log

Terminal 2: read the same object in both versions:

kubectl get staticsites.v1.web.lab.example extern -o jsonpath='{.spec.source}{"\n"}'
kubectl get staticsites.v1alpha1.web.lab.example extern \
  -o jsonpath='{.spec.contentFrom}{"\n"}'
kubectl get site

Expected:

{"configMapRef":{"name":"extern-html"}}
Warning: v1alpha1 ist veraltet, nutze v1
{"configMapRef":{"name":"extern-html"}}
NAME     READY   AVAILABLE   URL                         AGE
demo             False       http://demo.default.svc     11m
extern           False       http://extern.default.svc   11m

The source is back – it was in etcd as v1alpha1 the whole time, and the webhook translates on read. Now the storage migration: read every object once and write it back unchanged, and it ends up in etcd as v1. Only then may v1alpha1 disappear from storedVersions:

kubectl get staticsites.v1.web.lab.example -A -o json | kubectl replace -f -
kubectl patch crd staticsites.web.lab.example --subresource=status --type=merge \
  -p '{"status":{"storedVersions":["v1"]}}'
kubectl get crd staticsites.web.lab.example \
  -o jsonpath='{.status.storedVersions}{"\n"}'

Expected:

staticsite.web.lab.example/demo replaced
Warning: ConfigMap extern-html fehlt (noch)
staticsite.web.lab.example/extern replaced
customresourcedefinition.apiextensions.k8s.io/staticsites.web.lab.example patched
["v1"]

The warning comes from your validating webhook from Day 8 – a replace is an update too and runs through the whole admission chain.

Warning: Order is everything. Trimming storedVersions before all objects have been rewritten leaves objects behind that nobody can read anymore as soon as v1alpha1 is removed from the CRD.

Save your progress? Sign up for free

Checkpoint

cd ~/op-lab/tag09-site-operator
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
CRD=config/crd/bases/web.lab.example_staticsites.yaml
go test ./... -count=1 2>&1 | grep -q FAIL && nok "Tests" || ok "Tests grün"
grep -A1 'name: v1$' -m1 $CRD >/dev/null && grep -q 'storage: true' $CRD \
  && ok "v1 in der CRD" || nok "v1 fehlt"
grep -q 'func (\*StaticSite) Hub()' api/v1/staticsite_conversion.go \
  && ok "v1 ist Hub" || nok "Hub fehlt"
kubectl get staticsites.v1.web.lab.example extern -o jsonpath='{.spec.source}' \
  | grep -q extern-html && ok "Conversion live" || nok "Conversion (Operator?)"
kubectl get crd staticsites.web.lab.example -o jsonpath='{.status.storedVersions}' \
  | grep -qx '\["v1"\]' && ok "Migration abgeschlossen" || nok "storedVersions"

Expected:

✔ Tests grün
✔ v1 in der CRD
✔ v1 ist Hub
✔ Conversion live
✔ Migration abgeschlossen

Then stop the operator and envtest-up.

Challenge

v1 gets a new, optional field spec.title (max. 200 characters) that v1alpha1 does not know. Make sure a title survives the trip hub → spoke → hub, and prove it with a test.

Hint: Whatever the spoke does not know as a field, it parks in an annotation (web.lab.example/v1-title) – and ConvertTo fetches it back and removes the annotation again.

Solution

In api/v1/staticsite_types.go at the end of StaticSiteSpec:

    // title is the page title (new in v1).
    // +kubebuilder:validation:MaxLength=200
    // +optional
    Title string `json:"title,omitempty"`

In api/v1alpha1/staticsite_conversion.go (import "maps"): one constant and one block each at the end of ConvertTo and ConvertFrom:

// titleAnnotation parks v1 fields that v1alpha1 does not know.
const titleAnnotation = "web.lab.example/v1-title"

    // ConvertTo, before return nil:
    if t, ok := src.Annotations[titleAnnotation]; ok { // v1-Feld zurückholen
        dst.Spec.Title = t
        dst.Annotations = maps.Clone(src.Annotations)
        delete(dst.Annotations, titleAnnotation)
    }

    // ConvertFrom, before return nil:
    if src.Spec.Title != "" { // v1alpha1 kennt title nicht: parken
        dst.Annotations = maps.Clone(src.Annotations)
        if dst.Annotations == nil {
            dst.Annotations = map[string]string{}
        }
        dst.Annotations[titleAnnotation] = src.Spec.Title
    }

maps.Clone matters: dst.ObjectMeta = src.ObjectMeta shares the map with the source object – without a copy, you would modify the hub object as a side effect.

cat >> api/v1alpha1/staticsite_conversion_test.go <<'EOF'

// Hub -> spoke -> hub: the new field title survives in an annotation.
func TestRoundTripHub(t *testing.T) {
    in := &webv1.StaticSite{ObjectMeta: metav1.ObjectMeta{Name: "rt"},
        Spec: webv1.StaticSiteSpec{Content: "<h1>x</h1>", Title: "Willkommen"}}
    spoke := &StaticSite{}
    if err := spoke.ConvertFrom(in); err != nil {
        t.Fatal(err)
    }
    out := &webv1.StaticSite{}
    if err := spoke.ConvertTo(out); err != nil {
        t.Fatal(err)
    }
    if !equality.Semantic.DeepEqual(in, out) {
        t.Errorf("Round-Trip verändert:\n in %+v\nout %+v", in, out)
    }
    if in.Annotations != nil {
        t.Errorf("Hub-Objekt verändert: %v", in.Annotations)
    }
}
EOF
make manifests generate
go test ./api/... -run RoundTrip -v 2>&1 | grep -E '^(---|ok)'

Expected:

--- PASS: TestRoundTrip (0.00s)
--- PASS: TestRoundTripHub (0.00s)
ok      example.com/site-operator/api/v1alpha1  0.163s

Save your progress? Sign up for free

Enterprise Note

  • Versions are promises: v1alpha1 may break, v1 may not. New fields in v1 only as optional, renames only through a new version with conversion.
  • The conversion webhook sits in the read path of every access to older versions – including kubectl get and other controllers. It needs HA (≥ 2 replicas, PDB) like an admission webhook, otherwise the CRD’s entire API is gone.
  • Migration in the cluster: as in Drill 7, paginated for many objects – or with the kube-storage-version-migrator. Only then trim storedVersions, then served: false, and remove the old version only in the next release.
  • Helm, Argo CD and Flux keep using the old version as long as manifests still contain it: switch charts and repos before turning it off (the deprecation warning helps you find them).
  • Round-trip tests for every spoke version belong in CI; fuzzing-based tests (k8s.io/apimachinery/pkg/api/apitesting/roundtrip) find lost fields systematically.