Day 13: Hybrid Mode
Goal: Split Kong into a control plane with a database and data planes without a database, connect a second data plane on its own VM, and see what happens on failure and with mismatched plugins. · Duration: approx. 90 min · Prerequisite: Day 12
Briefing
- The control plane (CP) has Postgres, Admin API and Kong Manager, but processes no traffic.
- Data planes (DP) have no database. They connect to the CP via port 8005, receive the entire configuration and keep it in memory.
- CP and DP trust each other through a shared cluster certificate (mTLS).
- If the CP fails, the DPs keep working with the last configuration.
VM kong VM dp
┌──────────────────────────────────┐ ┌────────────────────────┐
│ kong (CP) + kong-db :8001 :8005 ◄─── mTLS ───┤ kong-dp :8000 │
│ kong-dp (DP) :8000 ─► notes-api … │ notes-api, echo (local)│
└──────────────────────────────────┘ └────────────────────────┘
Drill 1: The Cluster Certificate
source ~/kong-lab/setup/env.sh
mkdir -p ~/kong-lab/tag13 && cd ~/kong-lab/kong
mkdir -p cluster
docker run --rm -u $(id -u):$(id -g) -v $PWD/cluster:/c -w /c kong:3.9.4 \
kong hybrid gen_cert cluster.crt cluster.key
openssl x509 -in cluster/cluster.crt -noout -subject -enddate
ls -l cluster | awk 'NR>1 {print $1, $3, $9}'Expected (shortened):
Successfully generated certificate/key pairs, they have been written to:
'/c/cluster.crt' and '/c/cluster.key'.
subject=CN = kong_clustering
notAfter=Oct 31 13:30:30 2029 GMT
-rw-r--r-- ubuntu cluster.crt
-rw------- ubuntu cluster.key
A self-signed certificate, valid for three years. The CP and all DPs
get the same pair. The key is readable only by its owner, and that is
still ubuntu. It does not belong in Git:
echo 'cluster/' > .gitignore
git add .gitignore && git commit -qm "Cluster-Schlüssel nicht ins Git"
git status --short | wc -lExpected:
0
Save your progress? Sign up for free
Drill 2: Control Plane and Local Data Plane
cp ~/kong-lab/code/tag13/compose.yaml compose.yaml
grep -E 'KONG_ROLE|CLUSTER_LISTEN|CLUSTER_CONTROL_PLANE' compose.yaml
docker compose up -d --waitExpected (shortened):
KONG_ROLE: control_plane
KONG_CLUSTER_LISTEN: "0.0.0.0:8005"
KONG_ROLE: data_plane
KONG_CLUSTER_CONTROL_PLANE: kong:8005
Container kong-kong-1 Error dependency kong failed to start
dependency failed to start: container kong-kong-1 exited (1)
The CP does not start. The log says why:
docker compose logs kong 2>&1 \
| grep -m1 -o 'cannot load certificate key.*Permission denied'
docker run --rm kong:3.9.4 idExpected:
cannot load certificate key "/kong/cluster/cluster.key": BIO_new_file() failed (SSL:
error:8000000D:system library::Permission denied
uid=1001(kong) gid=1001(kong) groups=1001(kong)
Kong runs in the container as UID 1001; the key belongs to UID 1000
(ubuntu) with mode 600. Inside the container, the number
counts, not the name. First copy the pair for the second VM (Drill 5),
then give the key to Kong:
cp cluster/cluster.crt cluster/cluster.key ~/kong-lab/tag13/
sudo chown 1001:1001 cluster/cluster.key
docker compose up -d --wait
docker compose ps --format "table {{.Service}}\t{{.Status}}"Expected (shortened):
Container kong-kong-1 Healthy
Container kong-kong-dp-1 Healthy
SERVICE STATUS
kong Up 13 seconds (healthy)
kong-db Up 50 minutes (healthy)
kong-dp Up 6 seconds (healthy)
Save your progress? Sign up for free
Drill 3: Who Does What
curl -s $ADMIN | jq -r .configuration.role
curl -s $ADMIN/clustering/data-planes \
| jq -r '.data[] | [.hostname, .ip, .version, .sync_status] | @tsv'
curl -s -o /dev/null -w 'DP bereit: %{http_code}\n' localhost:8100/status/ready
curl -s $PROXY/api/v1/version | head -2
docker compose logs kong-dp 2>&1 \
| grep -m1 -o "in 'policy': expected one of: local, redis"Expected (shortened):
control_plane
4f93fe7c78f9 172.18.0.12 3.9.4 normal
DP bereit: 503
{
"message":"no Route matched with those values",
in 'policy': expected one of: local, redis
The DP is connected but not ready, and it does not know a single
route. It rejected the CP’s configuration: on Day 8 you
set policy: cluster for rate-limiting on
writing, that is, counters in Postgres. A DP has no database. A single
invalid setting, and the DP discards everything.
P='$..routes[?(@.name=="notes-schreiben")].plugins[?(@.name=="rate-limiting")].config'
deck file patch -s kong.yaml --selector "$P" --value 'policy:"local"' -o kong.yaml
ksync 2>/dev/null | tail -2
curl -s -o /dev/null -w 'DP bereit: %{http_code}\n' localhost:8100/status/ready
curl -s $PROXY/api/v1/version
curl -s -o /dev/null -w '%{http_code}\n' -H 'apikey: lab-key-anna-456' $PROXY/whoamiExpected:
Updated: 1
Deleted: 0
DP bereit: 200
0.2.0
200
ksync still talks to localhost:8001, that
is, to the CP. The traffic is now processed by the DP on port 8000.
Save your progress? Sign up for free
Drill 4: The Control Plane Fails
docker compose stop kong
curl -s -o /dev/null -w 'Admin API: %{http_code}\n' $ADMIN
curl -s $PROXY/api/v1/version
curl -s -o /dev/null -w '%{http_code}\n' -H 'apikey: lab-key-anna-456' $PROXY/whoami
docker compose start kong
sleep 10
curl -s $ADMIN/clustering/data-planes \
| jq -r '.data[] | [.hostname, .sync_status] | @tsv'Expected (shortened):
Container kong-kong-1 Stopped
Admin API: 000
0.3.0
200
Container kong-kong-1 Started
4f93fe7c78f9 normal
Without the CP, no changes, but traffic keeps flowing. That is the core of hybrid mode: the database is no longer a point of failure for requests.
Save your progress? Sign up for free
Drill 5: A Second VM as a Data Plane
On the laptop, create the VM dp and
bring the certificate over:
multipass launch 24.04 --name dp --cpus 1 --memory 2G --disk 15G
multipass exec dp -- mkdir -p /home/ubuntu/kong-lab/dp/cluster
multipass transfer kong:/home/ubuntu/kong-lab/tag13/cluster.crt \
kong:/home/ubuntu/kong-lab/tag13/cluster.key .
multipass transfer cluster.crt cluster.key dp:/home/ubuntu/kong-lab/dp/cluster/
rm cluster.keyIn the VM dp (via multipass shell dp):
install Docker as on Day 0, Drill 2, log in again, then companion code
and images:
git clone https://github.com/thomaszachmann/kong-zum-anfassen.git ~/kong-lab/code
cd ~/kong-lab/code/notes/notes-api
docker build -q --build-arg VERSION=0.2.0 -t notes-api:0.2.0 .
docker build -q --build-arg VERSION=0.3.0 -t notes-api:0.3.0 .
sudo chown 1001:1001 ~/kong-lab/dp/cluster/cluster.key
cp ~/kong-lab/code/tag13/dp/compose.yaml ~/kong-lab/dp/
cp -R ~/kong-lab/code/plugins ~/kong-lab/pluginsExpected (shortened):
sha256:baf6911e6f76…
sha256:ed2077bac750…
You copy the plugins right away: if a mounted directory is missing,
Docker creates it as root at startup, and later copies
fail. dp/compose.yaml brings the DP and a backend of its
own: two notes-api instances without a database and the echo service, as
at a second site. The CP’s address comes from a variable; you know the
IP of the VM kong as KONG_IP:
cd ~/kong-lab/dp
export CP_ADDR=192.168.64.7:8005
docker compose up -d --wait
sleep 5
curl -s -o /dev/null -w 'DP bereit: %{http_code}\n' localhost:8100/status/readyExpected (shortened):
Container kong-dp-kong-dp-1 Healthy
DP bereit: 503
On the VM kong, ask the CP:
curl -s $ADMIN/clustering/data-planes | jq -r '.data[] | [.ip, .sync_status] | @tsv'Expected (shortened):
172.18.0.12 normal
192.168.64.8 plugin_set_incompatible
The new DP does not know your jwt-identity plugin
(KONG_PLUGINS is set to bundled there), and
the configuration uses it. So the CP sends it nothing at all. Custom
plugins must be present on every DP and enabled there.
Back on the VM dp:
grep -E 'DP_PLUGINS|kong-plugins' compose.yaml
export DP_PLUGINS=bundled,jwt-identity
docker compose up -d --wait
sleep 8
curl -s -o /dev/null -w 'DP bereit: %{http_code}\n' localhost:8100/status/ready
for i in 1 2 3 4; do curl -s localhost:8000/api/v1/version; done
curl -s -H 'apikey: lab-key-anna-456' localhost:8000/whoami \
| jq -c '.headers."X-Consumer-Username"'
curl -s localhost:8000/api/v1/notes; echoExpected (shortened):
KONG_PLUGINS: ${DP_PLUGINS:-bundled}
KONG_LUA_PACKAGE_PATH: /kong-plugins/?.lua;;
- ../plugins:/kong-plugins:ro
Container kong-dp-kong-dp-1 Healthy
DP bereit: 200
0.3.0
0.2.0
0.3.0
0.2.0
["anna"]
[]
The same routes, consumers, keys and upstreams as on the VM
kong, but its own backends: the list of notes is empty,
because this site has no shared database.
Save your progress? Sign up for free
Drill 6: Metrics from the Data Plane
The proxy metrics are now produced in the DP, no longer in the CP. On
the VM kong, Prometheus points to kong-dp:
cd ~/kong-lab/observability
sed -i 's/"kong:8100"/"kong-dp:8100"/' prometheus.yml
docker compose restart prometheus
for i in $(seq 1 10); do curl -s -o /dev/null $PROXY/api/v1/version; done
sleep 12
curl -s localhost:9090/api/v1/targets \
| jq -r '.data.activeTargets[] | [.labels.instance,
.health] | @tsv'Expected:
Container observability-prometheus-1 Started
kong-dp:8100 up
The DP on the VM dp cannot reach Jaeger (different
network) and logs errors when sending traces. In production, every site
has its own collector.
Save the state:
cd ~/kong-lab/kong
git add compose.yaml kong.yaml && git commit -qm "Tag 13: Hybrid mit zwei Data Planes"Save your progress? Sign up for free
Checkpoint
curl -s $ADMIN/clustering/data-planes | jq -r '.data[] | .sync_status' | sort | uniq -cExpected: two data planes in sync (older entries from failed attempts disappear on their own after a while).
2 normal
curl -s -o /dev/null -w '%{http_code}\n' localhost:8100/status/readyExpected: the local DP is ready.
200
curl -s http://192.168.64.8:8000/api/v1/versionExpected: the DP on the VM dp also answers from outside
(insert the IP of dp).
0.2.0
Challenge
Create a new route through the Admin API and measure how long it takes until both data planes know it.
Note: The CP sends changes to all DPs after saving; query both in a loop.
Solution
curl -s -X POST $ADMIN/services -d name=neu -d url=http://echo:8080/anything \
| jq -r .name
curl -s -X POST $ADMIN/services/neu/routes -d 'paths[]=/neu' | jq -c .paths
t0=$(date +%s)
for dp in localhost 192.168.64.8; do
until [ "$(curl -s -o /dev/null -w '%{http_code}' http://$dp:8000/neu/get)" = 200 ]
do sleep 1; done
echo "$dp: $(( $(date +%s) - t0 )) s"
done
cd ~/kong-lab/kong && ksync 2>/dev/null | tail -1Expected (shortened, the seconds vary):
neu
["/neu"]
localhost: 6 s
192.168.64.8: 6 s
Deleted: 2
Both DPs get the change in the same push. ksync cleans
up the experiment again, because it is not in
kong.yaml.
Save your progress? Sign up for free
Enterprise Note
- Konnect is a control plane as a service from Kong: the DPs run at your site, the CP with analytics and portal in the cloud.
- Certificates: Instead of a shared pair
(
shared), each DP can have its own certificate signed by a CA (cluster_mtls = pki). Then a single DP can be blocked. - Versions: The CP must be at least as new as the DPs. When upgrading, first the CP, then the DPs (Day 14).
- Custom plugins belong in the DP’s image, not in a
mounted directory; otherwise you get exactly the
plugin_set_incompatiblefrom Drill 5. - Rate limits in hybrid mode: only
localorredis. For limits across all DPs you need Redis.