Day 9: Upstreams and Load Balancing
Goal: Distribute load across two instances of notes-api, take sick instances out automatically and give a new version to ten percent of requests as a canary. · Duration: approx. 60 min · Prerequisite: Day 8
Briefing
- An upstream is a virtual hostname with several
targets (
host:portwith a weight). A service points to it withhost: notes-upinstead of to a container. - The default algorithm is
round-robin, weighted byweight. - Active health checks query each target regularly; passive ones evaluate real responses. A sick target no longer receives requests.
- A canary is just a matter of weights: 90 to 10.
┌─► notes-api:8080 (0.2.0, weight 90)
Service notes ─► notes-up┤
host: notes-up └─► notes-api-2:8080 (0.3.0, weight 10)
Health check: GET /healthz every 2 s
Drill 1: A Second Instance
The second instance gets a new version so that you can see at
/version who is answering:
source ~/kong-lab/setup/env.sh
mkdir -p ~/kong-lab/tag09 && cd ~/kong-lab/code/notes/notes-api
docker build -q --build-arg VERSION=0.3.0 -t notes-api:0.3.0 .
cd ~/kong-lab/code/backend
grep -A2 'notes-api-2:' compose.yaml
docker compose --profile lb up -d --wait
docker compose ps --format "table {{.Service}}\t{{.Image}}\t{{.Status}}"Expected (shortened):
sha256:2b4915ed17c7…
notes-api-2: # zweite Instanz, neue Version, für Load Balancing (Tag 9)
image: notes-api:0.3.0
environment: *api-env
Container backend-notes-api-2-1 Healthy
SERVICE IMAGE STATUS
db postgres:18.6-alpine Up About an hour (healthy)
echo ghcr.io/mccutchen/go-httpbin:2.26.0 Up About an hour
notes-api notes-api:0.2.0 Up About an hour (healthy)
notes-api-2 notes-api:0.3.0 Up 2 seconds (healthy)
The lb profile additionally starts
notes-api-2. Both use the same database.
Save your progress? Sign up for free
Drill 2: One Upstream for Both
cd ~/kong-lab/tag09
cp ~/kong-lab/code/tag09/*.yaml .
grep -v '^#' upstream.yaml
deck file merge ~/kong-lab/kong/kong.yaml upstream.yaml -o ~/kong-lab/kong/kong.yaml
SEL='$.services[?(@.name=="notes" || @.name=="notes-version")]'
deck file patch -s ~/kong-lab/kong/kong.yaml --selector "$SEL" \
--value 'host:"notes-up"' -o ~/kong-lab/kong/kong.yaml
deck gateway diff ~/kong-lab/kong/kong.yaml 2>/dev/null \
| grep -E '^(creating|updating)|"host"'
ksync 2>/dev/null | tail -3Expected (shortened):
_format_version: "3.0"
upstreams:
- name: notes-up
algorithm: round-robin
targets:
- target: notes-api:8080
weight: 100
- target: notes-api-2:8080
weight: 100
updating service notes {
- "host": "notes-api",
+ "host": "notes-up",
updating service notes-version {
- "host": "notes-api",
+ "host": "notes-up",
creating upstream notes-up
creating target notes-api:8080 for upstream ef91f37a-…
creating target notes-api-2:8080 for upstream ef91f37a-…
Created: 3
Updated: 2
Deleted: 0
The services change only their host. Kong notices that
notes-up is an upstream and does not resolve it via DNS,
but through the targets.
Count who answers. /api/v1/version is a good fit because
the route has no rate limit:
zaehle() { for i in $(seq 1 ${1:-20}); do curl -s $PROXY/api/v1/version; done \
| sort | uniq -c; }
zaehle 20
curl -s $ADMIN/upstreams/notes-up/health | jq -r '.data[] | [.target, .health] | @tsv'Expected:
10 0.2.0
10 0.3.0
notes-api-2:8080 HEALTHCHECKS_OFF
notes-api:8080 HEALTHCHECKS_OFF
Strictly alternating. HEALTHCHECKS_OFF means: Kong does
not yet check whether the targets are alive.
Save your progress? Sign up for free
Drill 3: Health Checks
health.yaml is a patch file for decK: Kong should query
/healthz every two seconds and take a target out after two
failures.
grep -v '^#' health.yaml | sed -n '3,16p'
deck file patch -s ~/kong-lab/kong/kong.yaml health.yaml -o ~/kong-lab/kong/kong.yaml
ksync 2>/dev/null | tail -1
curl -s $ADMIN/upstreams/notes-up/health | jq -r '.data[] | [.target, .health] | @tsv'Expected:
- selectors:
- $.upstreams[?(@.name=="notes-up")]
values:
healthchecks:
active:
http_path: /healthz
healthy:
interval: 2
successes: 1
unhealthy:
interval: 2
http_failures: 2
tcp_failures: 2
timeouts: 2
Deleted: 0
notes-api-2:8080 HEALTHY
notes-api:8080 HEALTHY
Now an instance fails:
docker compose -f ~/kong-lab/code/backend/compose.yaml stop notes-api-2
sleep 7
curl -s $ADMIN/upstreams/notes-up/health | jq -r '.data[] | [.target, .health] | @tsv'
zaehle 10Expected:
Container backend-notes-api-2-1 Stopped
notes-api-2:8080 UNHEALTHY
notes-api:8080 HEALTHY
10 0.2.0
Not a single error at the client: Kong sends everything to the healthy instance. And back:
docker compose -f ~/kong-lab/code/backend/compose.yaml --profile lb start notes-api-2
sleep 7
curl -s $ADMIN/upstreams/notes-up/health | jq -r '.data[] | [.target, .health] | @tsv'Expected:
Container backend-notes-api-2-1 Started
notes-api-2:8080 HEALTHY
notes-api:8080 HEALTHY
notes-api’s /healthz also checks the database. If
Postgres fails, both instances report themselves sick, and Kong answers
with 503 (Day 15).
Save your progress? Sign up for free
Drill 4: Canary
Version 0.3.0 should get only ten percent of requests at first:
U='$.upstreams[?(@.name=="notes-up")].targets'
deck file patch -s ~/kong-lab/kong/kong.yaml \
--selector "$U[?(@.target==\"notes-api:8080\")]" \
--value 'weight:90' -o ~/kong-lab/kong/kong.yaml
deck file patch -s ~/kong-lab/kong/kong.yaml \
--selector "$U[?(@.target==\"notes-api-2:8080\")]" \
--value 'weight:10' -o ~/kong-lab/kong/kong.yaml
ksync 2>/dev/null | tail -2
zaehle 100Expected:
Updated: 2
Deleted: 0
90 0.2.0
10 0.3.0
If 0.3.0 runs well, the weights go step by step to 50/50 and 0/100.
If something goes wrong, the way back is the same patch with
weight:0.
For today, go back to equal weights, then save the state:
for t in notes-api notes-api-2; do
deck file patch -s ~/kong-lab/kong/kong.yaml \
--selector "$U[?(@.target==\"$t:8080\")]" \
--value 'weight:100' -o ~/kong-lab/kong/kong.yaml
done
ksync 2>/dev/null | tail -2
cd ~/kong-lab/kong && git add kong.yaml
git commit -qm "Tag 9: Upstream mit Health Checks"Expected:
Updated: 2
Deleted: 0
Save your progress? Sign up for free
Drill 5: The Algorithms
curl -s $ADMIN/upstreams/notes-up | jq -c '{algorithm, hash_on, slots}'
curl -s $ADMIN/schemas/upstreams | jq -r '.fields[] | select(.algorithm)
| .algorithm.one_of | join(", ")'Expected:
{"algorithm":"round-robin","hash_on":"none","slots":10000}
consistent-hashing, least-connections, round-robin, latency
| Algorithm | Distributes by | Typical use |
|---|---|---|
round-robin |
Taking turns, weighted | Default, equally strong instances |
least-connections |
fewest open connections | long requests |
latency |
fastest response time | instances of differing strength |
consistent-hashing |
hash over consumer, IP, header, cookie | caches, sessions |
Save your progress? Sign up for free
Checkpoint
curl -s $ADMIN/services/notes | jq -r .hostExpected: the service points to the upstream.
notes-up
curl -s $ADMIN/upstreams/notes-up/health | jq -r '.data[] | [.target, .health,
.weight] | @tsv'Expected: two healthy targets with equal weight.
notes-api-2:8080 HEALTHY 100
notes-api:8080 HEALTHY 100
for i in $(seq 1 10); do curl -s $PROXY/api/v1/version; done | sort | uniq -cExpected: half and half.
5 0.2.0
5 0.3.0
Challenge
What does a client see when both instances fail? And how long does it take until Kong delivers again once one is back?
Note: Stop both, then start one and query every second.
Solution
cd ~/kong-lab/code/backend
docker compose stop notes-api notes-api-2
sleep 7
curl -s -w ' %{http_code}\n' $PROXY/api/v1/version
docker compose start notes-api
until [ "$(curl -s -o /dev/null -w '%{http_code}' $PROXY/api/v1/version)" = 200 ]; do
echo -n '.'; sleep 1
done; echo ' wieder da'
docker compose --profile lb start notes-api-2Expected (shortened, the number of dots varies):
Container backend-notes-api-2-1 Stopped
Container backend-notes-api-1 Stopped
{
"message":"failure to get a peer from the ring-balancer",
"request_id":"…"
} 503
Container backend-notes-api-1 Started
... wieder da
Container backend-notes-api-2-1 Started
Without a healthy target, Kong answers with 503 itself. After the
start it takes one successful check (successes: 1, every 2
s) plus the instance’s startup time.
Save your progress? Sign up for free
Enterprise Note
- Passive health checks
(
healthchecks.passive) evaluate real responses and cost no extra requests; combined with active checks they detect failures faster. - Health checks per node: Each Kong node checks on its own. With ten nodes and an interval of 2 s, every backend gets five check requests per second.
- Service discovery: Instead of fixed targets, an upstream can be populated via DNS SRV or through the ingress controller (Day 14) with the pods of a Kubernetes service.
- Canary with rules (by header, consumer, percentage
per user) is offered by the Enterprise plugin
canary; OSS sticks with weights.