Skip to content
Hands-on IT IT Drills

Day 9: Upstreams and Load Balancing

Goal: Distribute load across two instances of notes-api, take sick instances out automatically and give a new version to ten percent of requests as a canary. · Duration: approx. 60 min · Prerequisite: Day 8

Briefing

  • An upstream is a virtual hostname with several targets (host:port with a weight). A service points to it with host: notes-up instead of to a container.
  • The default algorithm is round-robin, weighted by weight.
  • Active health checks query each target regularly; passive ones evaluate real responses. A sick target no longer receives requests.
  • A canary is just a matter of weights: 90 to 10.
                          ┌─► notes-api:8080    (0.2.0, weight 90)
 Service notes ─► notes-up┤
   host: notes-up         └─► notes-api-2:8080  (0.3.0, weight 10)
                     Health check: GET /healthz every 2 s

Drill 1: A Second Instance

The second instance gets a new version so that you can see at /version who is answering:

source ~/kong-lab/setup/env.sh
mkdir -p ~/kong-lab/tag09 && cd ~/kong-lab/code/notes/notes-api
docker build -q --build-arg VERSION=0.3.0 -t notes-api:0.3.0 .
cd ~/kong-lab/code/backend
grep -A2 'notes-api-2:' compose.yaml
docker compose --profile lb up -d --wait
docker compose ps --format "table {{.Service}}\t{{.Image}}\t{{.Status}}"

Expected (shortened):

sha256:2b4915ed17c7…
  notes-api-2:          # zweite Instanz, neue Version, für Load Balancing (Tag 9)
    image: notes-api:0.3.0
    environment: *api-env
 Container backend-notes-api-2-1 Healthy
SERVICE       IMAGE                                 STATUS
db            postgres:18.6-alpine                  Up About an hour (healthy)
echo          ghcr.io/mccutchen/go-httpbin:2.26.0   Up About an hour
notes-api     notes-api:0.2.0                       Up About an hour (healthy)
notes-api-2   notes-api:0.3.0                       Up 2 seconds (healthy)

The lb profile additionally starts notes-api-2. Both use the same database.

Save your progress? Sign up for free

Drill 2: One Upstream for Both

cd ~/kong-lab/tag09
cp ~/kong-lab/code/tag09/*.yaml .
grep -v '^#' upstream.yaml
deck file merge ~/kong-lab/kong/kong.yaml upstream.yaml -o ~/kong-lab/kong/kong.yaml
SEL='$.services[?(@.name=="notes" || @.name=="notes-version")]'
deck file patch -s ~/kong-lab/kong/kong.yaml --selector "$SEL" \
  --value 'host:"notes-up"' -o ~/kong-lab/kong/kong.yaml
deck gateway diff ~/kong-lab/kong/kong.yaml 2>/dev/null \
  | grep -E '^(creating|updating)|"host"'
ksync 2>/dev/null | tail -3

Expected (shortened):

_format_version: "3.0"
upstreams:
  - name: notes-up
    algorithm: round-robin
    targets:
      - target: notes-api:8080
        weight: 100
      - target: notes-api-2:8080
        weight: 100
updating service notes  {
-  "host": "notes-api",
+  "host": "notes-up",
updating service notes-version  {
-  "host": "notes-api",
+  "host": "notes-up",
creating upstream notes-up
creating target notes-api:8080 for upstream ef91f37a-…
creating target notes-api-2:8080 for upstream ef91f37a-…
  Created: 3
  Updated: 2
  Deleted: 0

The services change only their host. Kong notices that notes-up is an upstream and does not resolve it via DNS, but through the targets.

Count who answers. /api/v1/version is a good fit because the route has no rate limit:

zaehle() { for i in $(seq 1 ${1:-20}); do curl -s $PROXY/api/v1/version; done \
  | sort | uniq -c; }
zaehle 20
curl -s $ADMIN/upstreams/notes-up/health | jq -r '.data[] | [.target, .health] | @tsv'

Expected:

     10 0.2.0
     10 0.3.0
notes-api-2:8080    HEALTHCHECKS_OFF
notes-api:8080  HEALTHCHECKS_OFF

Strictly alternating. HEALTHCHECKS_OFF means: Kong does not yet check whether the targets are alive.

Save your progress? Sign up for free

Drill 3: Health Checks

health.yaml is a patch file for decK: Kong should query /healthz every two seconds and take a target out after two failures.

grep -v '^#' health.yaml | sed -n '3,16p'
deck file patch -s ~/kong-lab/kong/kong.yaml health.yaml -o ~/kong-lab/kong/kong.yaml
ksync 2>/dev/null | tail -1
curl -s $ADMIN/upstreams/notes-up/health | jq -r '.data[] | [.target, .health] | @tsv'

Expected:

  - selectors:
      - $.upstreams[?(@.name=="notes-up")]
    values:
      healthchecks:
        active:
          http_path: /healthz
          healthy:
            interval: 2
            successes: 1
          unhealthy:
            interval: 2
            http_failures: 2
            tcp_failures: 2
            timeouts: 2
  Deleted: 0
notes-api-2:8080    HEALTHY
notes-api:8080  HEALTHY

Now an instance fails:

docker compose -f ~/kong-lab/code/backend/compose.yaml stop notes-api-2
sleep 7
curl -s $ADMIN/upstreams/notes-up/health | jq -r '.data[] | [.target, .health] | @tsv'
zaehle 10

Expected:

 Container backend-notes-api-2-1 Stopped
notes-api-2:8080    UNHEALTHY
notes-api:8080  HEALTHY
     10 0.2.0

Not a single error at the client: Kong sends everything to the healthy instance. And back:

docker compose -f ~/kong-lab/code/backend/compose.yaml --profile lb start notes-api-2
sleep 7
curl -s $ADMIN/upstreams/notes-up/health | jq -r '.data[] | [.target, .health] | @tsv'

Expected:

 Container backend-notes-api-2-1 Started
notes-api-2:8080    HEALTHY
notes-api:8080  HEALTHY

notes-api’s /healthz also checks the database. If Postgres fails, both instances report themselves sick, and Kong answers with 503 (Day 15).

Save your progress? Sign up for free

Drill 4: Canary

Version 0.3.0 should get only ten percent of requests at first:

U='$.upstreams[?(@.name=="notes-up")].targets'
deck file patch -s ~/kong-lab/kong/kong.yaml \
  --selector "$U[?(@.target==\"notes-api:8080\")]" \
  --value 'weight:90' -o ~/kong-lab/kong/kong.yaml
deck file patch -s ~/kong-lab/kong/kong.yaml \
  --selector "$U[?(@.target==\"notes-api-2:8080\")]" \
  --value 'weight:10' -o ~/kong-lab/kong/kong.yaml
ksync 2>/dev/null | tail -2
zaehle 100

Expected:

  Updated: 2
  Deleted: 0
     90 0.2.0
     10 0.3.0

If 0.3.0 runs well, the weights go step by step to 50/50 and 0/100. If something goes wrong, the way back is the same patch with weight:0.

For today, go back to equal weights, then save the state:

for t in notes-api notes-api-2; do
  deck file patch -s ~/kong-lab/kong/kong.yaml \
    --selector "$U[?(@.target==\"$t:8080\")]" \
    --value 'weight:100' -o ~/kong-lab/kong/kong.yaml
done
ksync 2>/dev/null | tail -2
cd ~/kong-lab/kong && git add kong.yaml
git commit -qm "Tag 9: Upstream mit Health Checks"

Expected:

  Updated: 2
  Deleted: 0

Save your progress? Sign up for free

Drill 5: The Algorithms

curl -s $ADMIN/upstreams/notes-up | jq -c '{algorithm, hash_on, slots}'
curl -s $ADMIN/schemas/upstreams | jq -r '.fields[] | select(.algorithm)
  | .algorithm.one_of | join(", ")'

Expected:

{"algorithm":"round-robin","hash_on":"none","slots":10000}
consistent-hashing, least-connections, round-robin, latency
Algorithm Distributes by Typical use
round-robin Taking turns, weighted Default, equally strong instances
least-connections fewest open connections long requests
latency fastest response time instances of differing strength
consistent-hashing hash over consumer, IP, header, cookie caches, sessions

Save your progress? Sign up for free

Checkpoint

curl -s $ADMIN/services/notes | jq -r .host

Expected: the service points to the upstream.

notes-up
curl -s $ADMIN/upstreams/notes-up/health | jq -r '.data[] | [.target, .health,
  .weight] | @tsv'

Expected: two healthy targets with equal weight.

notes-api-2:8080    HEALTHY 100
notes-api:8080  HEALTHY 100
for i in $(seq 1 10); do curl -s $PROXY/api/v1/version; done | sort | uniq -c

Expected: half and half.

      5 0.2.0
      5 0.3.0

Challenge

What does a client see when both instances fail? And how long does it take until Kong delivers again once one is back?

Note: Stop both, then start one and query every second.

Solution

cd ~/kong-lab/code/backend
docker compose stop notes-api notes-api-2
sleep 7
curl -s -w ' %{http_code}\n' $PROXY/api/v1/version
docker compose start notes-api
until [ "$(curl -s -o /dev/null -w '%{http_code}' $PROXY/api/v1/version)" = 200 ]; do
  echo -n '.'; sleep 1
done; echo ' wieder da'
docker compose --profile lb start notes-api-2

Expected (shortened, the number of dots varies):

 Container backend-notes-api-2-1 Stopped
 Container backend-notes-api-1 Stopped
{
  "message":"failure to get a peer from the ring-balancer",
  "request_id":"…"
} 503
 Container backend-notes-api-1 Started
... wieder da
 Container backend-notes-api-2-1 Started

Without a healthy target, Kong answers with 503 itself. After the start it takes one successful check (successes: 1, every 2 s) plus the instance’s startup time.

Save your progress? Sign up for free

Enterprise Note

  • Passive health checks (healthchecks.passive) evaluate real responses and cost no extra requests; combined with active checks they detect failures faster.
  • Health checks per node: Each Kong node checks on its own. With ten nodes and an interval of 2 s, every backend gets five check requests per second.
  • Service discovery: Instead of fixed targets, an upstream can be populated via DNS SRV or through the ingress controller (Day 14) with the pods of a Kubernetes service.
  • Canary with rules (by header, consumer, percentage per user) is offered by the Enterprise plugin canary; OSS sticks with weights.