Skip to content
Hands-on IT IT Drills

Day 10: TLS

Goal: Equip Kong with a certificate from your own CA, enforce HTTPS for writing and the Admin API, detect old TLS versions and talk to the backend over TLS with a verified certificate. · Duration: approx. 75 min · Prerequisite: Day 9

Briefing

  • Kong terminates TLS on proxy port 8443. Which certificate it presents is decided by the hostname the client names in the handshake (SNI).
  • Without a matching certificate, Kong presents a self-generated default certificate for localhost.
  • protocols: [https] on a route forbids plain text: HTTP gets 426.
  • Toward the backend, Kong speaks HTTP or HTTPS. With HTTPS, it verifies the certificate only with tls_verify: true and a CA it trusts.
 curl ──TLS (SNI api.kong.lab)──► Kong :8443 ──TLS, checked──► echo-tls:8080
          Certificate api.kong.lab              CA: Kong Lab CA
          signed by Kong Lab CA

Drill 1: Your Own CA

source ~/kong-lab/setup/env.sh
mkdir -p ~/kong-lab/tag10/tls && cd ~/kong-lab/tag10/tls
openssl req -x509 -newkey rsa:2048 -nodes -keyout ca.key -out ca.crt -days 365 \
  -subj "/CN=Kong Lab CA" 2>/dev/null
openssl req -newkey rsa:2048 -nodes -keyout api.key -out api.csr \
  -subj "/CN=api.kong.lab" 2>/dev/null
printf 'subjectAltName=DNS:api.kong.lab,DNS:localhost\nextendedKeyUsage=serverAuth\n' \
  > api.ext
openssl x509 -req -in api.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out api.crt -days 90 -extfile api.ext
openssl x509 -in api.crt -noout -subject -issuer -ext subjectAltName

Expected:

Certificate request self-signature ok
subject=CN = api.kong.lab
subject=CN = api.kong.lab
issuer=CN = Kong Lab CA
X509v3 Subject Alternative Name:
    DNS:api.kong.lab, DNS:localhost

One CA, one server certificate for two names, valid for 90 days. Browsers and curl look at subjectAltName, not at CN.

Save your progress? Sign up for free

Drill 2: The Default Certificate

curl -s -o /dev/null -w '%{http_code}\n' https://localhost:8443/api/v1/version
echo "rc=$?"
curl -sk -o /dev/null -w '%{http_code}\n' https://localhost:8443/api/v1/version
echo | openssl s_client -connect localhost:8443 -servername api.kong.lab 2>/dev/null \
  | openssl x509 -noout -issuer

Expected:

000
rc=60
200
issuer=C = US, ST = California, L = San Francisco, O = Kong, OU = IT Department, CN = …

Error 60: curl does not trust the certificate. With -k (verification turned off) it works, but Kong generated the certificate itself at first start. For clients it is worthless.

Save your progress? Sign up for free

Drill 3: Certificate and SNI

Certificate and key go into your kong.yaml. decK requires a fixed id for certificates so that the SNIs can refer to them:

cat > tls.yaml <<EOF
certificates:
  - id: $(cat /proc/sys/kernel/random/uuid)
    cert: |
$(sed 's/^/      /' api.crt)
    key: |
$(sed 's/^/      /' api.key)
    snis:
      - name: api.kong.lab
      - name: localhost
EOF
head -3 tls.yaml
deck file merge ~/kong-lab/kong/kong.yaml tls.yaml -o ~/kong-lab/kong/kong.yaml
ksync 2>/dev/null | tail -3
curl -s --cacert ca.crt --resolve api.kong.lab:8443:127.0.0.1 \
  https://api.kong.lab:8443/api/v1/version
curl -s --cacert ca.crt https://localhost:8443/api/v1/version
curl -s $ADMIN/snis | jq -r '.data[].name'

Expected (shortened, your ID will differ):

certificates:
  - id: 06e00ffb-721a-4a0e-b96b-17b4c19b32bf
    cert: |
  Created: 3
  Updated: 0
  Deleted: 0
0.2.0
0.3.0
api.kong.lab
localhost

--resolve makes curl resolve the name api.kong.lab to 127.0.0.1 without changing /etc/hosts, and sends it as SNI. --cacert ca.crt instead of -k: now curl really verifies.

Security: Your kong.yaml now contains a private key. In the lab, the local Git repository stays on the VM. In production, the key belongs in a vault, and Kong reads it by reference ({vault://…}).

Save your progress? Sign up for free

Drill 4: Enforce HTTPS

Passwords (admin-api) and keys (notes-schreiben) should never travel in plain text:

SEL='$..routes[?(@.name=="notes-schreiben" || @.name=="admin-api")]'
deck file patch -s ~/kong-lab/kong/kong.yaml --selector "$SEL" \
  --value 'protocols:["https"]' -o ~/kong-lab/kong/kong.yaml
ksync 2>/dev/null | tail -1
H='Content-Type: application/json'
curl -s -w ' %{http_code}\n' -X POST -H 'apikey: lab-key-anna-456' -H "$H" \
  -d '{"text":"http"}' $PROXY/api/v1/notes | tr -d '\n'; echo
curl -s -w ' %{http_code}\n' --cacert ca.crt -X POST -H 'apikey: lab-key-anna-456' \
  -H "$H" -d '{"text":"per TLS"}' https://localhost:8443/api/v1/notes
curl -s -o /dev/null -w '%{http_code}\n' --cacert ca.crt -u anna:lab-user-pw-123 \
  https://localhost:8443/admin-api/status

Expected (shortened):

  Deleted: 0
{  "message":"Please use HTTPS protocol",  "request_id":"4ec2065a…"} 426
{"id":"c78656b2-…","text":"per TLS"}
 201
200

426 means “Upgrade Required”. If you want to redirect instead, set https_redirect_status_code: 301 on the route. Kong then sends a Location header to https://… without a port, because it only knows the port it listens on inside the container; in production, Kong sits on 443.

Save your progress? Sign up for free

Drill 5: Which TLS Versions

curl -s $ADMIN | jq -c '.configuration | {ssl_protocols, ssl_cipher_suite}'
for v in tls1_2 tls1_3; do
  echo | openssl s_client -$v -connect localhost:8443 -servername localhost 2>&1 \
    | grep '^New, '
done
echo | openssl s_client -tls1_1 -cipher 'DEFAULT@SECLEVEL=0' -connect localhost:8443 \
  2>&1 | grep -o 'tlsv1 alert protocol version'

Expected:

{"ssl_protocols":"TLSv1.2 TLSv1.3","ssl_cipher_suite":"intermediate"}
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
tlsv1 alert protocol version

intermediate follows Mozilla’s recommendation: TLS 1.2 and 1.3 with modern ciphers. For TLS 1.1 you already have to coax your own client (SECLEVEL=0), and then Kong refuses.

Save the state:

cd ~/kong-lab/kong
git add kong.yaml
git commit -qm "Tag 10: Zertifikat, HTTPS für Schreiben und Admin API"

Save your progress? Sign up for free

Drill 6: TLS to the Backend

Even behind the gateway, a network can be insecure. An echo service with its own certificate, signed by your CA:

cd ~/kong-lab/tag10/tls
openssl req -newkey rsa:2048 -nodes -keyout echo-tls.key -out echo-tls.csr \
  -subj "/CN=echo-tls" 2>/dev/null
printf 'subjectAltName=DNS:echo-tls\n' > echo.ext
openssl x509 -req -in echo-tls.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out echo-tls.crt -days 90 -extfile echo.ext 2>/dev/null
chmod 644 echo-tls.key
docker run -d --name echo-tls --network lab -v $PWD:/tls:ro \
  ghcr.io/mccutchen/go-httpbin:2.26.0 \
  -https-cert-file /tls/echo-tls.crt -https-key-file /tls/echo-tls.key > /dev/null
sleep 2
docker logs echo-tls 2>&1 | tail -1 | grep -o 'listening on .*'

Expected:

listening on https://0.0.0.0:8080"

Kong should verify the certificate. For that, it gets the CA as a ca_certificate, and the service refers to it:

CAID=$(cat /proc/sys/kernel/random/uuid)
cat > upstream-tls.yaml <<EOF
ca_certificates:
  - id: $CAID
    cert: |
$(sed 's/^/      /' ca.crt)
services:
  - name: echo-tls
    url: https://echo-tls:8080/anything
    tls_verify: true
    ca_certificates:
      - $CAID
    routes:
      - name: echo-tls
        paths: [/echo-tls]
EOF
deck file merge ~/kong-lab/kong/kong.yaml upstream-tls.yaml -o ~/kong-lab/kong/kong.yaml
ksync 2>/dev/null | tail -3
curl -s -o /dev/null -w '%{http_code}\n' $PROXY/echo-tls/get

Expected:

  Created: 3
  Updated: 0
  Deleted: 0
200

And without the CA? Apply the same file with an empty ca_certificates list:

deck file patch -s ~/kong-lab/kong/kong.yaml \
  --selector '$.services[?(@.name=="echo-tls")]' \
  --value 'ca_certificates:[]' -o ohne-ca.yaml
deck gateway sync ohne-ca.yaml 2>/dev/null | tail -1; sleep 5
curl -s -w ' %{http_code}\n' $PROXY/echo-tls/get | tr -d '\n'; echo
docker compose -f ~/kong-lab/kong/compose.yaml logs kong 2>&1 \
  | grep -o 'upstream SSL certificate verify error: ([^)]*)' | tail -1

Expected (shortened):

  Deleted: 0
{  "message":"An invalid response was received from the upstream server",  …} 502
upstream SSL certificate verify error: (20:unable to get local issuer certificate)

Kong does not trust the certificate and aborts; the client gets 502. Without tls_verify: true, Kong would have accepted any certificate, even an attacker’s.

The echo service with TLS was an experiment. Back to the state of Drill 5:

docker rm -f echo-tls
cd ~/kong-lab/kong && git checkout kong.yaml && ksync 2>/dev/null | tail -3

Expected:

echo-tls
Updated 1 path from the index
  Created: 0
  Updated: 0
  Deleted: 3

Save your progress? Sign up for free

Checkpoint

curl -s --cacert ~/kong-lab/tag10/tls/ca.crt https://localhost:8443/api/v1/version

Expected: Kong presents your certificate and curl verifies it successfully (0.2.0 or 0.3.0, depending on the target).

0.2.0
curl -s -o /dev/null -w '%{http_code}\n' -u anna:lab-user-pw-123 $PROXY/admin-api/status

Expected: the Admin API is blocked over HTTP.

426
curl -s $ADMIN/routes/notes-schreiben | jq -c .protocols

Expected: writing only over HTTPS.

["https"]

Challenge

Your certificate expires in 90 days. Find out with a single command exactly when, the way a client sees it, not from the file.

Note: openssl s_client fetches the certificate, openssl x509 -enddate shows the end.

Solution

echo | openssl s_client -connect localhost:8443 -servername api.kong.lab 2>/dev/null \
  | openssl x509 -noout -enddate -issuer

Expected (shortened, your date will differ):

notAfter=Jan 30 12:55:18 2027 GMT
issuer=CN = Kong Lab CA

This is exactly how monitoring checks the validity from the outside. An alert 30 days before notAfter prevents the most common TLS incident of all: the forgotten certificate.

Save your progress? Sign up for free

Enterprise Note

  • ACME: The acme plugin (in OSS too) fetches certificates automatically from Let’s Encrypt and renews them. Prerequisite: Kong is publicly reachable under that name.
  • Keys in a vault: Certificate fields in Kong can be referenced ({vault://env/…}); Enterprise connects HashiCorp Vault, OpenBao and cloud storage.
  • mTLS to clients: No OSS plugin verifies client certificates; Enterprise has mtls-auth. Toward the backend, Kong can present a client certificate itself (client_certificate on the service).
  • TLS in front of Kong: Often a load balancer terminates TLS. Even then, TLS between the load balancer and Kong is still needed, otherwise keys travel in plain text on the internal network.