Skip to content
Hands-on IT IT Drills

Day 14: Code LLM – Autocomplete and Chat in VS Code

Goal: A code model on rke2-gpu-1 delivers autocomplete and chat in VS Code via the LiteLLM alias code – measured, with its own virtual key, and without code leaving the network. · Duration: approx. 150 min · Prerequisite: Day 13

Briefing

  • FIM (fill-in-the-middle): The editor sends the code before and after the cursor to /v1/completions in a raw prompt with special tokens. Goal: < 500 ms until the suggestion; otherwise the human types faster.
  • Chat/edit: normal chat requests to /v1/chat/completions; seconds are fine.
  • vLLM doesn’t support the OpenAI parameter suffix. So the client (Continue, Tabby) builds the FIM prompt – with the model’s tokens.
  • Model: Qwen/Qwen2.5-Coder-7B-Instruct-AWQ – Apache-2.0, approx. 5.6 GB, can do FIM and chat.
VS Code (Continue) --HTTPS--> llm.lab.internal (Traefik) --> LiteLLM "code"
  FIM:  POST /v1/completions        <|fim_prefix|>…<|fim_suffix|>…<|fim_middle|>
  Chat: POST /v1/chat/completions             --> vllm-code.ki-models:8000

Drill 1: Choose a model – license and VRAM

Model License (as of 10/2026) Weights Fits on an L4 24 GB?
Qwen2.5-Coder-7B-Instruct-AWQ Apache-2.0 approx. 5.6 GB yes, alongside chat and embed
Qwen2.5-Coder-1.5B (base) Apache-2.0 approx. 3 GB (BF16) yes, pure FIM, very fast
Qwen2.5-Coder-3B Qwen Research License approx. 6 GB non-commercial only
Codestral 22B Mistral Non-Production License approx. 44 GB (BF16) no; production only with a paid license
Devstral Small (24B) Apache-2.0 approx. 47 GB (BF16) only quantized and on its own; for agents rather than FIM

Warning: Licenses change. Check the model card on Hugging Face before use – “open weights” doesn’t automatically mean “commercially usable”.

Recalculate the GPU budget (Qwen2.5-7B: 28 layers, 4 KV heads, head dim 128):

python3 -c '
gpu=24*0.33; w=5.6; ovh=1.0
kv=2*28*4*128*2
tok=(gpu-w-ovh)*1024**3/kv
print(f"KV pro Token: {kv/1024:.0f} KiB, KV-Budget: {gpu-w-ovh:.1f} GB, "
      f"Tokens: {tok:,.0f}")
print(f"parallel bei 8192: {tok/8192:.1f}, bei 1500 (Autocomplete): {tok/1500:.0f}")'

Expected:

KV pro Token: 56 KiB, KV-Budget: 1.3 GB, Tokens: 24,716
parallel bei 8192: 3.0, bei 1500 (Autocomplete): 16

Why? With 33% of the L4, the KV cache is enough for about 16 simultaneous autocomplete requests – enough for a small team. More developers need a dedicated GPU for code.

Save your progress? Sign up for free

Drill 2: Redistribute the GPU and start vllm-code

Chat 0.80 + embed 0.12 (Day 13) leave no room. New budget: chat 0.50, embed 0.12, code 0.33.

mkdir -p ~/ki-lab/tag14 && cd ~/ki-lab/tag14
export KUBECONFIG=~/ki-lab/rke2.yaml
kubectl -n ki-models get deploy vllm-chat \
  -o jsonpath='{.spec.template.spec.containers[0].args[3]}{"\n"}'
kubectl -n ki-models patch deploy vllm-chat --type=json -p '[
  {"op": "replace", "path": "/spec/template/spec/containers/0/args/2",
   "value": "--max-model-len=8192"},
  {"op": "replace", "path": "/spec/template/spec/containers/0/args/3",
   "value": "--gpu-memory-utilization=0.50"},
  {"op": "add", "path": "/spec/template/spec/containers/0/args/-",
   "value": "--kv-cache-dtype=fp8"}]'
kubectl -n ki-models rollout status deploy/vllm-chat --timeout=15m
kubectl apply -f ~/ki-lab/code/code-llm/vllm-code.yaml
kubectl -n ki-models rollout status deploy/vllm-code --timeout=15m
kubectl -n ki-models get pods -l 'app in (vllm-chat,vllm-embed,vllm-code)'

Expected (abbreviated, may differ depending on version):

--gpu-memory-utilization=0.80
deployment.apps/vllm-chat patched
deployment "vllm-chat" successfully rolled out
persistentvolumeclaim/vllm-code-cache created
deployment.apps/vllm-code created
service/vllm-code created
deployment "vllm-code" successfully rolled out
NAME                          READY   STATUS    RESTARTS   AGE
vllm-chat-7f9c8d6b4-qx2lm     1/1     Running   0          6m
vllm-code-5b7d9f8c6-h4k2p     1/1     Running   0          4m
vllm-embed-6c8f7d5b9-m3n8r    1/1     Running   0          1d

Why? Order matters: vllm-chat has to release memory first, then vllm-code starts. The other way around, vllm-code aborts with “not enough free memory”. With only 50% of the L4 left, a 16k sequence (about 2.3 GiB of KV cache) no longer fits into the cache – vLLM would abort with “max seq len”. That’s why the patch halves --max-model-len to 8192 and stores the KV cache in FP8. Also enter all three changes in code/vllm/vllm-chat.yaml; otherwise the next kubectl apply resets the old value.

Save your progress? Sign up for free

Drill 3: FIM directly against the model

kubectl -n ki-models port-forward svc/vllm-code 8002:8000 >/dev/null 2>&1 &
sleep 2
jq -n --arg p $'def is_even(n: int) -> bool:\n    return ' --arg s $'\n' '{
  model: "qwen2.5-coder-7b", max_tokens: 16, temperature: 0,
  prompt: ("<|fim_prefix|>" + $p + "<|fim_suffix|>" + $s + "<|fim_middle|>")}' \
  > fim.json
curl -s localhost:8002/v1/completions -H 'Content-Type: application/json' \
  -d @fim.json | jq '{text: .choices[0].text, usage}'
jq '.suffix = "\n" | .prompt = "def is_even(n: int) -> bool:\n    return "' fim.json \
  | curl -s localhost:8002/v1/completions -H 'Content-Type: application/json' \
  -d @- | jq -c '.error // .choices[0].text'

Expected (abbreviated, may differ depending on version):

{
  "text": "n % 2 == 0",
  "usage": {
    "prompt_tokens": 18,
    "total_tokens": 25,
    "completion_tokens": 7
  }
}
{"message":"suffix is not currently supported", ...}

Why? The first request is real FIM: the model fills exactly the gap and stops. The second uses the OpenAI parameter suffix – vLLM rejects it. That’s why every client needs a prompt template with the model’s FIM tokens.

Save your progress? Sign up for free

Drill 4: Alias code and team key

Add the entries from code/code-llm/litellm-models-code.yaml to ~/ki-lab/code/litellm-prod/config.yaml (code; you already have embed from Day 13) and turn off storing prompts:

  - model_name: code
    litellm_params:
      model: hosted_vllm/qwen2.5-coder-7b     # --served-model-name
      api_base: http://vllm-code.ki-models:8000/v1
      timeout: 30
    model_info:
      mode: chat
general_settings:
  store_prompts_in_spend_logs: false   # default, explicit here for the audit
litellm_settings:
  turn_off_message_logging: true       # no prompts to logging callbacks
cd ~/ki-lab/code/litellm-prod
kubectl -n ki-gateway create configmap litellm-config --from-file=config.yaml \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl -n ki-gateway rollout restart deploy/litellm
kubectl -n ki-gateway rollout status deploy/litellm
export LITELLM_MASTER_KEY=sk-lab-master-1234          # lab value
H=(-H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json')
API=https://llm.lab.internal
TEAM=$(curl -s $API/team/new "${H[@]}" -d '{"team_alias": "team-code",
  "models": ["code", "embed"], "max_budget": 30, "budget_duration": "30d",
  "rpm_limit": 600}' | jq -r .team_id)
CODE_KEY=$(curl -s $API/key/generate "${H[@]}" -d "{\"team_id\": \"$TEAM\",
  \"key_alias\": \"thomas-vscode\", \"models\": [\"code\", \"embed\"]}" \
  | jq -r .key)
echo "${CODE_KEY:0:6}..."
curl -s $API/v1/completions -H "Authorization: Bearer $CODE_KEY" \
  -H 'Content-Type: application/json' -d @<(jq '.model = "code"' fim.json) \
  | jq -r '.model, .choices[0].text'

Expected (abbreviated, may differ depending on version):

configmap/litellm-config configured
deployment "litellm" successfully rolled out
sk-Xa1...
code
n % 2 == 0

Why? Autocomplete quickly generates 20–60 requests per minute per developer. The team limit rpm_limit: 600 protects the GPU from a broken plugin. For code there is deliberately no fallback to chat-large-eu: source code never leaves the network.

Save your progress? Sign up for free

Drill 5: Measure latency with code_eval.py

code/code-llm/code_eval.py sends three FIM tasks (Python, Python, YAML) and two chat questions via LiteLLM, measures time to first token (TTFT) and total time, and checks simple expectations. First the cross-check on the laptop against LiteLLM from Day 4, where code points to a chat model:

source ~/ki-lab/.venv/bin/activate && pip install -q openai
LITELLM_URL=http://127.0.0.1:4000/v1 LITELLM_KEY=sk-lab-master-1234 \
  RUNS=3 FIM_P95_MS=3000 python ~/ki-lab/code/code-llm/code_eval.py

Expected (laptop, Ollama with ministral-3:3b, values fluctuate):

Modell: code  FIM-Stil: qwen  Läufe: 3
Art     TTFT p50  TTFT p95  Total p50  Total p95
FIM        579ms     708ms     1836ms     2078ms
Chat       566ms     628ms     2777ms     4610ms
Treffer: 4/11
✔ FIM p95 2078ms (Ziel ≤ 3000ms)

Only 4 of 11 hits: a chat model doesn’t know the FIM tokens and starts chatting (“Here’s the completed function…”). With DEBUG=1 you see the raw answers. Now against the real code model:

export SSL_CERT_FILE=~/ki-lab/lab-ca.crt
LITELLM_URL=$API/v1 LITELLM_KEY=$CODE_KEY RUNS=10 \
  python ~/ki-lab/code/code-llm/code_eval.py

Expected (L4, abbreviated, may differ depending on version and load):

Modell: code  FIM-Stil: qwen  Läufe: 10
Art     TTFT p50  TTFT p95  Total p50  Total p95
FIM         45ms      90ms      160ms      310ms
Chat        60ms     120ms     3900ms     5200ms
Treffer: 30/32
✔ FIM p95 310ms (Ziel ≤ 500ms)
Metric Autocomplete target Adjusting knob
TTFT p95 < 150 ms prefix caching, short prompt (maxPromptTokens)
Total p95 < 500 ms max_tokens 32–64, smaller model (1.5B)
Error rate < 1% dedicated GPU slots, rpm_limit per team

Why? The script measures time from sending the request. vLLM sends the HTTP headers only with the first token – if you measure from the response, you see a TTFT close to 0 ms.

Save your progress? Sign up for free

Drill 6: Continue in VS Code

Continue reads ~/.continue/config.yaml (format schema: v1; config.json is deprecated) and secrets from ~/.continue/.env.

mkdir -p ~/.continue
[ -f ~/.continue/config.yaml ] && cp ~/.continue/config.yaml ~/.continue/config.yaml.bak
sed "s|__HOME__|$HOME|g" ~/ki-lab/code/code-llm/continue-config.yaml \
  > ~/.continue/config.yaml
printf 'LITELLM_CODE_KEY=%s\n' "$CODE_KEY" > ~/.continue/.env
chmod 600 ~/.continue/.env
grep -nE 'model:|roles:|- (chat|autocomplete|embed)$|caBundlePath' \
  ~/.continue/config.yaml | head -8

Expected (home path will differ):

10:    model: code
14:      caBundlePath: /Users/thomas/ki-lab/lab-ca.crt
15:    roles:
16:      - chat
24:    model: code
28:      caBundlePath: /Users/thomas/ki-lab/lab-ca.crt
30:    roles:
31:      - autocomplete

The important parts of the autocomplete model:

Key Value Why
useLegacyCompletionsEndpoint true FIM goes to /v1/completions, not to chat
promptTemplates.autocomplete Qwen FIM tokens Continue recognizes the model by its name – code reveals nothing
requestOptions.caBundlePath ~/ki-lab/lab-ca.crt lab CA for llm.lab.internal
autocompleteOptions.debounceDelay 250 ms request only after a typing pause

Test in VS Code: install the “Continue” extension, create a new file test.py, type def fib(n): and wait briefly – a gray suggestion appears; Tab accepts it. In the chat panel (Cmd+L), choose the model “Lab Code (Chat)”. At the same time, in the cluster:

kubectl -n ki-gateway logs deploy/litellm --since=2m | grep -c 'POST /v1/completions'

Expected: a number > 0 (one request per typing pause).

Tip: If autocomplete stays silent, open Output → Continue in VS Code. “fetch failed” almost always means: wrong CA path, or llm.lab.internal doesn’t resolve to 10.10.20.100.

Save your progress? Sign up for free

Drill 7: Tabby as an alternative

Tabby is a separate server with a web UI, user management and plugins for VS Code and JetBrains. Here it doesn’t use its own GPU but the same LiteLLM aliases.

cd ~/ki-lab/tag14
kubectl create namespace ki-code --dry-run=client -o yaml | kubectl apply -f -
TABBY_KEY=$(curl -s $API/key/generate "${H[@]}" -d "{\"team_id\": \"$TEAM\",
  \"key_alias\": \"tabby\", \"models\": [\"code\", \"embed\"]}" | jq -r .key)
sed "s|__LITELLM_TABBY_KEY__|$TABBY_KEY|" \
  ~/ki-lab/code/code-llm/tabby-config.toml > config.toml
kubectl -n ki-code create secret generic tabby-config --from-file=config.toml
rm config.toml
kubectl apply -f ~/ki-lab/code/code-llm/tabby.yaml
kubectl -n ki-code rollout status deploy/tabby --timeout=5m
curl -s https://code.lab.internal/v1/health | jq '{model, chat_model, version}'

Expected (abbreviated, may differ depending on version):

namespace/ki-code created
secret/tabby-config created
persistentvolumeclaim/tabby-data created
deployment.apps/tabby created
service/tabby created
ingress.networking.k8s.io/tabby created
deployment "tabby" successfully rolled out
{
  "model": "code",
  "chat_model": "code",
  "version": { "git_describe": "v0.32.0", ... }
}

Then open https://code.lab.internal, create an admin account, and enter the server URL and your personal token in the VS Code plugin “Tabby”.

Criterion Continue Tabby
Operation client configuration only server with DB, users, tokens
Configuration config.yaml per developer central config.toml
Repo context local in the editor server indexes Git repos
Reporting via LiteLLM keys its own usage statistics plus LiteLLM

Why? For a small team, Continue is enough: nothing to operate, governance via virtual keys. Tabby pays off if you want to control centrally who gets which repos as context.

Save your progress? Sign up for free

Drill 8: Governance – no code leaves the network

kubectl apply -f ~/ki-lab/code/code-llm/netpol.yaml
kubectl -n default run np-test --rm -i --restart=Never \
  --image=curlimages/curl:8.16.0 -- curl -s -m 5 -o /dev/null \
  -w '%{http_code}\n' http://vllm-code.ki-models:8000/health || echo "blockiert"
kubectl -n ki-gateway exec deploy/litellm -- python -c "import urllib.request as u
print(u.urlopen('http://vllm-code.ki-models:8000/health').status)"
grep -A5 'model_name: code' ~/ki-lab/code/litellm-prod/config.yaml \
  | grep -c mistral

Expected (abbreviated, may differ depending on version):

networkpolicy.networking.k8s.io/vllm-code-ingress created
networkpolicy.networking.k8s.io/tabby-egress created
000
blockiert
200
0

Why? Cilium enforces Kubernetes NetworkPolicies: only LiteLLM (and Prometheus) reach vllm-code. So every code request goes through LiteLLM – with key, budget and audit. grep -c mistral = 0 proves: there’s no external fallback for code.

The logging policy for the team, short and in writing:

  • LiteLLM stores metadata (key, model, tokens, duration), no prompts: store_prompts_in_spend_logs: false, turn_off_message_logging: true.
  • vLLM doesn’t log prompts as long as nobody sets --enable-log-requests (plus DEBUG level) or --enable-log-outputs.
  • Continue: turn off telemetry in the extension settings. Tabby: TABBY_DISABLE_USAGE_COLLECTION=1 (it’s in tabby.yaml).
  • Secrets in code: .continueignore or onlyMyCode limit the context, but don’t replace a secret scanner in CI.

Save your progress? Sign up for free

Drill 9: Outlook – agentic coding with guardrails

Agents like Cline (VS Code extension) or OpenHands (its own server with sandbox containers) read files, run commands and call the model in loops – 50–200 requests per task are normal. Both speak the OpenAI API, so https://llm.lab.internal/v1 with a dedicated key.

AGENT_KEY=$(curl -s $API/key/generate "${H[@]}" -d "{\"team_id\": \"$TEAM\",
  \"key_alias\": \"agent-thomas\", \"models\": [\"code\"], \"max_budget\": 2,
  \"budget_duration\": \"1d\", \"rpm_limit\": 30, \"max_parallel_requests\": 2,
  \"metadata\": {\"use\": \"agentic\"}}" | jq -r .key)
curl -s "$API/key/info" -H "Authorization: Bearer $AGENT_KEY" \
  | jq -c '.info | {key_alias, max_budget, rpm_limit, max_parallel_requests}'

Expected (may differ depending on version):

{"key_alias":"agent-thomas","max_budget":2,"rpm_limit":30,"max_parallel_requests":2}
Guardrail Implementation
Cost and load dedicated key: daily budget, rpm_limit, max_parallel_requests
Execution agent only in a devcontainer or OpenHands sandbox, never on the host with a production kubeconfig
Approvals Cline: “Auto-approve” off for commands; confirm every shell action
Write access own Git branch, merge only via review; no push rights on main
Model 7B is enough for autocomplete; for agents rather 24B+ (Devstral) on a dedicated GPU

Why? An agent with shell access is a user with your permissions who types fast. The guardrails belong in infrastructure (key, sandbox, branch protection), not in the prompt.

Save your progress? Sign up for free

Checkpoint

#!/usr/bin/env bash
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
kubectl -n ki-models rollout status deploy/vllm-code --timeout=10s >/dev/null 2>&1 \
  && ok "vllm-code läuft" || nok "vllm-code läuft"
curl -s $API/v1/completions -H "Authorization: Bearer $CODE_KEY" \
  -H 'Content-Type: application/json' -d @<(jq '.model = "code"' fim.json) \
  | jq -e '.choices[0].text | test("%")' >/dev/null \
  && ok "FIM über LiteLLM" || nok "FIM über LiteLLM"
LITELLM_URL=$API/v1 LITELLM_KEY=$CODE_KEY RUNS=5 \
  python ~/ki-lab/code/code-llm/code_eval.py >/dev/null \
  && ok "FIM p95 ≤ 500 ms" || nok "FIM p95 ≤ 500 ms"
kubectl -n ki-models get networkpolicy vllm-code-ingress >/dev/null 2>&1 \
  && ok "NetworkPolicy vllm-code" || nok "NetworkPolicy vllm-code"
curl -sf https://code.lab.internal/v1/health >/dev/null \
  && ok "Tabby erreichbar" || nok "Tabby erreichbar"
grep -q 'LITELLM_CODE_KEY=sk-' ~/.continue/.env \
  && ok "Continue-Key gesetzt" || nok "Continue-Key gesetzt"

Challenge

A working student should get autocomplete only, no chat: a key that may call only /v1/completions, 30 requests per minute, valid for 30 days.

Hint: Besides models, /key/generate also knows allowed_routes and duration.

Solution

WS_KEY=$(curl -s $API/key/generate "${H[@]}" -d "{\"team_id\": \"$TEAM\",
  \"key_alias\": \"werkstudentin\", \"models\": [\"code\"], \"rpm_limit\": 30,
  \"duration\": \"30d\", \"allowed_routes\": [\"/v1/completions\",
  \"/completions\"]}" | jq -r .key)
for ep in completions chat/completions; do
  curl -s -o /dev/null -w "$ep: %{http_code}\n" $API/v1/$ep \
    -H "Authorization: Bearer $WS_KEY" -H 'Content-Type: application/json' \
    -d '{"model": "code", "prompt": "x", "max_tokens": 1,
         "messages": [{"role": "user", "content": "x"}]}'
done

Expected (checked against the docs, may differ depending on version):

completions: 200
chat/completions: 401

In Continue, this key gets only the model with roles: [autocomplete].

Save your progress? Sign up for free

Enterprise Note

  • From about 15 active developers on, code gets a dedicated GPU; you separate autocomplete and agents via two aliases and two vLLM Deployments.
  • Distribute the Continue configuration centrally (MDM, dotfiles repo); keys per person from self-service (LiteLLM UI with SSO), never shared team keys.
  • License review as an approval process: store model, version, license text and hash of the weights in the model registry.
  • Works council and data protection: metadata per person (who uses how much) is performance data – report only aggregated per team.
  • Agentic coding only in sandboxes without access to production secrets; OpenHands runtime containers with their own NetworkPolicy.