Day 14: Code LLM – Autocomplete and Chat in VS Code
Goal: A code model on
rke2-gpu-1delivers autocomplete and chat in VS Code via the LiteLLM aliascode– measured, with its own virtual key, and without code leaving the network. · Duration: approx. 150 min · Prerequisite: Day 13
Briefing
- FIM (fill-in-the-middle): The editor sends the code
before and after the cursor to
/v1/completionsin a raw prompt with special tokens. Goal: < 500 ms until the suggestion; otherwise the human types faster. - Chat/edit: normal chat requests to
/v1/chat/completions; seconds are fine. - vLLM doesn’t support the OpenAI parameter
suffix. So the client (Continue, Tabby) builds the FIM prompt – with the model’s tokens. - Model:
Qwen/Qwen2.5-Coder-7B-Instruct-AWQ– Apache-2.0, approx. 5.6 GB, can do FIM and chat.
VS Code (Continue) --HTTPS--> llm.lab.internal (Traefik) --> LiteLLM "code"
FIM: POST /v1/completions <|fim_prefix|>…<|fim_suffix|>…<|fim_middle|>
Chat: POST /v1/chat/completions --> vllm-code.ki-models:8000
Drill 1: Choose a model – license and VRAM
| Model | License (as of 10/2026) | Weights | Fits on an L4 24 GB? |
|---|---|---|---|
| Qwen2.5-Coder-7B-Instruct-AWQ | Apache-2.0 | approx. 5.6 GB | yes, alongside chat and embed |
| Qwen2.5-Coder-1.5B (base) | Apache-2.0 | approx. 3 GB (BF16) | yes, pure FIM, very fast |
| Qwen2.5-Coder-3B | Qwen Research License | approx. 6 GB | non-commercial only |
| Codestral 22B | Mistral Non-Production License | approx. 44 GB (BF16) | no; production only with a paid license |
| Devstral Small (24B) | Apache-2.0 | approx. 47 GB (BF16) | only quantized and on its own; for agents rather than FIM |
Warning: Licenses change. Check the model card on Hugging Face before use – “open weights” doesn’t automatically mean “commercially usable”.
Recalculate the GPU budget (Qwen2.5-7B: 28 layers, 4 KV heads, head dim 128):
python3 -c '
gpu=24*0.33; w=5.6; ovh=1.0
kv=2*28*4*128*2
tok=(gpu-w-ovh)*1024**3/kv
print(f"KV pro Token: {kv/1024:.0f} KiB, KV-Budget: {gpu-w-ovh:.1f} GB, "
f"Tokens: {tok:,.0f}")
print(f"parallel bei 8192: {tok/8192:.1f}, bei 1500 (Autocomplete): {tok/1500:.0f}")'Expected:
KV pro Token: 56 KiB, KV-Budget: 1.3 GB, Tokens: 24,716
parallel bei 8192: 3.0, bei 1500 (Autocomplete): 16
Why? With 33% of the L4, the KV cache is enough for about 16
simultaneous autocomplete requests – enough for a small team. More
developers need a dedicated GPU for code.
Save your progress? Sign up for free
Drill 2: Redistribute the GPU and start vllm-code
Chat 0.80 + embed 0.12 (Day 13) leave no room. New budget: chat 0.50, embed 0.12, code 0.33.
mkdir -p ~/ki-lab/tag14 && cd ~/ki-lab/tag14
export KUBECONFIG=~/ki-lab/rke2.yaml
kubectl -n ki-models get deploy vllm-chat \
-o jsonpath='{.spec.template.spec.containers[0].args[3]}{"\n"}'
kubectl -n ki-models patch deploy vllm-chat --type=json -p '[
{"op": "replace", "path": "/spec/template/spec/containers/0/args/2",
"value": "--max-model-len=8192"},
{"op": "replace", "path": "/spec/template/spec/containers/0/args/3",
"value": "--gpu-memory-utilization=0.50"},
{"op": "add", "path": "/spec/template/spec/containers/0/args/-",
"value": "--kv-cache-dtype=fp8"}]'
kubectl -n ki-models rollout status deploy/vllm-chat --timeout=15m
kubectl apply -f ~/ki-lab/code/code-llm/vllm-code.yaml
kubectl -n ki-models rollout status deploy/vllm-code --timeout=15m
kubectl -n ki-models get pods -l 'app in (vllm-chat,vllm-embed,vllm-code)'Expected (abbreviated, may differ depending on version):
--gpu-memory-utilization=0.80
deployment.apps/vllm-chat patched
deployment "vllm-chat" successfully rolled out
persistentvolumeclaim/vllm-code-cache created
deployment.apps/vllm-code created
service/vllm-code created
deployment "vllm-code" successfully rolled out
NAME READY STATUS RESTARTS AGE
vllm-chat-7f9c8d6b4-qx2lm 1/1 Running 0 6m
vllm-code-5b7d9f8c6-h4k2p 1/1 Running 0 4m
vllm-embed-6c8f7d5b9-m3n8r 1/1 Running 0 1d
Why? Order matters: vllm-chat has to release memory
first, then vllm-code starts. The other way around,
vllm-code aborts with “not enough free memory”. With only
50% of the L4 left, a 16k sequence (about 2.3 GiB of KV cache) no longer
fits into the cache – vLLM would abort with “max seq len”. That’s why
the patch halves --max-model-len to 8192 and stores the KV
cache in FP8. Also enter all three changes in
code/vllm/vllm-chat.yaml; otherwise the next
kubectl apply resets the old value.
Save your progress? Sign up for free
Drill 3: FIM directly against the model
kubectl -n ki-models port-forward svc/vllm-code 8002:8000 >/dev/null 2>&1 &
sleep 2
jq -n --arg p $'def is_even(n: int) -> bool:\n return ' --arg s $'\n' '{
model: "qwen2.5-coder-7b", max_tokens: 16, temperature: 0,
prompt: ("<|fim_prefix|>" + $p + "<|fim_suffix|>" + $s + "<|fim_middle|>")}' \
> fim.json
curl -s localhost:8002/v1/completions -H 'Content-Type: application/json' \
-d @fim.json | jq '{text: .choices[0].text, usage}'
jq '.suffix = "\n" | .prompt = "def is_even(n: int) -> bool:\n return "' fim.json \
| curl -s localhost:8002/v1/completions -H 'Content-Type: application/json' \
-d @- | jq -c '.error // .choices[0].text'Expected (abbreviated, may differ depending on version):
{
"text": "n % 2 == 0",
"usage": {
"prompt_tokens": 18,
"total_tokens": 25,
"completion_tokens": 7
}
}
{"message":"suffix is not currently supported", ...}
Why? The first request is real FIM: the model fills exactly the gap
and stops. The second uses the OpenAI parameter suffix –
vLLM rejects it. That’s why every client needs a prompt template with
the model’s FIM tokens.
Save your progress? Sign up for free
Drill 4: Alias code and team
key
Add the entries from
code/code-llm/litellm-models-code.yaml to
~/ki-lab/code/litellm-prod/config.yaml (code;
you already have embed from Day 13) and turn off storing
prompts:
- model_name: code
litellm_params:
model: hosted_vllm/qwen2.5-coder-7b # --served-model-name
api_base: http://vllm-code.ki-models:8000/v1
timeout: 30
model_info:
mode: chatgeneral_settings:
store_prompts_in_spend_logs: false # default, explicit here for the audit
litellm_settings:
turn_off_message_logging: true # no prompts to logging callbackscd ~/ki-lab/code/litellm-prod
kubectl -n ki-gateway create configmap litellm-config --from-file=config.yaml \
--dry-run=client -o yaml | kubectl apply -f -
kubectl -n ki-gateway rollout restart deploy/litellm
kubectl -n ki-gateway rollout status deploy/litellm
export LITELLM_MASTER_KEY=sk-lab-master-1234 # lab value
H=(-H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json')
API=https://llm.lab.internal
TEAM=$(curl -s $API/team/new "${H[@]}" -d '{"team_alias": "team-code",
"models": ["code", "embed"], "max_budget": 30, "budget_duration": "30d",
"rpm_limit": 600}' | jq -r .team_id)
CODE_KEY=$(curl -s $API/key/generate "${H[@]}" -d "{\"team_id\": \"$TEAM\",
\"key_alias\": \"thomas-vscode\", \"models\": [\"code\", \"embed\"]}" \
| jq -r .key)
echo "${CODE_KEY:0:6}..."
curl -s $API/v1/completions -H "Authorization: Bearer $CODE_KEY" \
-H 'Content-Type: application/json' -d @<(jq '.model = "code"' fim.json) \
| jq -r '.model, .choices[0].text'Expected (abbreviated, may differ depending on version):
configmap/litellm-config configured
deployment "litellm" successfully rolled out
sk-Xa1...
code
n % 2 == 0
Why? Autocomplete quickly generates 20–60 requests per minute per
developer. The team limit rpm_limit: 600 protects the GPU
from a broken plugin. For code there is deliberately
no fallback to chat-large-eu: source code
never leaves the network.
Save your progress? Sign up for free
Drill 5: Measure latency with code_eval.py
code/code-llm/code_eval.py sends three FIM tasks
(Python, Python, YAML) and two chat questions via LiteLLM, measures time
to first token (TTFT) and total time, and checks simple expectations.
First the cross-check on the laptop against LiteLLM from Day 4, where
code points to a chat model:
source ~/ki-lab/.venv/bin/activate && pip install -q openai
LITELLM_URL=http://127.0.0.1:4000/v1 LITELLM_KEY=sk-lab-master-1234 \
RUNS=3 FIM_P95_MS=3000 python ~/ki-lab/code/code-llm/code_eval.pyExpected (laptop, Ollama with ministral-3:3b, values
fluctuate):
Modell: code FIM-Stil: qwen Läufe: 3
Art TTFT p50 TTFT p95 Total p50 Total p95
FIM 579ms 708ms 1836ms 2078ms
Chat 566ms 628ms 2777ms 4610ms
Treffer: 4/11
✔ FIM p95 2078ms (Ziel ≤ 3000ms)
Only 4 of 11 hits: a chat model doesn’t know the FIM tokens and
starts chatting (“Here’s the completed function…”). With
DEBUG=1 you see the raw answers. Now against the real code
model:
export SSL_CERT_FILE=~/ki-lab/lab-ca.crt
LITELLM_URL=$API/v1 LITELLM_KEY=$CODE_KEY RUNS=10 \
python ~/ki-lab/code/code-llm/code_eval.pyExpected (L4, abbreviated, may differ depending on version and load):
Modell: code FIM-Stil: qwen Läufe: 10
Art TTFT p50 TTFT p95 Total p50 Total p95
FIM 45ms 90ms 160ms 310ms
Chat 60ms 120ms 3900ms 5200ms
Treffer: 30/32
✔ FIM p95 310ms (Ziel ≤ 500ms)
| Metric | Autocomplete target | Adjusting knob |
|---|---|---|
| TTFT p95 | < 150 ms | prefix caching, short prompt (maxPromptTokens) |
| Total p95 | < 500 ms | max_tokens 32–64, smaller model (1.5B) |
| Error rate | < 1% | dedicated GPU slots, rpm_limit per team |
Why? The script measures time from sending the request. vLLM sends the HTTP headers only with the first token – if you measure from the response, you see a TTFT close to 0 ms.
Save your progress? Sign up for free
Drill 6: Continue in VS Code
Continue reads ~/.continue/config.yaml (format
schema: v1; config.json is deprecated) and
secrets from ~/.continue/.env.
mkdir -p ~/.continue
[ -f ~/.continue/config.yaml ] && cp ~/.continue/config.yaml ~/.continue/config.yaml.bak
sed "s|__HOME__|$HOME|g" ~/ki-lab/code/code-llm/continue-config.yaml \
> ~/.continue/config.yaml
printf 'LITELLM_CODE_KEY=%s\n' "$CODE_KEY" > ~/.continue/.env
chmod 600 ~/.continue/.env
grep -nE 'model:|roles:|- (chat|autocomplete|embed)$|caBundlePath' \
~/.continue/config.yaml | head -8Expected (home path will differ):
10: model: code
14: caBundlePath: /Users/thomas/ki-lab/lab-ca.crt
15: roles:
16: - chat
24: model: code
28: caBundlePath: /Users/thomas/ki-lab/lab-ca.crt
30: roles:
31: - autocomplete
The important parts of the autocomplete model:
| Key | Value | Why |
|---|---|---|
useLegacyCompletionsEndpoint |
true |
FIM goes to /v1/completions, not to chat |
promptTemplates.autocomplete |
Qwen FIM tokens | Continue recognizes the model by its name – code
reveals nothing |
requestOptions.caBundlePath |
~/ki-lab/lab-ca.crt |
lab CA for llm.lab.internal |
autocompleteOptions.debounceDelay |
250 ms | request only after a typing pause |
Test in VS Code: install the “Continue” extension, create a new file
test.py, type def fib(n): and wait briefly – a
gray suggestion appears; Tab accepts it. In the chat panel
(Cmd+L), choose the model “Lab Code (Chat)”. At the same
time, in the cluster:
kubectl -n ki-gateway logs deploy/litellm --since=2m | grep -c 'POST /v1/completions'Expected: a number > 0 (one request per typing pause).
Tip: If autocomplete stays silent, open Output → Continue in VS Code. “fetch failed” almost always means: wrong CA path, or
llm.lab.internaldoesn’t resolve to10.10.20.100.
Save your progress? Sign up for free
Drill 7: Tabby as an alternative
Tabby is a separate server with a web UI, user management and plugins for VS Code and JetBrains. Here it doesn’t use its own GPU but the same LiteLLM aliases.
cd ~/ki-lab/tag14
kubectl create namespace ki-code --dry-run=client -o yaml | kubectl apply -f -
TABBY_KEY=$(curl -s $API/key/generate "${H[@]}" -d "{\"team_id\": \"$TEAM\",
\"key_alias\": \"tabby\", \"models\": [\"code\", \"embed\"]}" | jq -r .key)
sed "s|__LITELLM_TABBY_KEY__|$TABBY_KEY|" \
~/ki-lab/code/code-llm/tabby-config.toml > config.toml
kubectl -n ki-code create secret generic tabby-config --from-file=config.toml
rm config.toml
kubectl apply -f ~/ki-lab/code/code-llm/tabby.yaml
kubectl -n ki-code rollout status deploy/tabby --timeout=5m
curl -s https://code.lab.internal/v1/health | jq '{model, chat_model, version}'Expected (abbreviated, may differ depending on version):
namespace/ki-code created
secret/tabby-config created
persistentvolumeclaim/tabby-data created
deployment.apps/tabby created
service/tabby created
ingress.networking.k8s.io/tabby created
deployment "tabby" successfully rolled out
{
"model": "code",
"chat_model": "code",
"version": { "git_describe": "v0.32.0", ... }
}
Then open https://code.lab.internal, create an admin
account, and enter the server URL and your personal token in the VS Code
plugin “Tabby”.
| Criterion | Continue | Tabby |
|---|---|---|
| Operation | client configuration only | server with DB, users, tokens |
| Configuration | config.yaml per developer |
central config.toml |
| Repo context | local in the editor | server indexes Git repos |
| Reporting | via LiteLLM keys | its own usage statistics plus LiteLLM |
Why? For a small team, Continue is enough: nothing to operate, governance via virtual keys. Tabby pays off if you want to control centrally who gets which repos as context.
Save your progress? Sign up for free
Drill 8: Governance – no code leaves the network
kubectl apply -f ~/ki-lab/code/code-llm/netpol.yaml
kubectl -n default run np-test --rm -i --restart=Never \
--image=curlimages/curl:8.16.0 -- curl -s -m 5 -o /dev/null \
-w '%{http_code}\n' http://vllm-code.ki-models:8000/health || echo "blockiert"
kubectl -n ki-gateway exec deploy/litellm -- python -c "import urllib.request as u
print(u.urlopen('http://vllm-code.ki-models:8000/health').status)"
grep -A5 'model_name: code' ~/ki-lab/code/litellm-prod/config.yaml \
| grep -c mistralExpected (abbreviated, may differ depending on version):
networkpolicy.networking.k8s.io/vllm-code-ingress created
networkpolicy.networking.k8s.io/tabby-egress created
000
blockiert
200
0
Why? Cilium enforces Kubernetes NetworkPolicies: only LiteLLM (and
Prometheus) reach vllm-code. So every code request goes
through LiteLLM – with key, budget and audit.
grep -c mistral = 0 proves: there’s no external fallback
for code.
The logging policy for the team, short and in writing:
- LiteLLM stores metadata (key, model, tokens, duration),
no prompts:
store_prompts_in_spend_logs: false,turn_off_message_logging: true. - vLLM doesn’t log prompts as long as nobody sets
--enable-log-requests(plus DEBUG level) or--enable-log-outputs. - Continue: turn off telemetry in the extension settings. Tabby:
TABBY_DISABLE_USAGE_COLLECTION=1(it’s intabby.yaml). - Secrets in code:
.continueignoreoronlyMyCodelimit the context, but don’t replace a secret scanner in CI.
Save your progress? Sign up for free
Drill 9: Outlook – agentic coding with guardrails
Agents like Cline (VS Code extension) or OpenHands (its own server
with sandbox containers) read files, run commands and call the model in
loops – 50–200 requests per task are normal. Both speak the OpenAI API,
so https://llm.lab.internal/v1 with a dedicated key.
AGENT_KEY=$(curl -s $API/key/generate "${H[@]}" -d "{\"team_id\": \"$TEAM\",
\"key_alias\": \"agent-thomas\", \"models\": [\"code\"], \"max_budget\": 2,
\"budget_duration\": \"1d\", \"rpm_limit\": 30, \"max_parallel_requests\": 2,
\"metadata\": {\"use\": \"agentic\"}}" | jq -r .key)
curl -s "$API/key/info" -H "Authorization: Bearer $AGENT_KEY" \
| jq -c '.info | {key_alias, max_budget, rpm_limit, max_parallel_requests}'Expected (may differ depending on version):
{"key_alias":"agent-thomas","max_budget":2,"rpm_limit":30,"max_parallel_requests":2}
| Guardrail | Implementation |
|---|---|
| Cost and load | dedicated key: daily budget, rpm_limit,
max_parallel_requests |
| Execution | agent only in a devcontainer or OpenHands sandbox, never on the host with a production kubeconfig |
| Approvals | Cline: “Auto-approve” off for commands; confirm every shell action |
| Write access | own Git branch, merge only via review; no push rights on
main |
| Model | 7B is enough for autocomplete; for agents rather 24B+ (Devstral) on a dedicated GPU |
Why? An agent with shell access is a user with your permissions who types fast. The guardrails belong in infrastructure (key, sandbox, branch protection), not in the prompt.
Save your progress? Sign up for free
Checkpoint
#!/usr/bin/env bash
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
kubectl -n ki-models rollout status deploy/vllm-code --timeout=10s >/dev/null 2>&1 \
&& ok "vllm-code läuft" || nok "vllm-code läuft"
curl -s $API/v1/completions -H "Authorization: Bearer $CODE_KEY" \
-H 'Content-Type: application/json' -d @<(jq '.model = "code"' fim.json) \
| jq -e '.choices[0].text | test("%")' >/dev/null \
&& ok "FIM über LiteLLM" || nok "FIM über LiteLLM"
LITELLM_URL=$API/v1 LITELLM_KEY=$CODE_KEY RUNS=5 \
python ~/ki-lab/code/code-llm/code_eval.py >/dev/null \
&& ok "FIM p95 ≤ 500 ms" || nok "FIM p95 ≤ 500 ms"
kubectl -n ki-models get networkpolicy vllm-code-ingress >/dev/null 2>&1 \
&& ok "NetworkPolicy vllm-code" || nok "NetworkPolicy vllm-code"
curl -sf https://code.lab.internal/v1/health >/dev/null \
&& ok "Tabby erreichbar" || nok "Tabby erreichbar"
grep -q 'LITELLM_CODE_KEY=sk-' ~/.continue/.env \
&& ok "Continue-Key gesetzt" || nok "Continue-Key gesetzt"Challenge
A working student should get autocomplete only, no chat: a key that
may call only /v1/completions, 30 requests per minute,
valid for 30 days.
Hint: Besides models,
/key/generate also knows allowed_routes and
duration.
Solution
WS_KEY=$(curl -s $API/key/generate "${H[@]}" -d "{\"team_id\": \"$TEAM\",
\"key_alias\": \"werkstudentin\", \"models\": [\"code\"], \"rpm_limit\": 30,
\"duration\": \"30d\", \"allowed_routes\": [\"/v1/completions\",
\"/completions\"]}" | jq -r .key)
for ep in completions chat/completions; do
curl -s -o /dev/null -w "$ep: %{http_code}\n" $API/v1/$ep \
-H "Authorization: Bearer $WS_KEY" -H 'Content-Type: application/json' \
-d '{"model": "code", "prompt": "x", "max_tokens": 1,
"messages": [{"role": "user", "content": "x"}]}'
doneExpected (checked against the docs, may differ depending on version):
completions: 200
chat/completions: 401
In Continue, this key gets only the model with
roles: [autocomplete].
Save your progress? Sign up for free
Enterprise Note
- From about 15 active developers on,
codegets a dedicated GPU; you separate autocomplete and agents via two aliases and two vLLM Deployments. - Distribute the Continue configuration centrally (MDM, dotfiles repo); keys per person from self-service (LiteLLM UI with SSO), never shared team keys.
- License review as an approval process: store model, version, license text and hash of the weights in the model registry.
- Works council and data protection: metadata per person (who uses how much) is performance data – report only aggregated per team.
- Agentic coding only in sandboxes without access to production secrets; OpenHands runtime containers with their own NetworkPolicy.