Skip to content
Hands-on IT IT Drills

Day 11: Embeddings and RAG – docsearch queries your notes

Goal: Split Markdown notes into chunks, index them as vectors, search semantically and answer questions with source citations – plus measure how well it works · Duration: approx. 150 min · Prerequisite: Day 10

Briefing

  • An embedding is a vector (here 384 numbers) that represents the meaning of a text. Similar texts lie close to each other.
  • Cosine similarity measures the angle between two vectors: 1 = identical, 0 = unrelated.
  • RAG (Retrieval-Augmented Generation): first search for matching chunks, then give only those to the LLM as context – with their source.
  • qdrant stores vectors and searches fast; in local mode without a server, just a folder.
  • Without evaluation, every RAG change is gut feeling.
notes/*.md ─chunk─> chunks ─embed (all-minilm)─> vectors ──> qdrant (.docsearch/)
                                                                │
query ─embed─> vector ─────────── top-k search ─────────────────┘
                                      │
                        hits + query ─> ministral-3:3b ─> answer [file.md#section]

Drill 1: Project, notes and embedding model

cd ~/py-lab
uv init -q --package --name docsearch tag11-docsearch
cd tag11-docsearch
uv add -q openai numpy qdrant-client typer
uv add -q --dev pytest ruff
cat >> pyproject.toml <<'EOF'

[tool.ruff.lint]
extend-select = ["E501", "B", "UP", "SIM"]

[tool.pytest.ini_options]
testpaths = ["tests"]
EOF
uv tree --depth 1 | grep -E 'numpy|qdrant'
ollama pull all-minilm
ollama list | grep -E 'NAME|all-minilm'

Expected (abridged, versions may differ):

├── numpy v2.5.3
├── qdrant-client v1.19.1
NAME                 ID              SIZE      MODIFIED
all-minilm:latest    1b226e2802db    45 MB     4 minutes ago

Now five small notes as a test corpus:

mkdir -p notes
cat > notes/vault.md <<'EOF'
# Vault Betrieb

## Entsiegeln
Nach einem Neustart ist Vault versiegelt. Drei von fünf Key-Shares eingeben:
`vault operator unseal` dreimal ausführen, Status mit `vault status` prüfen.

## Token erneuern
Lab-Token `sk-lab-master-1234` ist nur für das Lab. Eigene Tokens mit
`vault token renew` verlängern, TTL-Maximum beträgt 32 Tage.
EOF
cat > notes/k8s.md <<'EOF'
# Kubernetes Debugging

## CrashLoopBackOff
Ein Pod startet immer wieder neu. Erst `kubectl logs <pod> --previous`, dann
`kubectl describe pod <pod>` lesen. Häufig: falsches Secret oder OOMKilled.

## Pending Pods
Pod bleibt Pending: meist fehlen CPU/RAM-Ressourcen oder ein PVC ist nicht gebunden.
`kubectl get events --sort-by=.lastTimestamp` zeigt den Grund.
EOF
cat > notes/backup.md <<'EOF'
# Backup

## Zeitplan
Die Sicherung läuft jede Nacht um 02:00 Uhr per restic auf das NAS `nas01`.
Aufbewahrung: 7 tägliche, 4 wöchentliche, 12 monatliche Snapshots.

## Restore
Wiederherstellen mit `restic restore latest --target /tmp/restore`.
Vorher immer `restic snapshots` prüfen.
EOF
cat > notes/oncall.md <<'EOF'
# Rufbereitschaft

## Eskalation
Reagiert der Primary nicht innerhalb von 15 Minuten, eskaliert PagerDuty an
den Secondary. Nach 30 Minuten wird die Teamleitung angerufen.

## Übergabe
Übergabe jeden Montag 10:00 Uhr im Ops-Channel mit offenen Incidents.
EOF
cat > notes/python.md <<'EOF'
# Python Setup

## Neues Projekt
`uv init --package` legt ein Projekt mit src-Layout an, `uv add` fügt
Abhängigkeiten hinzu, `uv run` startet im Projekt-venv.

## Tests
Tests liegen unter `tests/` und laufen mit `uv run pytest -q`.
EOF
ls notes

Expected:

backup.md   k8s.md      oncall.md   python.md   vault.md

Tip: all-minilm is small (45 MB) and fast, but trained on English. For German texts, multilingual models are better – you will see the consequences in Drill 7.

Save your progress? Sign up for free

Drill 2: Chunking at headings

One chunk per ## section. The heading stays in the text – it often carries most of the meaning.

cat > src/docsearch/chunking.py <<'EOF'
"""Markdown-Dateien in Abschnitte (Chunks) zerlegen."""

from dataclasses import dataclass
from pathlib import Path


@dataclass(frozen=True)
class Chunk:
    source: str  # z. B. "vault.md#Entsiegeln"
    text: str


def split_long(text: str, max_chars: int) -> list[str]:
    """Fasst Absätze zusammen, bis max_chars erreicht ist."""
    parts: list[str] = []
    current = ""
    for para in text.split("\n\n"):
        if current and len(current) + len(para) > max_chars:
            parts.append(current)
            current = ""
        current = f"{current}\n\n{para}".strip()
    if current:
        parts.append(current)
    return parts


def split_markdown(path: Path, max_chars: int = 800) -> list[Chunk]:
    """Ein Chunk pro Überschrift; die Überschrift bleibt als Kontext im Text."""
    chunks: list[Chunk] = []
    section = path.stem
    lines: list[str] = []

    def flush() -> None:
        body = "\n".join(lines).strip()
        for part in split_long(body, max_chars) if body else []:
            chunks.append(Chunk(f"{path.name}#{section}", f"{section}\n{part}"))
        lines.clear()

    for line in path.read_text(encoding="utf-8").splitlines():
        if line.startswith("#"):
            flush()
            section = line.lstrip("#").strip()
        else:
            lines.append(line)
    flush()
    return chunks


def load_notes(folder: Path) -> list[Chunk]:
    return [c for p in sorted(folder.rglob("*.md")) for c in split_markdown(p)]
EOF
uv run python -c "
from pathlib import Path
from docsearch.chunking import load_notes
chunks = load_notes(Path('notes'))
print(len(chunks), 'Chunks')
for c in chunks[:3]:
    print(c.source, '|', c.text.splitlines()[1][:50])
"

Expected:

10 Chunks
backup.md#Zeitplan | Die Sicherung läuft jede Nacht um 02:00 Uhr per re
backup.md#Restore | Wiederherstellen mit `restic restore latest --targ
k8s.md#CrashLoopBackOff | Ein Pod startet immer wieder neu. Erst `kubectl lo

Why 10 and not 15? The # title lines have no text of their own and produce no chunk. source later becomes the source citation. For long sections, split_long splits at paragraphs instead of mid-sentence.

Save your progress? Sign up for free

Drill 3: Embeddings and cosine with numpy

cat > src/docsearch/vectors.py <<'EOF'
"""Embeddings über Ollama, Kosinus-Ähnlichkeit mit numpy."""

import os

import numpy as np
from openai import OpenAI

BASE_URL = os.environ.get("DOCSEARCH_BASE_URL", "http://127.0.0.1:11434/v1")
EMBED_MODEL = os.environ.get("DOCSEARCH_EMBED_MODEL", "all-minilm")


def make_client() -> OpenAI:
    key = os.environ.get("DOCSEARCH_API_KEY", "ollama")
    return OpenAI(base_url=BASE_URL, api_key=key, timeout=120, max_retries=1)


def embed(client: OpenAI, texts: list[str]) -> np.ndarray:
    """Liefert eine (n, dim)-Matrix mit Länge 1 je Zeile."""
    resp = client.embeddings.create(model=EMBED_MODEL, input=texts)
    vecs = np.array([d.embedding for d in resp.data], dtype=np.float32)
    return vecs / np.linalg.norm(vecs, axis=1, keepdims=True)


def top_k(query: np.ndarray, matrix: np.ndarray, k: int) -> list[tuple[int, float]]:
    """Kosinus = Skalarprodukt, weil alle Vektoren normiert sind."""
    scores = matrix @ query
    best = np.argsort(scores)[::-1][:k]
    return [(int(i), float(scores[i])) for i in best]
EOF
uv run python -c "
from pathlib import Path
from docsearch.chunking import load_notes
from docsearch.vectors import embed, make_client, top_k
client = make_client()
chunks = load_notes(Path('notes'))
m = embed(client, [c.text for c in chunks])
print(m.shape, m.dtype, round(float((m[0] ** 2).sum()), 3))
for q in ['Wie entsiegle ich Vault?', 'Mein Pod startet ständig neu']:
    print(q)
    for i, score in top_k(embed(client, [q])[0], m, 2):
        print(f'  {score:.2f} {chunks[i].source}')
"

Expected (values may differ slightly):

(10, 384) float32 1.0
Wie entsiegle ich Vault?
  0.59 vault.md#Entsiegeln
  0.45 vault.md#Token erneuern
Mein Pod startet ständig neu
  0.56 k8s.md#CrashLoopBackOff
  0.47 oncall.md#Übergabe

Why normalize? Cosine is a·b / (|a|·|b|). If all vectors have length 1, only a·b remains – and matrix @ query computes all dot products in one go. One batch call for all chunks is much faster than one call per chunk.

Save your progress? Sign up for free

Drill 4: A persistent index with qdrant

numpy is enough for a hundred chunks in RAM. For thousands of notes, persistence and filters, you use a vector DB. Here qdrant runs embedded, with no server at all.

cat > src/docsearch/store.py <<'EOF'
"""Persistenter Vektorindex mit qdrant im lokalen Modus (kein Server)."""

from dataclasses import dataclass
from pathlib import Path

import numpy as np
from qdrant_client import QdrantClient, models

from docsearch.chunking import Chunk

COLLECTION = "notes"


@dataclass
class Hit:
    source: str
    text: str
    score: float


def open_store(path: Path) -> QdrantClient:
    return QdrantClient(path=str(path))


def rebuild(store: QdrantClient, chunks: list[Chunk], vectors: np.ndarray) -> int:
    if store.collection_exists(COLLECTION):
        store.delete_collection(COLLECTION)
    store.create_collection(
        COLLECTION,
        vectors_config=models.VectorParams(
            size=vectors.shape[1], distance=models.Distance.COSINE
        ),
    )
    points = [
        models.PointStruct(
            id=i, vector=v.tolist(), payload={"source": c.source, "text": c.text}
        )
        for i, (c, v) in enumerate(zip(chunks, vectors, strict=True))
    ]
    store.upsert(COLLECTION, points=points)
    return store.count(COLLECTION).count


def search(store: QdrantClient, vector: np.ndarray, k: int = 3) -> list[Hit]:
    res = store.query_points(COLLECTION, query=vector.tolist(), limit=k)
    return [
        Hit(p.payload["source"], p.payload["text"], p.score)
        for p in res.points
        if p.payload
    ]
EOF
uv run python -c "
from pathlib import Path
from docsearch.chunking import load_notes
from docsearch.store import open_store, rebuild, search
from docsearch.vectors import embed, make_client
client = make_client()
chunks = load_notes(Path('notes'))
store = open_store(Path('.docsearch'))
print('Punkte:', rebuild(store, chunks, embed(client, [c.text for c in chunks])))
for h in search(store, embed(client, ['Wann läuft die Sicherung?'])[0], k=2):
    print(f'{h.score:.2f} {h.source}')
store.close()
"
ls .docsearch

Expected:

Punkte: 10
0.32 oncall.md#Übergabe
0.32 oncall.md#Eskalation
collection  meta.json

Why isn’t the backup note at the top? The question says “Sicherung” (backup), and so does the note – but all-minilm understands German only roughly. Low scores around 0.3 are a warning sign: that is not a real hit. This is exactly why you build a threshold into Drill 5.

Warning: Local mode locks the folder for one process. Two simultaneous docsearch calls on the same .docsearch/ fail – for multiple users, you start the qdrant server.

Save your progress? Sign up for free

Drill 5: RAG – answer only from the context, with source

cat > src/docsearch/rag.py <<'EOF'
"""Retrieval-Augmented Generation: Treffer als Kontext an das Chat-Modell."""

import os

from openai import OpenAI

from docsearch.store import Hit

CHAT_MODEL = os.environ.get("DOCSEARCH_CHAT_MODEL", "ministral-3:3b")
MIN_SCORE = 0.35  # darunter gilt ein Treffer als Rauschen
NO_ANSWER = "Dazu finde ich nichts in deinen Notizen."
SYSTEM = (
    "Beantworte die Frage NUR mit dem Kontext, auf Deutsch, in max. 3 Sätzen. "
    "Nenne nach jeder Aussage die Quelle in eckigen Klammern, z. B. "
    f"[vault.md#Entsiegeln]. Steht es nicht im Kontext, antworte: {NO_ANSWER}"
)


def build_prompt(question: str, hits: list[Hit]) -> str:
    context = "\n\n".join(f"[{h.source}]\n{h.text}" for h in hits)
    return f"Kontext:\n{context}\n\nFrage: {question}"


def answer(client: OpenAI, question: str, hits: list[Hit]) -> str:
    relevant = [h for h in hits if h.score >= MIN_SCORE]
    if not relevant:
        return NO_ANSWER  # spart den LLM-Aufruf und verhindert Halluzination
    resp = client.chat.completions.create(
        model=CHAT_MODEL,
        messages=[
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": build_prompt(question, relevant)},
        ],
        temperature=0,
    )
    return resp.choices[0].message.content or ""
EOF
uv run ruff check src

Expected:

All checks passed!

Why temperature=0? With RAG you want faithfulness to the context, not creativity. And the threshold: without MIN_SCORE, the model would get only noise as context and would still make something up. “I don’t know” is a good answer in RAG.

Save your progress? Sign up for free

Drill 6: The CLI with four commands

cat > src/docsearch/cli.py <<'EOF'
"""docsearch – eigene Markdown-Notizen durchsuchen und befragen."""

import json
from pathlib import Path
from typing import Annotated

import typer

from docsearch.chunking import load_notes
from docsearch.rag import answer
from docsearch.store import open_store, rebuild, search
from docsearch.vectors import embed, make_client

app = typer.Typer(add_completion=False, no_args_is_help=True)
DbOpt = Annotated[Path, typer.Option("--db", envvar="DOCSEARCH_DB")]
K = Annotated[int, typer.Option("-k", min=1, max=20)]


@app.command()
def index(folder: Path, db: DbOpt = Path(".docsearch")) -> None:
    """Liest alle *.md unter FOLDER und baut den Index neu."""
    chunks = load_notes(folder)
    if not chunks:
        typer.secho(f"Keine Markdown-Abschnitte in {folder}", fg="red", err=True)
        raise typer.Exit(1)
    vectors = embed(make_client(), [c.text for c in chunks])
    store = open_store(db)
    count = rebuild(store, chunks, vectors)
    store.close()
    typer.echo(f"{count} Chunks aus {folder} indiziert ({vectors.shape[1]} Dim.)")


@app.command("search")
def search_cmd(query: str, k: K = 3, db: DbOpt = Path(".docsearch")) -> None:
    """Zeigt die k ähnlichsten Abschnitte."""
    store = open_store(db)
    for hit in search(store, embed(make_client(), [query])[0], k):
        first = hit.text.splitlines()[1][:50]
        typer.echo(f"{hit.score:.2f}  {hit.source:28} {first}")
    store.close()


@app.command()
def ask(question: str, k: K = 3, db: DbOpt = Path(".docsearch")) -> None:
    """Beantwortet QUESTION nur aus den Notizen, mit Quellen."""
    client = make_client()
    store = open_store(db)
    hits = search(store, embed(client, [question])[0], k)
    store.close()
    typer.echo(answer(client, question, hits))


@app.command("eval")
def eval_cmd(cases: Path, k: K = 3, db: DbOpt = Path(".docsearch")) -> None:
    """Misst Hit@1 und Hit@k: Ist die erwartete Datei unter den Treffern?"""
    data = json.loads(cases.read_text(encoding="utf-8"))
    client = make_client()
    store = open_store(db)
    hit1 = hitk = 0
    for case in data:
        hits = search(store, embed(client, [case["question"]])[0], k)
        files = [h.source.split("#")[0] for h in hits]
        hit1 += files[:1] == [case["source"]]
        hitk += case["source"] in files
        mark = "✔" if case["source"] in files else "✘"
        typer.echo(f"{mark} {case['question'][:40]:40} -> {files[0] if files else '-'}")
    store.close()
    n = len(data)
    typer.echo(f"Hit@1: {hit1 / n:.2f}  Hit@{k}: {hitk / n:.2f}  (n={n})")
EOF
cat > src/docsearch/__init__.py <<'EOF'
"""docsearch – eigene Markdown-Notizen durchsuchen und befragen."""

import openai
import typer

from docsearch.cli import app


def main() -> None:
    try:
        app()
    except openai.APIConnectionError:
        typer.secho("Fehler: Ollama nicht erreichbar", fg="red", err=True)
        raise SystemExit(1) from None
EOF
cat > src/docsearch/__main__.py <<'EOF'
from docsearch import main

main()
EOF
rm -rf .docsearch
uv run docsearch index notes
uv run docsearch search "Pod startet ständig neu"
uv run docsearch ask "Wer wird nach 30 Minuten angerufen?"
uv run docsearch ask "Wann läuft die Sicherung?"

Expected (text varies, abridged):

10 Chunks aus notes indiziert (384 Dim.)
0.59  k8s.md#CrashLoopBackOff      Ein Pod startet immer wieder neu. Erst `kubectl lo
0.45  k8s.md#Pending Pods          Pod bleibt Pending: meist fehlen CPU/RAM-Ressource
0.45  oncall.md#Übergabe           Übergabe jeden Montag 10:00 Uhr im Ops-Channel mit
Nach 30 Minuten wird die **Teamleitung** (meist der Oncall-Leiter oder Ops Manager)
per Telefon oder Notruf kontaktiert [oncall.md#Eskalation].
Die genaue Rolle variiert je nach Organisation, ... [oncall.md#Übergabe].
Dazu finde ich nichts in deinen Notizen.

The backup question stays unanswered, even though the note exists: all scores are below MIN_SCORE (Drill 4). Retrieval is the problem, not the LLM. Why does main() in __init__.py catch the connection error? Once, centrally, instead of in each of the four commands – with exit code 1, as in ask (Day 10).

Warning: Read the answer critically: “meist der Oncall-Leiter oder Ops Manager” (usually the on-call lead or ops manager) is in no note, and [oncall.md#Übergabe] does not back up the second sentence. A source citation from the LLM is a claim, not proof – the Challenge at least checks whether cited sources were in the context at all.

Security: docsearch search token -k 1 finds the section containing sk-lab-master-1234. Everything you index sits in plain text in .docsearch/ and ends up in the prompt. Secrets do not belong in notes you index.

Save your progress? Sign up for free

Drill 7: Evaluation – numbers instead of gut feeling

A small golden set: a question plus the file that contains the answer.

cat > eval.json <<'EOF'
[
  {"question": "Wie entsiegle ich Vault?", "source": "vault.md"},
  {"question": "Pod startet immer wieder neu", "source": "k8s.md"},
  {"question": "Wann läuft die Sicherung?", "source": "backup.md"},
  {"question": "Wer wird nach 30 Minuten angerufen?", "source": "oncall.md"},
  {"question": "Wie starte ich Tests?", "source": "python.md"},
  {"question": "Warum bleibt mein Pod Pending?", "source": "k8s.md"},
  {"question": "Wie hole ich Dateien aus dem Backup zurück?", "source": "backup.md"},
  {"question": "Wie lange gilt ein Vault-Token maximal?", "source": "vault.md"}
]
EOF
uv run docsearch eval eval.json

Expected:

✔ Wie entsiegle ich Vault?                 -> vault.md
✔ Pod startet immer wieder neu             -> k8s.md
✘ Wann läuft die Sicherung?                -> oncall.md
✔ Wer wird nach 30 Minuten angerufen?      -> oncall.md
✔ Wie starte ich Tests?                    -> python.md
✔ Warum bleibt mein Pod Pending?           -> k8s.md
✔ Wie hole ich Dateien aus dem Backup zurü -> oncall.md
✔ Wie lange gilt ein Vault-Token maximal?  -> vault.md
Hit@1: 0.75  Hit@3: 0.88  (n=8)

Why two numbers? Hit@1 says how often the best hit is right; Hit@3 says whether the right file ends up in the context at all – that is what matters for RAG. Notice: oncall.md#Übergabe shows up near the top for many questions. Short, generic chunks become “all-purpose hits” this way. Levers you can now compare measurably: a multilingual embedding model (DOCSEARCH_EMBED_MODEL), different chunking, hybrid search with keywords.

Tip: Put eval.json into the repo too and run docsearch eval in CI. If Hit@3 drops after a change, you know immediately.

Save your progress? Sign up for free

Drill 8: Tests without a model

Embeddings and chat are mocked; qdrant runs for real, but in tmp_path with 2-dimensional vectors.

mkdir -p tests
cat > tests/test_docsearch.py <<'EOF'
from pathlib import Path
from types import SimpleNamespace as NS
from unittest.mock import MagicMock

import numpy as np

from docsearch.chunking import Chunk, split_markdown
from docsearch.rag import NO_ANSWER, answer
from docsearch.store import Hit, open_store, rebuild, search
from docsearch.vectors import embed


def test_split_markdown_one_chunk_per_section(tmp_path: Path) -> None:
    note = tmp_path / "ops.md"
    note.write_text("# Ops\n\n## Restart\nsystemctl restart x\n\n## Logs\njournalctl\n")
    chunks = split_markdown(note)
    assert [c.source for c in chunks] == ["ops.md#Restart", "ops.md#Logs"]
    assert chunks[0].text == "Restart\nsystemctl restart x"


def test_embed_normalizes() -> None:
    client = MagicMock()
    client.embeddings.create.return_value = NS(data=[NS(embedding=[3.0, 4.0])])
    vec = embed(client, ["x"])
    assert np.allclose(vec, [[0.6, 0.8]])


def test_store_roundtrip(tmp_path: Path) -> None:
    chunks = [Chunk("a.md#A", "alpha"), Chunk("b.md#B", "beta")]
    store = open_store(tmp_path / "db")
    assert rebuild(store, chunks, np.array([[1.0, 0.0], [0.0, 1.0]])) == 2
    hits = search(store, np.array([0.1, 0.9]), k=1)
    store.close()
    assert hits[0].source == "b.md#B"


def test_answer_skips_llm_when_scores_low() -> None:
    client = MagicMock()
    assert answer(client, "?", [Hit("a.md#A", "x", 0.1)]) == NO_ANSWER
    client.chat.completions.create.assert_not_called()


def test_answer_sends_sources_in_prompt() -> None:
    client = MagicMock()
    client.chat.completions.create.return_value = NS(
        choices=[NS(message=NS(content="ok [a.md#A]"))]
    )
    assert answer(client, "Frage?", [Hit("a.md#A", "Inhalt", 0.9)]) == "ok [a.md#A]"
    prompt = client.chat.completions.create.call_args.kwargs["messages"][1]["content"]
    assert "[a.md#A]" in prompt and "Frage?" in prompt
EOF
uv run pytest -q
uv run ruff check . && uv run --with mypy mypy --strict src

Expected:

.....                                                                    [100%]
5 passed in 0.44s
All checks passed!
Success: no issues found in 7 source files

Why assert_not_called()? The most important path is the one in which the LLM is not asked. Only a test proves it to you.

Save your progress? Sign up for free

Drill 9: The tool is done

uv tool install --editable .
export DOCSEARCH_DB=~/.cache/docsearch
docsearch index ~/py-lab/tag11-docsearch/notes
cd ~
docsearch ask "Wie lange gilt ein Vault-Token maximal?"
docsearch search "restic" -k 1

Expected (text varies):

Installed 1 executable: docsearch
10 Chunks aus /Users/thomas/py-lab/tag11-docsearch/notes indiziert (384 Dim.)
Ein Vault-Token gilt maximal **32 Tage** [vault.md#Token erneuern].
Dazu finde ich nichts in deinen Notizen.
0.32  backup.md#Restore            Wiederherstellen mit `restic restore latest --targ

The appended “Dazu finde ich nichts” (“I can’t find anything on that”) sentence shows: a 3B model follows the system prompt literally, but not smartly. Why DOCSEARCH_DB? With a fixed index path, docsearch works from any directory. Hook index up to your notes folder as a cron job or Git hook, and the index stays current.

Save your progress? Sign up for free

Checkpoint

cd ~/py-lab/tag11-docsearch
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
DB=/tmp/ds-check
check "Embedding-Modell da" "ollama list | grep -q all-minilm"
check "Tests grün" "uv run pytest -q"
check "ruff sauber" "uv run ruff check ."
check "mypy --strict sauber" "uv run --with mypy mypy --strict src"
check "Index gebaut" "uv run docsearch index notes --db $DB | grep -q '10 Chunks'"
check "Hit@3 >= 0.8" \
  "uv run docsearch eval eval.json --db $DB | grep -qE 'Hit@3: (0\.[89]|1\.00)'"
check "Antwort mit Quelle" \
  "uv run docsearch ask 'Wann ist die Übergabe?' --db $DB | grep -q '\['"
rm -rf "$DB"

Expected:

✔ Embedding-Modell da
✔ Tests grün
✔ ruff sauber
✔ mypy --strict sauber
✔ Index gebaut
✔ Hit@3 >= 0.8
✔ Antwort mit Quelle

Challenge

The LLM sometimes invents sources. Add unknown_citations(text, hits), which returns every cited [datei.md#Abschnitt] (file and section) that was not among the relevant hits, and have docsearch ask print a warning on stderr for each one.

Hint: A regular expression on \[...\.md#...\] plus a set difference (set - set) is enough.

Solution

In cli.py, change the import to from docsearch.rag import answer, unknown_citations and make ask end like this:

    text = answer(client, question, hits)
    typer.echo(text)
    for source in unknown_citations(text, hits):
        typer.secho(f"Warnung: Quelle {source} nicht im Kontext", fg="red", err=True)

Then add the function and the test:

cat >> src/docsearch/rag.py <<'EOF'


CITATION = re.compile(r"\[([^\[\]]+\.md#[^\[\]]+)\]")


def unknown_citations(text: str, hits: list[Hit]) -> list[str]:
    """Zitierte Quellen, die gar nicht im Kontext standen."""
    allowed = {h.source for h in hits if h.score >= MIN_SCORE}
    return sorted(set(CITATION.findall(text)) - allowed)
EOF
perl -pi -e 's/^import os$/import os\nimport re/' src/docsearch/rag.py
cat >> tests/test_docsearch.py <<'EOF'


def test_unknown_citations_flags_invented_sources() -> None:
    from docsearch.rag import unknown_citations

    hits = [Hit("a.md#A", "x", 0.9), Hit("b.md#B", "y", 0.1)]
    text = "Eins [a.md#A]. Zwei [b.md#B]. Drei [c.md#C]."
    assert unknown_citations(text, hits) == ["b.md#B", "c.md#C"]
EOF
uv run pytest -q

Expected:

......                                                                   [100%]
6 passed in 0.47s

Why does b.md#B count as unknown even though it was a hit? With a score of 0.1, it was below MIN_SCORE and was never in the prompt.

Save your progress? Sign up for free

Enterprise Note

  • Permissions belong in the index: every chunk carries ACL metadata, and the search filters by user – otherwise RAG answers questions from documents the asker was never allowed to see.
  • Scan for and remove secrets and personal data before indexing (e.g. with gitleaks rules); the vector index is a copy of all the data.
  • Run qdrant as a server (or pgvector) with backup, TLS and an API key; local mode is only for single-user tools.
  • Version the embedding model and the chunking: switching models requires re-indexing; vectors from different models are not comparable.
  • Maintain the golden set and watch Hit@k, answer quality (LLM-as-judge or spot checks) and the “I don’t know” rate as metrics.