Day 11: Embeddings and RAG – docsearch
queries your notes
Goal: Split Markdown notes into chunks, index them as vectors, search semantically and answer questions with source citations – plus measure how well it works · Duration: approx. 150 min · Prerequisite: Day 10
Briefing
- An embedding is a vector (here 384 numbers) that represents the meaning of a text. Similar texts lie close to each other.
- Cosine similarity measures the angle between two vectors: 1 = identical, 0 = unrelated.
- RAG (Retrieval-Augmented Generation): first search for matching chunks, then give only those to the LLM as context – with their source.
- qdrant stores vectors and searches fast; in local mode without a server, just a folder.
- Without evaluation, every RAG change is gut feeling.
notes/*.md ─chunk─> chunks ─embed (all-minilm)─> vectors ──> qdrant (.docsearch/)
│
query ─embed─> vector ─────────── top-k search ─────────────────┘
│
hits + query ─> ministral-3:3b ─> answer [file.md#section]
Drill 1: Project, notes and embedding model
cd ~/py-lab
uv init -q --package --name docsearch tag11-docsearch
cd tag11-docsearch
uv add -q openai numpy qdrant-client typer
uv add -q --dev pytest ruff
cat >> pyproject.toml <<'EOF'
[tool.ruff.lint]
extend-select = ["E501", "B", "UP", "SIM"]
[tool.pytest.ini_options]
testpaths = ["tests"]
EOF
uv tree --depth 1 | grep -E 'numpy|qdrant'
ollama pull all-minilm
ollama list | grep -E 'NAME|all-minilm'Expected (abridged, versions may differ):
├── numpy v2.5.3
├── qdrant-client v1.19.1
NAME ID SIZE MODIFIED
all-minilm:latest 1b226e2802db 45 MB 4 minutes ago
Now five small notes as a test corpus:
mkdir -p notes
cat > notes/vault.md <<'EOF'
# Vault Betrieb
## Entsiegeln
Nach einem Neustart ist Vault versiegelt. Drei von fünf Key-Shares eingeben:
`vault operator unseal` dreimal ausführen, Status mit `vault status` prüfen.
## Token erneuern
Lab-Token `sk-lab-master-1234` ist nur für das Lab. Eigene Tokens mit
`vault token renew` verlängern, TTL-Maximum beträgt 32 Tage.
EOF
cat > notes/k8s.md <<'EOF'
# Kubernetes Debugging
## CrashLoopBackOff
Ein Pod startet immer wieder neu. Erst `kubectl logs <pod> --previous`, dann
`kubectl describe pod <pod>` lesen. Häufig: falsches Secret oder OOMKilled.
## Pending Pods
Pod bleibt Pending: meist fehlen CPU/RAM-Ressourcen oder ein PVC ist nicht gebunden.
`kubectl get events --sort-by=.lastTimestamp` zeigt den Grund.
EOF
cat > notes/backup.md <<'EOF'
# Backup
## Zeitplan
Die Sicherung läuft jede Nacht um 02:00 Uhr per restic auf das NAS `nas01`.
Aufbewahrung: 7 tägliche, 4 wöchentliche, 12 monatliche Snapshots.
## Restore
Wiederherstellen mit `restic restore latest --target /tmp/restore`.
Vorher immer `restic snapshots` prüfen.
EOF
cat > notes/oncall.md <<'EOF'
# Rufbereitschaft
## Eskalation
Reagiert der Primary nicht innerhalb von 15 Minuten, eskaliert PagerDuty an
den Secondary. Nach 30 Minuten wird die Teamleitung angerufen.
## Übergabe
Übergabe jeden Montag 10:00 Uhr im Ops-Channel mit offenen Incidents.
EOF
cat > notes/python.md <<'EOF'
# Python Setup
## Neues Projekt
`uv init --package` legt ein Projekt mit src-Layout an, `uv add` fügt
Abhängigkeiten hinzu, `uv run` startet im Projekt-venv.
## Tests
Tests liegen unter `tests/` und laufen mit `uv run pytest -q`.
EOF
ls notesExpected:
backup.md k8s.md oncall.md python.md vault.md
Tip:
all-minilmis small (45 MB) and fast, but trained on English. For German texts, multilingual models are better – you will see the consequences in Drill 7.
Save your progress? Sign up for free
Drill 2: Chunking at headings
One chunk per ## section. The heading stays in the text
– it often carries most of the meaning.
cat > src/docsearch/chunking.py <<'EOF'
"""Markdown-Dateien in Abschnitte (Chunks) zerlegen."""
from dataclasses import dataclass
from pathlib import Path
@dataclass(frozen=True)
class Chunk:
source: str # z. B. "vault.md#Entsiegeln"
text: str
def split_long(text: str, max_chars: int) -> list[str]:
"""Fasst Absätze zusammen, bis max_chars erreicht ist."""
parts: list[str] = []
current = ""
for para in text.split("\n\n"):
if current and len(current) + len(para) > max_chars:
parts.append(current)
current = ""
current = f"{current}\n\n{para}".strip()
if current:
parts.append(current)
return parts
def split_markdown(path: Path, max_chars: int = 800) -> list[Chunk]:
"""Ein Chunk pro Überschrift; die Überschrift bleibt als Kontext im Text."""
chunks: list[Chunk] = []
section = path.stem
lines: list[str] = []
def flush() -> None:
body = "\n".join(lines).strip()
for part in split_long(body, max_chars) if body else []:
chunks.append(Chunk(f"{path.name}#{section}", f"{section}\n{part}"))
lines.clear()
for line in path.read_text(encoding="utf-8").splitlines():
if line.startswith("#"):
flush()
section = line.lstrip("#").strip()
else:
lines.append(line)
flush()
return chunks
def load_notes(folder: Path) -> list[Chunk]:
return [c for p in sorted(folder.rglob("*.md")) for c in split_markdown(p)]
EOF
uv run python -c "
from pathlib import Path
from docsearch.chunking import load_notes
chunks = load_notes(Path('notes'))
print(len(chunks), 'Chunks')
for c in chunks[:3]:
print(c.source, '|', c.text.splitlines()[1][:50])
"Expected:
10 Chunks
backup.md#Zeitplan | Die Sicherung läuft jede Nacht um 02:00 Uhr per re
backup.md#Restore | Wiederherstellen mit `restic restore latest --targ
k8s.md#CrashLoopBackOff | Ein Pod startet immer wieder neu. Erst `kubectl lo
Why 10 and not 15? The # title lines have no text of
their own and produce no chunk. source later becomes the
source citation. For long sections, split_long splits at
paragraphs instead of mid-sentence.
Save your progress? Sign up for free
Drill 3: Embeddings and cosine with numpy
cat > src/docsearch/vectors.py <<'EOF'
"""Embeddings über Ollama, Kosinus-Ähnlichkeit mit numpy."""
import os
import numpy as np
from openai import OpenAI
BASE_URL = os.environ.get("DOCSEARCH_BASE_URL", "http://127.0.0.1:11434/v1")
EMBED_MODEL = os.environ.get("DOCSEARCH_EMBED_MODEL", "all-minilm")
def make_client() -> OpenAI:
key = os.environ.get("DOCSEARCH_API_KEY", "ollama")
return OpenAI(base_url=BASE_URL, api_key=key, timeout=120, max_retries=1)
def embed(client: OpenAI, texts: list[str]) -> np.ndarray:
"""Liefert eine (n, dim)-Matrix mit Länge 1 je Zeile."""
resp = client.embeddings.create(model=EMBED_MODEL, input=texts)
vecs = np.array([d.embedding for d in resp.data], dtype=np.float32)
return vecs / np.linalg.norm(vecs, axis=1, keepdims=True)
def top_k(query: np.ndarray, matrix: np.ndarray, k: int) -> list[tuple[int, float]]:
"""Kosinus = Skalarprodukt, weil alle Vektoren normiert sind."""
scores = matrix @ query
best = np.argsort(scores)[::-1][:k]
return [(int(i), float(scores[i])) for i in best]
EOF
uv run python -c "
from pathlib import Path
from docsearch.chunking import load_notes
from docsearch.vectors import embed, make_client, top_k
client = make_client()
chunks = load_notes(Path('notes'))
m = embed(client, [c.text for c in chunks])
print(m.shape, m.dtype, round(float((m[0] ** 2).sum()), 3))
for q in ['Wie entsiegle ich Vault?', 'Mein Pod startet ständig neu']:
print(q)
for i, score in top_k(embed(client, [q])[0], m, 2):
print(f' {score:.2f} {chunks[i].source}')
"Expected (values may differ slightly):
(10, 384) float32 1.0
Wie entsiegle ich Vault?
0.59 vault.md#Entsiegeln
0.45 vault.md#Token erneuern
Mein Pod startet ständig neu
0.56 k8s.md#CrashLoopBackOff
0.47 oncall.md#Übergabe
Why normalize? Cosine is a·b / (|a|·|b|). If all vectors
have length 1, only a·b remains – and
matrix @ query computes all dot products in one go. One
batch call for all chunks is much faster than one call per chunk.
Save your progress? Sign up for free
Drill 4: A persistent index with qdrant
numpy is enough for a hundred chunks in RAM. For thousands of notes, persistence and filters, you use a vector DB. Here qdrant runs embedded, with no server at all.
cat > src/docsearch/store.py <<'EOF'
"""Persistenter Vektorindex mit qdrant im lokalen Modus (kein Server)."""
from dataclasses import dataclass
from pathlib import Path
import numpy as np
from qdrant_client import QdrantClient, models
from docsearch.chunking import Chunk
COLLECTION = "notes"
@dataclass
class Hit:
source: str
text: str
score: float
def open_store(path: Path) -> QdrantClient:
return QdrantClient(path=str(path))
def rebuild(store: QdrantClient, chunks: list[Chunk], vectors: np.ndarray) -> int:
if store.collection_exists(COLLECTION):
store.delete_collection(COLLECTION)
store.create_collection(
COLLECTION,
vectors_config=models.VectorParams(
size=vectors.shape[1], distance=models.Distance.COSINE
),
)
points = [
models.PointStruct(
id=i, vector=v.tolist(), payload={"source": c.source, "text": c.text}
)
for i, (c, v) in enumerate(zip(chunks, vectors, strict=True))
]
store.upsert(COLLECTION, points=points)
return store.count(COLLECTION).count
def search(store: QdrantClient, vector: np.ndarray, k: int = 3) -> list[Hit]:
res = store.query_points(COLLECTION, query=vector.tolist(), limit=k)
return [
Hit(p.payload["source"], p.payload["text"], p.score)
for p in res.points
if p.payload
]
EOF
uv run python -c "
from pathlib import Path
from docsearch.chunking import load_notes
from docsearch.store import open_store, rebuild, search
from docsearch.vectors import embed, make_client
client = make_client()
chunks = load_notes(Path('notes'))
store = open_store(Path('.docsearch'))
print('Punkte:', rebuild(store, chunks, embed(client, [c.text for c in chunks])))
for h in search(store, embed(client, ['Wann läuft die Sicherung?'])[0], k=2):
print(f'{h.score:.2f} {h.source}')
store.close()
"
ls .docsearchExpected:
Punkte: 10
0.32 oncall.md#Übergabe
0.32 oncall.md#Eskalation
collection meta.json
Why isn’t the backup note at the top? The question says “Sicherung”
(backup), and so does the note – but all-minilm understands
German only roughly. Low scores around 0.3 are a warning sign: that is
not a real hit. This is exactly why you build a threshold into Drill
5.
Warning: Local mode locks the folder for one process. Two simultaneous
docsearchcalls on the same.docsearch/fail – for multiple users, you start the qdrant server.
Save your progress? Sign up for free
Drill 5: RAG – answer only from the context, with source
cat > src/docsearch/rag.py <<'EOF'
"""Retrieval-Augmented Generation: Treffer als Kontext an das Chat-Modell."""
import os
from openai import OpenAI
from docsearch.store import Hit
CHAT_MODEL = os.environ.get("DOCSEARCH_CHAT_MODEL", "ministral-3:3b")
MIN_SCORE = 0.35 # darunter gilt ein Treffer als Rauschen
NO_ANSWER = "Dazu finde ich nichts in deinen Notizen."
SYSTEM = (
"Beantworte die Frage NUR mit dem Kontext, auf Deutsch, in max. 3 Sätzen. "
"Nenne nach jeder Aussage die Quelle in eckigen Klammern, z. B. "
f"[vault.md#Entsiegeln]. Steht es nicht im Kontext, antworte: {NO_ANSWER}"
)
def build_prompt(question: str, hits: list[Hit]) -> str:
context = "\n\n".join(f"[{h.source}]\n{h.text}" for h in hits)
return f"Kontext:\n{context}\n\nFrage: {question}"
def answer(client: OpenAI, question: str, hits: list[Hit]) -> str:
relevant = [h for h in hits if h.score >= MIN_SCORE]
if not relevant:
return NO_ANSWER # spart den LLM-Aufruf und verhindert Halluzination
resp = client.chat.completions.create(
model=CHAT_MODEL,
messages=[
{"role": "system", "content": SYSTEM},
{"role": "user", "content": build_prompt(question, relevant)},
],
temperature=0,
)
return resp.choices[0].message.content or ""
EOF
uv run ruff check srcExpected:
All checks passed!
Why temperature=0? With RAG you want faithfulness to the
context, not creativity. And the threshold: without
MIN_SCORE, the model would get only noise as context and
would still make something up. “I don’t know” is a good answer in
RAG.
Save your progress? Sign up for free
Drill 6: The CLI with four commands
cat > src/docsearch/cli.py <<'EOF'
"""docsearch – eigene Markdown-Notizen durchsuchen und befragen."""
import json
from pathlib import Path
from typing import Annotated
import typer
from docsearch.chunking import load_notes
from docsearch.rag import answer
from docsearch.store import open_store, rebuild, search
from docsearch.vectors import embed, make_client
app = typer.Typer(add_completion=False, no_args_is_help=True)
DbOpt = Annotated[Path, typer.Option("--db", envvar="DOCSEARCH_DB")]
K = Annotated[int, typer.Option("-k", min=1, max=20)]
@app.command()
def index(folder: Path, db: DbOpt = Path(".docsearch")) -> None:
"""Liest alle *.md unter FOLDER und baut den Index neu."""
chunks = load_notes(folder)
if not chunks:
typer.secho(f"Keine Markdown-Abschnitte in {folder}", fg="red", err=True)
raise typer.Exit(1)
vectors = embed(make_client(), [c.text for c in chunks])
store = open_store(db)
count = rebuild(store, chunks, vectors)
store.close()
typer.echo(f"{count} Chunks aus {folder} indiziert ({vectors.shape[1]} Dim.)")
@app.command("search")
def search_cmd(query: str, k: K = 3, db: DbOpt = Path(".docsearch")) -> None:
"""Zeigt die k ähnlichsten Abschnitte."""
store = open_store(db)
for hit in search(store, embed(make_client(), [query])[0], k):
first = hit.text.splitlines()[1][:50]
typer.echo(f"{hit.score:.2f} {hit.source:28} {first}")
store.close()
@app.command()
def ask(question: str, k: K = 3, db: DbOpt = Path(".docsearch")) -> None:
"""Beantwortet QUESTION nur aus den Notizen, mit Quellen."""
client = make_client()
store = open_store(db)
hits = search(store, embed(client, [question])[0], k)
store.close()
typer.echo(answer(client, question, hits))
@app.command("eval")
def eval_cmd(cases: Path, k: K = 3, db: DbOpt = Path(".docsearch")) -> None:
"""Misst Hit@1 und Hit@k: Ist die erwartete Datei unter den Treffern?"""
data = json.loads(cases.read_text(encoding="utf-8"))
client = make_client()
store = open_store(db)
hit1 = hitk = 0
for case in data:
hits = search(store, embed(client, [case["question"]])[0], k)
files = [h.source.split("#")[0] for h in hits]
hit1 += files[:1] == [case["source"]]
hitk += case["source"] in files
mark = "✔" if case["source"] in files else "✘"
typer.echo(f"{mark} {case['question'][:40]:40} -> {files[0] if files else '-'}")
store.close()
n = len(data)
typer.echo(f"Hit@1: {hit1 / n:.2f} Hit@{k}: {hitk / n:.2f} (n={n})")
EOF
cat > src/docsearch/__init__.py <<'EOF'
"""docsearch – eigene Markdown-Notizen durchsuchen und befragen."""
import openai
import typer
from docsearch.cli import app
def main() -> None:
try:
app()
except openai.APIConnectionError:
typer.secho("Fehler: Ollama nicht erreichbar", fg="red", err=True)
raise SystemExit(1) from None
EOF
cat > src/docsearch/__main__.py <<'EOF'
from docsearch import main
main()
EOF
rm -rf .docsearch
uv run docsearch index notes
uv run docsearch search "Pod startet ständig neu"
uv run docsearch ask "Wer wird nach 30 Minuten angerufen?"
uv run docsearch ask "Wann läuft die Sicherung?"Expected (text varies, abridged):
10 Chunks aus notes indiziert (384 Dim.)
0.59 k8s.md#CrashLoopBackOff Ein Pod startet immer wieder neu. Erst `kubectl lo
0.45 k8s.md#Pending Pods Pod bleibt Pending: meist fehlen CPU/RAM-Ressource
0.45 oncall.md#Übergabe Übergabe jeden Montag 10:00 Uhr im Ops-Channel mit
Nach 30 Minuten wird die **Teamleitung** (meist der Oncall-Leiter oder Ops Manager)
per Telefon oder Notruf kontaktiert [oncall.md#Eskalation].
Die genaue Rolle variiert je nach Organisation, ... [oncall.md#Übergabe].
Dazu finde ich nichts in deinen Notizen.
The backup question stays unanswered, even though the note exists:
all scores are below MIN_SCORE (Drill 4). Retrieval is the
problem, not the LLM. Why does main() in
__init__.py catch the connection error? Once, centrally,
instead of in each of the four commands – with exit code 1, as in
ask (Day 10).
Warning: Read the answer critically: “meist der Oncall-Leiter oder Ops Manager” (usually the on-call lead or ops manager) is in no note, and
[oncall.md#Übergabe]does not back up the second sentence. A source citation from the LLM is a claim, not proof – the Challenge at least checks whether cited sources were in the context at all.
Security:
docsearch search token -k 1finds the section containingsk-lab-master-1234. Everything you index sits in plain text in.docsearch/and ends up in the prompt. Secrets do not belong in notes you index.
Save your progress? Sign up for free
Drill 7: Evaluation – numbers instead of gut feeling
A small golden set: a question plus the file that contains the answer.
cat > eval.json <<'EOF'
[
{"question": "Wie entsiegle ich Vault?", "source": "vault.md"},
{"question": "Pod startet immer wieder neu", "source": "k8s.md"},
{"question": "Wann läuft die Sicherung?", "source": "backup.md"},
{"question": "Wer wird nach 30 Minuten angerufen?", "source": "oncall.md"},
{"question": "Wie starte ich Tests?", "source": "python.md"},
{"question": "Warum bleibt mein Pod Pending?", "source": "k8s.md"},
{"question": "Wie hole ich Dateien aus dem Backup zurück?", "source": "backup.md"},
{"question": "Wie lange gilt ein Vault-Token maximal?", "source": "vault.md"}
]
EOF
uv run docsearch eval eval.jsonExpected:
✔ Wie entsiegle ich Vault? -> vault.md
✔ Pod startet immer wieder neu -> k8s.md
✘ Wann läuft die Sicherung? -> oncall.md
✔ Wer wird nach 30 Minuten angerufen? -> oncall.md
✔ Wie starte ich Tests? -> python.md
✔ Warum bleibt mein Pod Pending? -> k8s.md
✔ Wie hole ich Dateien aus dem Backup zurü -> oncall.md
✔ Wie lange gilt ein Vault-Token maximal? -> vault.md
Hit@1: 0.75 Hit@3: 0.88 (n=8)
Why two numbers? Hit@1 says how often the best hit
is right; Hit@3 says whether the right file ends up in
the context at all – that is what matters for RAG. Notice:
oncall.md#Übergabe shows up near the top for many
questions. Short, generic chunks become “all-purpose hits” this way.
Levers you can now compare measurably: a multilingual embedding model
(DOCSEARCH_EMBED_MODEL), different chunking, hybrid search
with keywords.
Tip: Put
eval.jsoninto the repo too and rundocsearch evalin CI. If Hit@3 drops after a change, you know immediately.
Save your progress? Sign up for free
Drill 8: Tests without a model
Embeddings and chat are mocked; qdrant runs for real, but in
tmp_path with 2-dimensional vectors.
mkdir -p tests
cat > tests/test_docsearch.py <<'EOF'
from pathlib import Path
from types import SimpleNamespace as NS
from unittest.mock import MagicMock
import numpy as np
from docsearch.chunking import Chunk, split_markdown
from docsearch.rag import NO_ANSWER, answer
from docsearch.store import Hit, open_store, rebuild, search
from docsearch.vectors import embed
def test_split_markdown_one_chunk_per_section(tmp_path: Path) -> None:
note = tmp_path / "ops.md"
note.write_text("# Ops\n\n## Restart\nsystemctl restart x\n\n## Logs\njournalctl\n")
chunks = split_markdown(note)
assert [c.source for c in chunks] == ["ops.md#Restart", "ops.md#Logs"]
assert chunks[0].text == "Restart\nsystemctl restart x"
def test_embed_normalizes() -> None:
client = MagicMock()
client.embeddings.create.return_value = NS(data=[NS(embedding=[3.0, 4.0])])
vec = embed(client, ["x"])
assert np.allclose(vec, [[0.6, 0.8]])
def test_store_roundtrip(tmp_path: Path) -> None:
chunks = [Chunk("a.md#A", "alpha"), Chunk("b.md#B", "beta")]
store = open_store(tmp_path / "db")
assert rebuild(store, chunks, np.array([[1.0, 0.0], [0.0, 1.0]])) == 2
hits = search(store, np.array([0.1, 0.9]), k=1)
store.close()
assert hits[0].source == "b.md#B"
def test_answer_skips_llm_when_scores_low() -> None:
client = MagicMock()
assert answer(client, "?", [Hit("a.md#A", "x", 0.1)]) == NO_ANSWER
client.chat.completions.create.assert_not_called()
def test_answer_sends_sources_in_prompt() -> None:
client = MagicMock()
client.chat.completions.create.return_value = NS(
choices=[NS(message=NS(content="ok [a.md#A]"))]
)
assert answer(client, "Frage?", [Hit("a.md#A", "Inhalt", 0.9)]) == "ok [a.md#A]"
prompt = client.chat.completions.create.call_args.kwargs["messages"][1]["content"]
assert "[a.md#A]" in prompt and "Frage?" in prompt
EOF
uv run pytest -q
uv run ruff check . && uv run --with mypy mypy --strict srcExpected:
..... [100%]
5 passed in 0.44s
All checks passed!
Success: no issues found in 7 source files
Why assert_not_called()? The most important path is the
one in which the LLM is not asked. Only a test proves
it to you.
Save your progress? Sign up for free
Drill 9: The tool is done
uv tool install --editable .
export DOCSEARCH_DB=~/.cache/docsearch
docsearch index ~/py-lab/tag11-docsearch/notes
cd ~
docsearch ask "Wie lange gilt ein Vault-Token maximal?"
docsearch search "restic" -k 1Expected (text varies):
Installed 1 executable: docsearch
10 Chunks aus /Users/thomas/py-lab/tag11-docsearch/notes indiziert (384 Dim.)
Ein Vault-Token gilt maximal **32 Tage** [vault.md#Token erneuern].
Dazu finde ich nichts in deinen Notizen.
0.32 backup.md#Restore Wiederherstellen mit `restic restore latest --targ
The appended “Dazu finde ich nichts” (“I can’t find anything on
that”) sentence shows: a 3B model follows the system prompt literally,
but not smartly. Why DOCSEARCH_DB? With a fixed index path,
docsearch works from any directory. Hook index
up to your notes folder as a cron job or Git hook, and the index stays
current.
Save your progress? Sign up for free
Checkpoint
cd ~/py-lab/tag11-docsearch
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
DB=/tmp/ds-check
check "Embedding-Modell da" "ollama list | grep -q all-minilm"
check "Tests grün" "uv run pytest -q"
check "ruff sauber" "uv run ruff check ."
check "mypy --strict sauber" "uv run --with mypy mypy --strict src"
check "Index gebaut" "uv run docsearch index notes --db $DB | grep -q '10 Chunks'"
check "Hit@3 >= 0.8" \
"uv run docsearch eval eval.json --db $DB | grep -qE 'Hit@3: (0\.[89]|1\.00)'"
check "Antwort mit Quelle" \
"uv run docsearch ask 'Wann ist die Übergabe?' --db $DB | grep -q '\['"
rm -rf "$DB"Expected:
✔ Embedding-Modell da
✔ Tests grün
✔ ruff sauber
✔ mypy --strict sauber
✔ Index gebaut
✔ Hit@3 >= 0.8
✔ Antwort mit Quelle
Challenge
The LLM sometimes invents sources. Add
unknown_citations(text, hits), which returns every cited
[datei.md#Abschnitt] (file and section) that was
not among the relevant hits, and have
docsearch ask print a warning on stderr for each one.
Hint: A regular expression on
\[...\.md#...\] plus a set difference
(set - set) is enough.
Solution
In cli.py, change the import to
from docsearch.rag import answer, unknown_citations and
make ask end like this:
text = answer(client, question, hits)
typer.echo(text)
for source in unknown_citations(text, hits):
typer.secho(f"Warnung: Quelle {source} nicht im Kontext", fg="red", err=True)Then add the function and the test:
cat >> src/docsearch/rag.py <<'EOF'
CITATION = re.compile(r"\[([^\[\]]+\.md#[^\[\]]+)\]")
def unknown_citations(text: str, hits: list[Hit]) -> list[str]:
"""Zitierte Quellen, die gar nicht im Kontext standen."""
allowed = {h.source for h in hits if h.score >= MIN_SCORE}
return sorted(set(CITATION.findall(text)) - allowed)
EOF
perl -pi -e 's/^import os$/import os\nimport re/' src/docsearch/rag.py
cat >> tests/test_docsearch.py <<'EOF'
def test_unknown_citations_flags_invented_sources() -> None:
from docsearch.rag import unknown_citations
hits = [Hit("a.md#A", "x", 0.9), Hit("b.md#B", "y", 0.1)]
text = "Eins [a.md#A]. Zwei [b.md#B]. Drei [c.md#C]."
assert unknown_citations(text, hits) == ["b.md#B", "c.md#C"]
EOF
uv run pytest -qExpected:
...... [100%]
6 passed in 0.47s
Why does b.md#B count as unknown even though it was a
hit? With a score of 0.1, it was below MIN_SCORE and was
never in the prompt.
Save your progress? Sign up for free
Enterprise Note
- Permissions belong in the index: every chunk carries ACL metadata, and the search filters by user – otherwise RAG answers questions from documents the asker was never allowed to see.
- Scan for and remove secrets and personal data before indexing (e.g. with gitleaks rules); the vector index is a copy of all the data.
- Run qdrant as a server (or pgvector) with backup, TLS and an API key; local mode is only for single-user tools.
- Version the embedding model and the chunking: switching models requires re-indexing; vectors from different models are not comparable.
- Maintain the golden set and watch Hit@k, answer quality (LLM-as-judge or spot checks) and the “I don’t know” rate as metrics.