Skip to main content

Lab — Local RAG, Entity Graph, and a Small Agent with Ollama and SQLite

Lab

Service: Sovereign & Private AI, GenAI Proof of Concept (PoC) · Typical shape: 1-hour hands-on lab, or the first afternoon of a RAG PoC; the hands-on companion to RAG Systems

By the end of this lab you will have a working question-answering system over a set of policy documents, running entirely on your laptop. Nothing leaves the machine: embeddings, retrieval, and generation all run locally. You will read the code that does each step, ask a question in English and in Hungarian, and measure retrieval quality with a golden set. In the second half you extract entities and relations into a small graph with the same chat model, and run a three-step agent that uses the index and the graph as its knowledge base.

The code lives in examples/local-rag/ of the DevTraining repository. Read Local AI Basics first for hardware and runtime vocabulary. The Inference Engineering page explains why the numbers below behave the way they do.


Goal and what you build​

You build a pipeline with two phases:

  • Ingest (offline): split Markdown policies into heading-aware chunks, embed each chunk, and store the text in an FTS5 table and the vector in a sqlite-vec table. Both live in one SQLite file.
  • Query (online): embed the question, run a dense search and a BM25 search, fuse the two rankings with Reciprocal Rank Fusion, pass the top passages to a small chat model with numbered sources, and stream a cited answer.

Repository layout:

FileResponsibility
rag.pyConfig, schema, embedding calls, dense and BM25 search, RRF, retrieve()
ingest.pyChunking and batched indexing
query.pyRetrieval, grounded prompt, streamed answer
eval.pyhit@k and MRR per retrieval mode; optional answer check
graph.pySchema-constrained entity and relation extraction, entity resolution, graph retrieval
agent.pyThree-step agent with JSON actions over the index and the graph
golden.jsonl8 questions (EN and HU) with expected document and fact
data/Six sample policy documents (travel, security, remote work, AI usage, onboarding, leave)

Prerequisites​

ItemRequirementNotes
HardwareAny laptop or workstation that runs Ollama; no GPU requiredThe captured outputs come from an 8-vCPU, CPU-only sandbox
DiskAbout 7 GB freeTwo models: embeddinggemma-2:270m (303 MB), gemma4:e4b (6.6 GB)
Python3.11 or newerChecked with 3.13
uvLatest releaseCreates the virtual environment from pyproject.toml. Install from docs.astral.sh/uv
Ollama0.40.8 or newer (as of 2026-10)Native install, or docker compose up -d from the lab folder
Expect slow generations on CPU

Answers took 92 to 190 seconds, the graph build 94 minutes, and an agent run 2.5 to 8 minutes on the CPU-only sandbox used for this page. The host was close to 100 % busy with other work, so these are worst-case numbers. On a GPU or Apple Silicon every LLM step is much faster; the retrieval results do not change.


Step-by-step​

Step 1 — Install​

cd examples/local-rag
uv --version
make up # optional: docker compose up -d, if you do not run Ollama natively
ollama --version # native install; skip if you use docker compose

Dependencies are two packages, pinned in pyproject.toml:

dependencies = [
"openai==3.32.0",
"sqlite-vec==0.1.9",
]

openai is the client for any OpenAI-compatible server. sqlite-vec adds the vec0 virtual table for vector search to SQLite. No vector database server is needed.

Step 2 — Pull the models​

make models

This runs ollama pull embeddinggemma-2:270m and ollama pull gemma4:e4b. Check that both are present:

ollama list
ModelRoleSizeVersion (as of 2026-10)
embeddinggemma-2:270mText embeddings, 768 dimensions (Matryoshka: 512, 256, 128), 8K context303 MBEmbeddingGemma 2, text-only 270M tag, Apache-2.0
gemma4:e4bChat model for answers, extraction and the agent6.6 GBGemma 4 E4B

Why these two: EmbeddingGemma 2 is the newest small multilingual embedder in the Ollama library (as of 2026-10), and it is small enough to re-embed the corpus in minutes on a CPU. Gemma 4 E4B is the small tier of Gemma 4, which Google positions for agentic workflows and tool use. It handles the schema-constrained JSON the graph and agent steps need. On a machine with 8 GB or less, gemma4:e2b (4.6 GB) is the fallback: set CHAT_MODEL=gemma4:e2b. For the larger open-weight models and other embedders, see Open Model Landscape.

Step 3 — Chunk and ingest​

Run:

make ingest

Chunking happens in two stages. First, sections() splits each file at Markdown headings and keeps the heading path, for example Travel and Expenses Policy > Accommodation. Then split() packs paragraphs into chunks of at most 1200 characters and carries the last 200 characters into the next chunk:

examples/local-rag/ingest.py (excerpt)
def split(text: str, max_chars: int, overlap: int) -> list[str]:
"""Pack paragraphs up to max_chars; carry the tail of the previous chunk as overlap."""
chunks: list[str] = []
current = ""
for para in re.split(r"\n\s*\n", text):
if current and len(current) + len(para) > max_chars:
chunks.append(current)
current = current[-overlap:].split(" ", 1)[-1] + "\n\n" if overlap else ""
current += para + "\n\n"
if current.strip():
chunks.append(current.strip())
return [c.strip() for c in chunks]


def chunk_docs(folder: Path, max_chars: int, overlap: int) -> list[tuple[str, str, str]]:
return [
(md.name, heading, f"{heading}\n\n{piece}")
for md in sorted(folder.glob("*.md"))
for heading, body in sections(md.read_text(encoding="utf-8"))
for piece in split(body, max_chars, overlap)
]

Two decisions matter here. The heading is prepended to every chunk, so a passage about "Per diem" still says which policy and section it belongs to. Splitting on paragraphs rather than fixed character windows keeps each rule intact. Chunks are embedded in batches of 16 and each row is inserted into both tables:

examples/local-rag/ingest.py (excerpt)
vectors = embed_docs([(section.split(" > ")[0], text) for _, section, text in batch])
for offset, ((doc, section, text), vec) in enumerate(zip(batch, vectors, strict=True)):
cid = start + offset + 1
db.execute("INSERT INTO chunks (id, doc, section, text) VALUES (?, ?, ?, ?)", (cid, doc, section, text))
db.execute("INSERT INTO chunks_vec (rowid, embedding) VALUES (?, ?)", (cid, sqlite_vec.serialize_float32(vec)))

The schema in rag.py is one table for the text, one FTS5 table over it (external content, unicode61 tokenizer with diacritics removed), and one vec0 table with cosine distance:

CREATE TABLE chunks (id INTEGER PRIMARY KEY, doc TEXT, section TEXT, text TEXT);
CREATE VIRTUAL TABLE chunks_fts USING fts5(
section, text, content='chunks', content_rowid='id',
tokenize='unicode61 remove_diacritics 2');
CREATE VIRTUAL TABLE chunks_vec USING vec0(
embedding float[768] distance_metric=cosine);

Captured output:

embedded 16/34 chunks
embedded 32/34 chunks
embedded 34/34 chunks
indexed 34 chunks from 6 docs with embeddinggemma-2:270m in 166.9s -> .../examples/local-rag/.rag/index.db
Our sandbox was slow; time yours

The 166.9 s above was measured on a shared 8-vCPU sandbox whose host CPU was close to 100 % busy with other work. The model itself is 303 MB and 270M parameters, so on an idle laptop expect a fraction of that. Before you blame the index code, time one curl call against /v1/embeddings and check uptime for CPU contention.

Step 4 — Ask a question in English​

make ask

make ask runs query.py with "What is the hotel limit in Munich and how fast do I need to submit my expense claim?". EmbeddingGemma 2 is an asymmetric embedder: queries and documents get different prompts, as listed on its model card. Queries use a task prefix; documents use title: … | text: …, and the lab passes the policy title. Wrong or missing prompts are a silent quality loss, so rag.py picks them per model family:

examples/local-rag/rag.py (excerpt)
if EMBED_MODEL.startswith("embeddinggemma"):
QUERY_PROMPT, DOC_PROMPT = "task: question answering | query: {q}", "title: {title} | text: {text}"
elif EMBED_MODEL.startswith("qwen3-embedding"):
QUERY_PROMPT, DOC_PROMPT = f"Instruct: {QUERY_TASK}\nQuery:{{q}}", "{text}"
else:
QUERY_PROMPT, DOC_PROMPT = "{q}", "{text}"

def embed_query(question: str) -> list[float]:
return embed([QUERY_PROMPT.format(q=question)])[0]

Retrieval runs two searches over a pool of at least 20 candidates. The dense leg is a KNN query over chunks_vec. The BM25 leg is an FTS5 query. The FTS5 query turns each word of at least three letters into a prefix on its first six letters, which gives crude stemming for agglutinative Hungarian:

examples/local-rag/rag.py (excerpt)
def fts_query(question: str) -> str:
# Prefix match on the first 6 letters: crude stemming that helps agglutinative Hungarian.
terms = {w.lower()[:6] for w in re.findall(r"\w+", question) if len(w) >= 3}
return " OR ".join(f'"{t}"*' for t in sorted(terms))

The two rankings are fused with RRF, which uses only rank positions:

examples/local-rag/rag.py (excerpt)
RRF_K = 60

def rrf(*rankings: list[tuple[int, float]]) -> list[tuple[int, float]]:
scores: dict[int, float] = {}
for ranking in rankings:
for rank, (cid, _) in enumerate(ranking, start=1):
scores[cid] = scores.get(cid, 0.0) + 1 / (RRF_K + rank)
return sorted(scores.items(), key=lambda kv: kv[1], reverse=True)

# inside retrieve(), mode="hybrid":
ranked = rrf(dense_search(db, question, pool), bm25_search(db, question, pool))[:k]

The top five passages go into a grounded prompt. Each passage gets a number and its document name, and the system prompt forbids answers outside the passages:

examples/local-rag/query.py (excerpt)
SYSTEM = (
"You answer questions about Kestrel Systems internal policies using ONLY the numbered context passages. "
"Cite every claim with the passage number in square brackets, e.g. [1] or [2][3]. "
"If the context does not contain the answer, say that the policies do not cover it. "
"Answer in the same language as the question, in at most five sentences."
)

def build_messages(question: str, hits: list[Hit]) -> list[dict[str, str]]:
context = "\n\n".join(f"[{n}] ({h.doc})\n{h.text}" for n, h in enumerate(hits, start=1))
return [
{"role": "system", "content": SYSTEM},
{"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
]

The generation call uses temperature=0.1, reasoning_effort="none", and stream=True. The reasoning_effort="none" setting asks the server not to emit a thinking section, so the streamed text is the answer.

Captured output:

Q: What is the hotel limit in Munich and how fast do I need to submit my expense claim?

A: The hotel cap in Munich is EUR 190 per night [1]. You must submit expense claims in the finance app within 30 days of returning, and you must include a photo of every receipt [2].

Sources:
[1] travel-expenses.md § Travel and Expenses Policy > Accommodation (score 0.0325)
[2] travel-expenses.md § Travel and Expenses Policy > Submitting expenses (score 0.0325)
[3] travel-expenses.md § Travel and Expenses Policy > Booking travel (score 0.0315)
[4] security-incidents.md § Security Incident Reporting > How to report (score 0.0310)
[5] travel-expenses.md § Travel and Expenses Policy > Per diem (score 0.0303)

retrieval 4.40s · generation 189.98s · gemma4:e4b · mode=hybrid

Check the answer against data/travel-expenses.md: EUR 190 for Munich, 30 days after returning, and the receipt photo are all in the source. Both citations point at the right section. Passage 4, from the incident policy, is retrieval noise that the model ignored. The score column is the RRF score, so it is small by design; compare the ranks, not the magnitudes.

Step 5 — Ask a question in Hungarian​

make ask-hu

The question is "Hány nap betegszabadság jár évente, és meddig kell szólni a vezetőnek?" (how many sick-leave days per year, and by when must I tell my manager?). The prompt, the index, and the model are the same as in English. The system prompt tells the model to answer in the question's language:

Q: Hány nap betegszabadság jár évente, és meddig kell szólni a vezetőnek?

A: Évente 15 munkanap a betegszabadság [1]. Betegség esetén reggel 9 óráig kell értesíteni a vezetőt és a csapatot [1].

Sources:
[1] szabadsag.md § Szabadság és távollét szabályzat > Betegszabadság és táppénz (score 0.0328)
[2] szabadsag.md § Szabadság és távollét szabályzat > Szabadság igénylése (score 0.0320)
[3] szabadsag.md § Szabadság és távollét szabályzat > Éves szabadság (score 0.0318)
[4] szabadsag.md § Szabadság és távollét szabályzat > Egyéb távollétek (score 0.0315)
[5] szabadsag.md § Szabadság és távollét szabályzat (score 0.0154)

retrieval 4.10s · generation 91.55s · gemma4:e4b · mode=hybrid

The answer is correct: 15 working days of sick leave, and the manager and the team are told by 9:00 on the day. All five passages come from the Hungarian leave policy; only the first is needed.

Step 6 — Measure retrieval​

make eval

eval.py runs the eight questions in golden.jsonl through each retrieval mode. A question counts as a hit if the expected document appears in the top k. MRR is the mean reciprocal rank of the first correct document.

mode hit@3 MRR s/query
bm25 0.88 0.88 0.00 misses: ['How many paid volunteering days do employees get?']
dense 1.00 0.94 1.19
hybrid 0.88 0.81 5.26 misses: ['How many paid volunteering days do employees get?']

The per-query times include CPU contention on the sandbox; the hit rates do not depend on it. make eval-answers also generates and checks each answer for an expected fact and a citation. We did not run it for this page: one generation took 92 to 190 s on the sandbox, so the full pass would take 15 to 25 minutes there.

Read these numbers carefully. Eight questions is one miss per 0.125 of hit rate. This set is too small to rank the modes with confidence. What it does show is a concrete failure mode, analysed in the next section.

Step 7 — Extract an entity graph​

make graph # one schema-constrained extraction call per chunk
make graph-query # linked entities, one-hop facts and the chunks they come from

Vector search finds passages that look like the question. It does not know that the team lead who approves production access is the same role across documents, or that "report it as a security incident" in the AI policy points at the incident procedure in another file. graph.py adds that structure with the same local chat model. It is the minimal version of Graph RAG with Entity Extraction, section 2: schema first, provenance on every edge, and resolution by normalised name.

Each chunk gets one call with a JSON schema. The entity and relation types are enums, so the decoder cannot invent a type, and maxItems bounds the output length:

examples/local-rag/graph.py (excerpt)
ENTITY_TYPES = ["role", "team", "system", "channel", "policy", "document", "course", "location"]
RELATIONS = ["owns", "approves", "requires", "reports_to", "handles", "uses", "applies_to", "located_in"]

resp = client.chat.completions.create(
model=CHAT_MODEL,
temperature=0,
reasoning_effort="none",
messages=[{"role": "user", "content": PROMPT.format(doc=doc, text=text)}],
response_format={"type": "json_schema", "json_schema": {"name": "graph", "schema": SCHEMA}},
)

Three details make the graph usable:

  • Provenance. Every mention and every relation row stores the chunk_id it was extracted from. A fact in the graph always leads back to a passage the agent can cite.
  • Resolution by key. key() lowercases, drops articles and possessives, and turns cto_office or ai-assistant into cto office and ai assistant, so "your manager" in one policy and "manager" in another become one node. This is the cheapest resolution step; section 2.5 of the graph page covers embedding-based merging for when it is not enough.
  • A cache keyed by model, prompt, schema and input. Extraction is the expensive step. Re-running make graph after a crash, or after a change to the resolution code, costs nothing; changing the prompt or the schema re-extracts, as it should.

Captured output:

extracted 1/34 chunks (207s)
extracted 2/34 chunks (1225s)
...
extracted 34/34 chunks (5502s)
graph: 94 entities, 74 relations, 132 mentions from 34 chunks with gemma4:e4b in 5616s
Constrained decoding is not free on a CPU

The build above took 94 minutes for 34 chunks on the contended CPU sandbox. Decoding under the JSON schema ran at 0.3 to 0.5 tokens per second, against 0.6 to 2.2 tokens per second without a schema. These were measured at different times on a busy host, so treat the ratio as indicative. llama.cpp checks candidate tokens against the grammar on every step, and Gemma 4 has a 262,144-token vocabulary (config). Dropping the schema is not the fix: unconstrained, the same model pretty-printed its JSON and spent 392 tokens on one chunk, against 84 to 227 with the schema. On a GPU, or with vLLM's batched structured outputs, extraction is a batch job; measure with and without the schema on your hardware (Graph RAG, section 8).

At query time, graph_search() links the question to entities (exact name match first, then the nearest entity names by embedding above a threshold), takes every relation one hop away from them, and ranks chunks by how many seed entities they mention and how many of those relations they ground:

$ make graph-query # "What is the People Team responsible for?"
linked: people team, teams, kestrel systems

facts:
(people team) -[owns]-> (remote and hybrid work policy) chunk 12
(kestrel systems) -[uses]-> (remote and hybrid work policy) chunk 13
(teams) -[located_in]-> (budapest office) chunk 13
(remote and hybrid work policy) -[applies_to]-> (people team) chunk 16
(people team) -[approves]-> (working from outside eu) chunk 16
(people team) -[owns]-> (szabadság és távollét szabályzat) chunk 24
(szabadság és távollét szabályzat) -[applies_to]-> (kestrel systems) chunk 24
(travel and expenses policy) -[applies_to]-> (kestrel systems) chunk 29

chunks:
szabadsag.md § Szabadság és távollét szabályzat (score 3.0)
remote-work.md § Remote and Hybrid Work Policy > Working model (score 3.0)
remote-work.md § Remote and Hybrid Work Policy > Working from abroad (score 2.0)
remote-work.md § Remote and Hybrid Work Policy (score 1.5)
travel-expenses.md § Travel and Expenses Policy (score 1.5)

This is an entity-centric question, the kind where a graph helps. The People Team owns two policies in two languages, and the one decision it approves (working from outside the EU) sits in a section whose text never says "responsible". The graph ranks that section third. For comparison, retrieve() in hybrid mode returns the two policy headers, then three sections from onboarding.md and security-incidents.md, and not Working from abroad.

Read the facts critically as well:

  • The model mislabels edges. For "Who approves production access?" the graph holds (production access) -[requires]-> (team lead), while the source says the team lead approves it. Keep provenance so a person can check an edge against its chunk, and measure extraction precision on a hand-labelled sample before trusting the graph (graph page, section 7).
  • Generic entities are noise. The model extracted team and code as entities. Before the "longest match wins" rule in link(), the generic team matched inside "people team" and pulled in onboarding chunks. Fix noise with rules in code, not by re-prompting.
  • Relation direction blurs. (remote and hybrid work policy) -[applies_to]-> (people team) restates the owner relation the wrong way round. Small models blur closely related relation types; keep the enum short.

Step 8 — A small agent with the index and the graph as its knowledge base​

make agent # multi-document question
make agent-unknown # question the policies do not cover

A small model is not a reliable planner. agent.py follows the pattern from RAG as a Knowledge Base for Small Agents: code owns the loop, the model picks one action per turn from a closed set, and every claim must cite a source the code numbered.

  • Pre-retrieval. The top three hybrid hits are in the prompt before the first turn, so an easy question is answered in one step.
  • Three actions, one JSON schema. search_policies, lookup_entity and final. On the last step the enum shrinks to final, so the budget is enforced by the decoder, not by a prompt.
  • Citation check in code. A final answer whose cited numbers are not shown sources is rejected and sent back once. NOT_FOUND is a valid answer.
  • Stable prompt prefix. Sources are only ever appended, and the step counter goes in a trailing message rather than the system prompt. The instructions and the earlier sources therefore stay a prefix the server can reuse from its KV cache; everything after the newest source is recomputed. See Inference Engineering on prefix caching.
examples/local-rag/agent.py (excerpt)
for step in range(1, MAX_STEPS + 1):
last = step == MAX_STEPS
tools = ["final"] if last else ["search_policies", "lookup_entity", "final"]
system = SYSTEM.format(not_found=NOT_FOUND, sources=sources.render())
note = "Last step: return final." if last else f"Steps left: {MAX_STEPS - step + 1}. Return one action."
raw = client.chat.completions.create(
model=CHAT_MODEL, temperature=0, reasoning_effort="none",
messages=[{"role": "system", "content": system}, *transcript, {"role": "user", "content": note}],
response_format={"type": "json_schema", "json_schema": {"name": "action", "schema": action_schema(tools)}},
).choices[0].message.content

Captured output for "I pasted an API key into a public chatbot. What do I have to do, and how fast?":

$ make agent
Q: I pasted an API key into a public chatbot. What do I have to do, and how fast? (gemma4:e4b, max 3 steps)
step 1: search_policies('API key exposed in AI tool incident response') [115s]
step 2: search_policies('secrets exposed in AI tool incident response') [156s]
step 3: final('If you pasted an API key into any AI tool, you must rotate t') [295s]
{
"answer": "If you pasted an API key into any AI tool, you must rotate the secret immediately and report it as a security incident [1]. Pasting secrets into any AI tool is a violation that could be considered a security incident, such as credentials pasted into a public chat [5]. Depending on the severity, a confirmed breach of client or personal data (which exposing an API key could be) requires an Incident commander to be assigned within 30 minutes [4].",
"cited": [1, 4, 5],
"steps": 3,
"sources": {
"1": "ai-usage.md § Generative AI Usage Policy > Data classification rules",
"4": "security-incidents.md § Security Incident Reporting > Severity levels",
"5": "security-incidents.md § Security Incident Reporting > What counts as an incident"
}
}

Grade it against data/ before you accept it. The first part is right: rotate the key and report it as a security incident, from the AI policy. The second hop is only half right. The agent reached the incident policy, but the answer to "how fast" is in How to report: within 15 minutes. The model never saw that section. It filled the gap with the SEV1 target of 30 minutes, which is about assigning an incident commander, and it speculated that the leak "could be" a confirmed breach. Every sentence has a valid citation, so the citation check passed. Citations prove where a sentence came from, not that the answer is complete.

The run before this one was worse. Without the two tool-description lines about not repeating searches and following a pointer to another procedure, the agent ran two near-identical searches, stayed inside the AI policy, and answered that the policy "does not specify a required timeframe". Adding the lines got it into the right document. Neither run used lookup_entity. Small models act on what is already in front of them, so deciding when to expand belongs in code; Exercise 12 asks you to do that and measure it. Note also what we did not do: the tool hint uses a finance example, not the incident wording, because a prompt tuned on the test question tells you nothing about the next one.

Captured output for "What is the dress code for client meetings?":

$ make agent-unknown
Q: What is the dress code for client meetings? (gemma4:e4b, max 3 steps)
step 1: search_policies('dress code client meetings policy') [145s]
step 2: search_policies('client meeting attire policy') [280s]
step 3: final('NOT_FOUND') [374s]
{
"answer": "NOT_FOUND",
"cited": [],
"steps": 3
}

This is the behaviour you want from a small agent: two searches, nothing relevant, and an explicit NOT_FOUND instead of a plausible dress code. The step budget is what ends the run: on the last step the schema only allows final. Wall time was 2.5 to 6 minutes per question on the contended sandbox, nearly all of it model time.


How it works​

Why hybrid​

Dense and sparse retrieval fail on different queries:

Query typeDense (embeddings)Sparse (BM25)
Paraphrase: "fast sick-day notice" vs "report illness by 9 am"StrongWeak, no shared words
Exact token: policy ID, error code, product SKU, person's nameWeak, the vector smooths over rare stringsStrong
Cross-lingual: English question, Hungarian documentWorks if the model is multilingualFails, no shared tokens

Hybrid retrieval is the default because production corpora contain all three kinds of query. The lab's eval shows the cost of the combination. On this set, dense retrieval alone is best. The one miss is the English question about volunteering days, whose answer sits in a Hungarian document:

LegWhat it returned for that question
Dense, rank of szabadsag.md1 (dense ranks the Hungarian leave policy first, and three of its top four chunks come from it)
BM25, rank of szabadsag.mdNot in the top 20. It matches English words such as "days", "employees", and "paid" against English documents
RRF top 3No chunk from szabadsag.md, so the hit is lost

The BM25 leg pushes English documents into the top ranks. RRF gives every rank the same weight, so the dense leg's correct answer falls out of the top 3. The fix is not to drop BM25. Weight the legs, add a reranker, or grow the golden set so the decision rests on data. Exercises 4 and 5 below do this.

Why RRF​

Dense scores are cosine similarities in roughly [0, 1]. BM25 scores are unbounded and corpus-dependent. Adding them directly needs a normalisation that the data rarely supports. RRF uses only the rank of each document in each list:

score(d) = Σ over rankers r of 1 / (k + rank_r(d)), k = 60

Cormack, Clarke and Büttcher (SIGIR 2009) used k = 60 in their experiments and reported that the fusion outperformed the individual rankers, so the lab uses that value without tuning. A document ranked first in one list and absent from the other scores 1/61. A document ranked second in both scores 2/62, which beats it. That is the behaviour you want: agreement between legs is rewarded without any score calibration.

Why small models​

  • The task is narrow. The model reads a few passages and quotes them. A 4B-effective model with numbered citations is enough for that. Multi-step reasoning across documents is where it slips, as the agent in Step 8 shows; that needs a larger model or more control in code, and the eval would show it.
  • Embedding size is cheap. embeddinggemma-2:270m produces 768-dimension vectors from a 303 MB model. With Matryoshka truncation you can store 256 dimensions instead, which the model card reports as close to lossless; set EMBED_DIM=256 and re-ingest to try it. The retrieval step takes about one second per query.
  • Generation is the expensive step. On the contended sandbox CPU, gemma4:e4b needed 92 to 190 s per answer and 94 minutes to extract the graph. The same code on a GPU is much faster. Measure on your target hardware before you promise a latency.
  • Trade-off: smaller models make more mistakes in Hungarian and in long context. The eval and the answer check are how you find out, so keep them.

Exercises​

Basic​

  1. Predict, then measure the k effect. Run python eval.py -k 1 and python eval.py -k 5 before running them. Which mode gains most from a larger k, and why does BM25 gain less?
  2. Change the chunk size. Re-ingest with python ingest.py --max-chars 400 --overlap 50, then make eval. Record the chunk count from the ingest output and the hit@3 and MRR changes. Chunks smaller than a policy section lose the section's context; check whether that shows up in the misses.
  3. Add a golden question. Write a question about the per diem in Hungarian or English, add one line to golden.jsonl with doc and expect (a fact that appears verbatim in the source), and run make eval. Confirm the fact is in the source before you blame retrieval.
  4. Test refusal. Ask "Can I expense a gym membership?" with make ask. The correct answer is that the policies do not cover it. If the model invents a rule, tighten the system prompt and rerun.

Advanced​

  1. Weight the RRF legs. Change rrf() to multiply each leg by a weight (for example 1.0 dense, 0.5 BM25) and rerun make eval. Do not decide on the 8-question set alone: extend golden.jsonl to at least 50 questions first, covering both languages, or the result will be noise.

  2. Swap the embedding model. Pull another embedding model, set EMBED_MODEL and EMBED_DIM to match its output size, run make ingest, then make eval. The index must be rebuilt: query and document vectors have to come from the same model.

  3. Add a reranker. Retrieve 20 candidates instead of 5, score each (question, passage) pair with a cross-encoder such as Qwen3-Reranker-0.6B, and keep the top k. Measure hit@3 and latency; the reranker's cost grows with the candidate count.

  4. Serve with vLLM or llama.cpp. Point the lab at another OpenAI-compatible server without changing the code:

    OPENAI_BASE_URL=http://gpu-host:8000/v1 OPENAI_API_KEY=local \
    EMBED_MODEL=Qwen/Qwen3-Embedding-0.6B CHAT_MODEL=<served-chat-model> \
    python query.py "What is the per diem for an international business trip?"

    Both the embedding and the chat model must be served by that endpoint. Check that its /v1/embeddings accepts the dimensions field and returns the size you set in EMBED_DIM. For llama-server, start with --embeddings for the embedding model.

  5. Tune the Hungarian stemming. The FTS5 query uses the first six letters of each word. Try four, five, and full words, and measure hit@3 on the Hungarian questions only. Explain the result in terms of Hungarian suffixes.

  6. Measure extraction precision. Pick 10 relations from .rag/index.db at random (SELECT … FROM relations ORDER BY random() LIMIT 10), open each source chunk, and mark every edge correct, wrong type, or wrong direction. Then change one thing (the relation enum, the prompt, or CHAT_MODEL=gemma4:e2b), run make graph, and re-grade the same sample. The cache re-extracts only when the model, prompt or schema changes.

  7. Merge entities by embedding. List near-duplicate entity names (teams and team, manager and cost centre manager) by cosine similarity over entities_vec. Decide which pairs should merge and which must not; a cost-centre manager is not every manager. Write the rule in key() or as an alias table, and justify it with the precision sample from Exercise 10.

  8. Make the agent find the 15-minute rule. Add the API-key question to a small agent golden set with the expected fact 15. Try, one at a time: (a) after each search, run graph_search() on the entities named in the new passages and add its top chunk; (b) k=5 instead of 3 for searches; (c) a larger model on the same code, for example CHAT_MODEL=gpt-oss:20b or qwen3.8:27b on a GPU host. Record the answer, the steps and the time for each. The code change in (a) is the one that does not depend on model luck.


Troubleshooting​

SymptomCauseFix
ModuleNotFoundError: No module named 'rag'python -I script.py removes the script's folder from sys.pathRun make ingest / python ingest.py without -I
enable_load_extension raises, or sqlite_vec.load failsThe Python build has SQLite extension loading disabledUse a uv-managed or Homebrew Python, then uv sync again
connection refused on port 11434Ollama is not runningollama serve, or docker compose up -d; check OPENAI_BASE_URL
model "…" not foundModel not pulled, or name differs from EMBED_MODEL / CHAT_MODELollama list, then make models
Ingest takes minutes for 34 chunksCPU contention, or a large embedding modelTime one /v1/embeddings call; check uptime and ollama ps
Vector insert fails with a dimension errorEMBED_DIM does not match the embedding model's outputSet EMBED_DIM to the model's size and run make clean && make ingest
Hit rates look random after changing the modelThe index still holds vectors from the old modelmake ingest after every change to EMBED_MODEL or EMBED_DIM
Answers are very slowModel partly on CPU, a large context, or a busy hostollama ps (PROCESSOR column) and uptime; use a smaller model or quant, or a GPU
make graph takes hoursOne constrained LLM call per chunk on a CPUExpected on CPU (Step 7). Interrupted runs resume from .rag/extractions.jsonl; use a GPU host for real corpora
make graph-query links nothingGraph not built after the last make ingest, or no entity name in the questionmake graph after every ingest (chunk IDs change); lower LINK_MIN_SIM only after checking the linked names
The agent returns uncited_answer_rejectedThe model cited numbers it was not shownExpected occasionally with small models; the code refuses the answer. Check the trace, then the prompt
Hungarian queries missStemming too aggressive, or the document is not in the indexRun make eval, check the misses, then try Exercise 9
The model answers from outside the passagesWeak model or a prompt that is too permissiveKeep temperature low, keep the "ONLY the numbered passages" rule, and add a golden question for it

Definition of done​

  • make models completed and ollama list shows both models
  • make ingest prints 34 chunks from 6 documents (plain model names, same data)
  • make ask and make ask-hu return cited answers that match data/
  • make eval runs and you can explain each miss
  • One change from the exercises made and measured against the baseline
  • You can name when you would choose dense, BM25, or hybrid for a customer corpus
  • make graph completed, and you graded at least 10 extracted relations against their chunks
  • make agent and make agent-unknown ran, and you can say for each answer what is right, missing, or unsupported

Sources​