Ugrás a fő tartalomhoz

RAG as a Knowledge Base for Small Agents

Basic Advanced

Service: Agentic AI Platforms, RAG Systems on Private Infrastructure · Typical shape: one model on a single 24–48 GB GPU (or a 32 GB workstation for quantized 20–27B models), one workflow owner, the knowledge base built from existing runbooks and tickets

A small local model (2–30B parameters) is a weak planner and a poor source of facts, but it follows instructions it is shown well. This page is about making that strength do the work: knowledge, procedures, tool usage examples, and past resolutions live in a retrieval layer, the model sees only the slice it needs for the current step, and every output passes a schema check and a deterministic tool layer before anything happens. The model's job becomes narrow: pick one action from a short list, fill its arguments, and quote the source it used.

The page assumes the agent pattern from Agentic AI Platforms and the retrieval stack from RAG Systems. For the hands-on version on a laptop, do the Local RAG Lab first. For graph-shaped knowledge, see Graph RAG. For choosing the model itself, see Model Landscape and Best Practices.


1. Why this works​

Failure of a small modelMitigation in the retrieval layer
Invents facts (limits, IDs, policy numbers)Facts come from retrieved, cited passages; the prompt forbids other sources; no-hit returns "not found" without calling the model
Forgets the procedure over several stepsThe relevant SOP is injected before step 1 and re-injected each turn; steps are numbered and checked by code
Calls the wrong tool or invents argumentsOnly the top-k tools for this turn are offered; the action schema enumerates their names; arguments are validated by Pydantic before execution
Loops or driftsHard step and tool-call budgets; a state machine owns the sequence, not the model
Weak at multi-hop reasoningDecompose in code (fixed sub-queries or a workflow graph), not by asking the model to plan

The principle: a small model follows retrieved instructions well and plans poorly. Move planning into code and knowledge into the index. The model then does pattern-matching over a few kilobytes of relevant text, which is the regime where 2–30B models are reliable.

What this does not fix

Retrieval cannot compensate for a model that cannot emit valid JSON in your format, or one that ignores its context. Test both in the model-selection step (section 7) before building on a model.


2. What to store​

Store more than documents. A small agent needs material shaped for step-by-step use.

KindContentChunk shapeRetrieved when
SOP / runbookOrdered steps for a taskOne procedure per record; numbered steps; preconditions and expected result at the topThe task matches the procedure title
Tool cardName, purpose, arguments, return shape, side effectsOne tool per record; "when to use" line firstEvery turn, to select the tools to expose
Example trajectoryA resolved past case as a sequence of call, observation, answerQuestion, ordered calls with arguments, final answer, source IDsDynamic few-shot: the examples most similar to the current request
Reference docsPolicies, product docs, API docsStructure-aware chunks with heading path prepended (see RAG Systems)Factual questions
Glossary / entity cardCanonical names, IDs, aliases, ownersOne entity per recordQuery mentions an alias or ID
FAQ pairQuestion and vetted answerQuestion as title, answer under 150 wordsNear-duplicate questions
Resolved ticketSymptom, root cause, fix, verificationSummary, symptoms, fix, verification; PII removedSimilar incidents; example trajectories

Chunk format for small models​

Every record uses the same envelope, so the retriever and the prompt builder stay simple:

kb/sop-refund-duplicate-charge.md
---
id: sop-refund-duplicate-charge
kind: sop
title: Refund a duplicate charge
when: Customer reports being charged twice for one order
acl: [support-l1, support-l2]
version: 3
reviewed_by: ops-lead
updated: 2026-10-02
---

Preconditions: order_id known; order status is "paid".

1. Call lookup_order with the order_id.
2. If two payments share the same amount and timestamp within 5 minutes, the second is the duplicate.
3. Call issue_refund with the duplicate payment amount. Do not refund more than the duplicate.
4. Answer with the refund ID and cite this SOP as [1].

If the order has one payment only: do not refund; answer that no duplicate was found.

Rules for this shape:

  • Self-contained: a record must make sense when retrieved alone. Write "the order" not "it", and repeat the entity name.
  • Title plus "when" line: the title and when are embedded with the body, so the retriever matches on the situation, not only on the topic.
  • Numbered steps, one action per step: small models follow a numbered list far more reliably than prose.
  • Bounded length: keep a record under about 400 tokens. Split long runbooks into procedures with one when each.
  • Stable IDs and versions: citations reference the ID and version; the eval suite pins them.
  • Metadata set at ingestion (kind, acl, version, reviewed_by, updated), never by the model.

Ingestion and review​

  • Authoring is by humans; the pipeline converts, embeds, and indexes. Generated tool cards are generated from the function signature, then reviewed.
  • Each record has a status (draft, reviewed, retired). Only reviewed records are indexed for production retrieval.
  • Re-embed on every change into a new index alias (see RAG Systems, section 11).

3. Retrieval patterns for weak agents​

3.1 Pre-retrieval injection vs retrieval-as-a-tool​

Pre-retrieval injectionRetrieval as a tool
Who decides to retrieveCode, before step 1The model, mid-loop
Failure modeRetrieves once; may miss a fact needed laterModel skips retrieval and answers from memory, or retrieves with a bad query
CostOne retrieval per request; predictable latencyExtra model turns; variable latency
Fit for 2–30B modelsDefault. Removes the decision the small model is worst atAdd only as a single narrow tool for a second lookup

Recommendation: inject the SOP, the top examples, and the top-k tool cards before the first model call. Expose at most one retrieval tool (search_kb(query, kind)) for follow-up lookups, with a budget of two calls per run. Do not give a small model a general "search" tool that it has to reason about in every turn.

3.2 Tool-description retrieval (top-k tools per turn)​

Exposing a large tool catalogue to a small model degrades tool selection. RAG-MCP (Gan et al., 2025) indexes MCP tool descriptions and sends only the matching tools to the model. On the MCPBench web-search subset, tool-selection accuracy was 43.13 % with RAG-MCP against 13.62 % for the "Blank Conditioning" baseline and 18.20 % for "Actual Match". Average prompt tokens were 1,084, against 2,134 for Blank Conditioning (−49 %) and 1,646 for Actual Match (−34 %). Those numbers come from that benchmark; treat them as direction, not a guarantee for your catalogue.

Practical rules:

  • Index one tool card per tool: name, a one-line "use when", argument schema summary, and one example call.
  • Retrieve k = 3–6 per turn; always include a fixed core set (for example, lookup_order) that the workflow requires.
  • Put the retrieved tool names into the action schema as an enum (section 4.1). The model then cannot name a tool that was not offered.
  • Keep the total catalogue small per agent. Splitting into routed sub-agents beats one agent with 50 tools.

3.3 Example and skill retrieval (dynamic few-shot)​

Small models imitate examples more reliably than they follow abstract rules. Retrieve 2–3 resolved trajectories similar to the current request and show them as call-observation-answer sequences:

prompt fragment: retrieved example
Example 2 (sop-refund-duplicate-charge, resolved 2026-09-14)
User: I was charged twice for order A-1182.
Action: {"tool": "lookup_order", "args": {"order_id": "A-1182"}}
Observation: {"payments": [{"id": "p1", "amount": 39.90, "ts": "09:01"}, {"id": "p2", "amount": 39.90, "ts": "09:02"}]}
Action: {"tool": "issue_refund", "args": {"order_id": "A-1182", "amount_eur": 39.90}}
Final: {"answer": "Refunded the duplicate payment of EUR 39.90 (refund r-88). Source [1].", "cited": [1]}

Rules:

  • Examples are retrieved by similarity to the request, filtered by ACL and by reviewed status, not hand-picked per prompt.
  • Store examples as data: the same trajectory record feeds the golden-trajectory eval (section 8).
  • Keep examples short. Two examples of 150 tokens cost less than one long instruction the model ignores.

3.4 Query rewriting by template, not by the model​

Asking a small model to rewrite the user's query adds a failure point and a latency cost. Build the query from fields the code already knows:

query template
def build_query(ticket: dict) -> str:
return f'{ticket["product"]} {ticket["issue_type"]}: {ticket["summary"]}'

Use the model for rewriting only when a measured retrieval hit rate shows it helps (section 8). Keep the original user text in the query as well; BM25 needs the literal tokens such as order IDs and error codes.

3.5 Hybrid search and reranking​

Use the same stack as RAG Systems, section 5: dense plus BM25 fused with RRF, then a cross-encoder rerank over the top 20–50, keeping 3–5 for the prompt. Exact identifiers (order IDs, SKUs, error codes) are the cases where dense-only retrieval fails most visibly in agent workloads; BM25 keeps them retrievable.

3.6 Metadata filters set by code​

The code derives filters from the authenticated session and the workflow state, never from model output:

FilterSourceApplied to
acl contains user's groupValidated token, server-side lookupEvery query
kind in exampleWorkflow step definitionProcedure and example queries
kind = toolFixedTool-card query
status = reviewed and retired = falseFixedEvery query
product = ticket.productTicket system fieldReference and example queries

A model-extracted filter is a hallucination risk: the filter silently drops the right record. If you need model-extracted filters, constrain them with the same schema mechanism as section 4 and check them against the allowed values.


4. Constraining the model​

4.1 Structured outputs​

Every model call returns a JSON object that a schema validates. Three serving stacks, verified against their docs:

RuntimeMechanismSyntax (as of 2026-10)
vLLMresponse_format with json_schema on the OpenAI-compatible server; the older guided_* request fields were removed in v0.12.0 and the docs now point to structured_outputsresponse_format={"type": "json_schema", "json_schema": {"name": "action", "schema": Schema.model_json_schema()}}; backends include xgrammar and guidance per the vLLM docs
Ollamaformat on the native API accepts a JSON schema; the OpenAI-compatible layer accepts response_formatPython: chat(model=..., messages=..., format=Schema.model_json_schema()), per the Ollama docs
llama.cppGBNF grammars, or a JSON schema converted to a grammar: json_schema on completion endpoints, response_format on chat completionsSupported subset only; unsupported keywords are skipped silently, so validate the grammar (gbnf-validator) per the llama.cpp grammar guide

Rules that follow from the llama.cpp docs and the engine behaviour:

  • The schema constrains output but is not added to the prompt. Describe the expected format in the system prompt and show one example.
  • Check that every construct you use is supported by the engine. not, conditionals, patternProperties, and uniqueItems are gaps in llama.cpp's converter.
  • Keep the schema flat. Deeply nested unions raise the grammar cost and the rate of invalid partial outputs.
  • Still validate in code. The schema guarantees shape, not business rules.

4.2 A small action space​

The action space is the set of things the model may emit at a given step:

ActionArgumentsNotes
One of the offered toolsArguments per tool schemaNames enumerated from the top-k retrieval
finalanswer (string), cited (list of source numbers)Only valid when cited is non-empty or answer is the not-found sentinel

4.3 State machine around the model​

Wrap the model in a workflow graph. The graph decides which step runs, which context is injected, which tools are offered, and when to stop. The model fills one node at a time.

Choosing the implementation:

OptionUse when
Plain code (a for loop with explicit steps, as in section 6)Up to about five steps, one agent, no pause/resume needs
LangGraph (StateGraph with conditional edges and a checkpointer)Branching, human approval, or pause/resume; see Agentic AI Platforms for durable execution options

Either way, the graph is code you can unit-test without a model: mock the LLM node and assert transitions.

4.4 Step and iteration budgets​

  • Model calls per request: 4–6 for a SOP-driven workflow.
  • Tool calls per request: a per-tool limit plus a total limit; write tools limited to one per request unless the SOP says otherwise.
  • Repair retries: at most 2 per step; on schema failure, return the validation error to the model once, then escalate.
  • Wall-clock timeout: per request, enforced outside the model.

Exhausting a budget is a normal outcome. Return budget_exhausted and route to a human rather than letting the loop continue.

4.5 Citations and "answer only from context"​

The system prompt and the output schema work together:

prompts/system.txt
Follow the procedure in SOURCES exactly. Use the examples only as a pattern for the call format.
Use only facts in SOURCES. Cite each source you used by its number in "cited".
If SOURCES does not cover the request, return the action final with answer "NOT_FOUND" and cited [].

Code checks after final:

  • Every number in cited exists in the source list shown to the model.
  • answer is NOT_FOUND only when cited is empty, and a non-empty cited never accompanies NOT_FOUND.
  • Optional: the cited record's text contains the numeric values that appear in the answer. This catches a common small-model error, a correct citation with a wrong amount.

4.6 Abstention thresholds on retrieval scores​

Abstain in code before the model is called:

if not hits or hits[0].score < MIN_SCORE:
return {"answer": "NOT_FOUND", "cited": [], "reason": "no_supporting_record"}

Calibrate MIN_SCORE on the golden set (section 8): plot the top-1 score for questions that have an answer against questions that do not, and choose the threshold that meets your abstention target. Do not reuse a score threshold across embedding models; scores are not comparable between models.


5. Memory​

Separate two stores with different owners and lifetimes.

Short-term scratchpadLong-term memory
ScopeOne request or one sessionAcross sessions and users
ContentsTool results, step history, current SOPFacts learned from resolved cases, customer preferences, corrected procedures
OwnerThe workflow runtimeWritten by the pipeline; reviewed by a human before it becomes retrievable
LifetimeDiscarded at request end; trimmed to a token budgetVersioned, with provenance and expiry
RetrievalAlways in the prompt, trimmed by code (keep the last 2 observations, summarise older ones deterministically)Retrieved like any KB record, filtered by ACL and status

Rules:

  • Write-back goes to a review queue, never directly into the retrievable index. Otherwise one wrong resolution becomes an instruction for every future request. This matches the ASI06 memory-poisoning control in Agentic AI Platforms.
  • Store tool output as data, not as instructions, and mark it as such in the prompt.
  • Expire memories with a valid_until or a review date.

Memory frameworks (versions from PyPI and licences from each repository's LICENSE file, checked 2026-10-18):

ProjectLatest (PyPI)Description (from its README)LicenceModel
Graphitigraphiti-core 0.31.1 (15 Oct 2026)"A Framework for Building Temporal Knowledge Graphs"; tracks when facts were valid and invalidates superseded onesApache-2.0Temporal knowledge graph; backs Zep
Mem0mem0ai 2.2.1 (25 Sep 2026)"The Memory Layer for Personalized AI"; drop-in memory infrastructureApache-2.0Fact extraction and consolidation over a vector store, optional graph
Lettaletta 0.34.12 (12 Oct 2026)"Build stateful agents with memory that can learn and improve over time"; the agent edits its own memory blocks through toolsApache-2.0Agent runtime with tiered memory, descended from MemGPT

Recommendation for small agents: start with the two-store design above in plain code and a Postgres table with a review status. Adopt Graphiti when facts change over time and you need "what was true on date X"; adopt Mem0 or Letta only if you want their runtime, because each one owns the write path and that decision is hard to reverse. Letta's self-editing memory is the model writing its own memory, which is the step this page recommends against for 2–30B models without a review gate.


6. Reference architecture and minimal example​

6.1 Architecture​

6.2 Minimal runnable example​

The example talks to any OpenAI-compatible endpoint: Ollama on http://localhost:11434/v1 or vLLM on http://localhost:8000/v1. It retrieves an SOP and examples, injects them with numbered sources, constrains the output to a JSON schema with the tool names enumerated from the top-k tool cards, validates arguments, executes the tool, and cites its sources. Dense retrieval only, for brevity; section 3.5 covers the production stack.

Requirements: pip install openai pydantic numpy (numpy is optional, the code uses plain Python). Pull a chat model and an embedding model on the serving host; the defaults below assume Ollama tags gpt-oss:20b and qwen3-embedding:0.6b. Both tags exist in the Ollama library (checked 2026-10-18); confirm them on your host with ollama list.

The index input is kb/chunks.jsonl, one record per line with id, kind, title, when, body, acl, and status.

agent/kb_agent.py
import json
import math
import os
from typing import Callable, Literal, NamedTuple

from openai import OpenAI
from pydantic import BaseModel, ValidationError, create_model

client = OpenAI(base_url=os.environ.get("LLM_BASE_URL", "http://localhost:11434/v1"), api_key="local")
CHAT = os.environ.get("CHAT_MODEL", "gpt-oss:20b")
EMBED = os.environ.get("EMBED_MODEL", "qwen3-embedding:0.6b")
MIN_SCORE = float(os.environ.get("MIN_SCORE", "0.45")) # illustrative; calibrate on the golden set
MAX_STEPS = 4

KB = [r for r in map(json.loads, open("kb/chunks.jsonl", encoding="utf-8")) if r["status"] == "reviewed"]


def embed(texts: list[str]) -> list[list[float]]:
out = client.embeddings.create(model=EMBED, input=texts).data
return [[x / math.sqrt(sum(v * v for v in d.embedding)) for x in d.embedding] for d in out]


VECS = embed([f'{r["title"]}. {r["when"]}\n{r["body"]}' for r in KB])


def search(query: str, kinds: set[str], k: int) -> list[tuple[float, dict]]:
qv = embed([query])[0]
scored = [
(sum(a * b for a, b in zip(qv, v)), r)
for v, r in zip(VECS, KB)
if r["kind"] in kinds
]
return [(s, r) for s, r in sorted(scored, key=lambda x: -x[0])[:k] if s >= MIN_SCORE]


class Tool(NamedTuple):
args: type[BaseModel]
run: Callable[..., dict]


class LookupOrder(BaseModel):
order_id: str


class IssueRefund(BaseModel):
order_id: str
amount_eur: float


ORDERS = {"A-1182": {"payments": [{"id": "p1", "amount": 39.90, "ts": "09:01"},
{"id": "p2", "amount": 39.90, "ts": "09:02"}]}}
TOOLS: dict[str, Tool] = {
"lookup_order": Tool(LookupOrder, lambda order_id: ORDERS.get(order_id, {"error": "not_found"})),
"issue_refund": Tool(IssueRefund, lambda order_id, amount_eur: {"refund_id": "r-88", "amount_eur": amount_eur}),
}

SYSTEM = """Follow the procedure in the "sop" source exactly. "example" sources show the call format; use them as a pattern only.
Use only facts in SOURCES. Cite sources by number in "cited".
If SOURCES do not cover the request, return tool "final" with answer "NOT_FOUND" and cited [].
Return exactly one JSON action per turn.

TOOLS:
{tools}

SOURCES:
{sources}"""


def run(question: str) -> dict:
sop = search(question, {"sop"}, k=1)
if not sop:
return {"answer": "NOT_FOUND", "cited": [], "reason": "no_supporting_record"}
examples = search(question, {"example"}, k=2)
tool_cards = search(question, {"tool"}, k=5)
sources = [sop[0][1]] + [r for _, r in examples]
names = [r["id"] for _, r in tool_cards] + ["final"]

Action = create_model(
"Action",
tool=(Literal[tuple(names)], ...),
args=(dict, ...),
cited=(list[int], ...),
answer=(str | None, None),
)
system = SYSTEM.format(
tools="\n".join(f'{r["id"]}: {r["when"]}' for _, r in tool_cards),
sources="\n\n".join(f'[{i}] ({r["kind"]}) {r["title"]}\n{r["body"]}' for i, r in enumerate(sources, 1)),
)
messages = [{"role": "system", "content": system}, {"role": "user", "content": question}]
schema = {"name": "action", "schema": Action.model_json_schema()}

for step in range(1, MAX_STEPS + 1):
raw = client.chat.completions.create(
model=CHAT, temperature=0, messages=messages,
response_format={"type": "json_schema", "json_schema": schema},
).choices[0].message.content
try:
act = Action.model_validate_json(raw)
except ValidationError as e:
messages += [{"role": "assistant", "content": raw},
{"role": "user", "content": f"Invalid action: {e.errors()[0]['msg']}. Return one valid action."}]
continue

if act.tool == "final":
if act.answer != "NOT_FOUND" and not set(act.cited) <= set(range(1, len(sources) + 1)):
messages.append({"role": "user", "content": "Cited source numbers are invalid. Fix and return final."})
continue
return {"answer": act.answer, "cited": act.cited, "steps": step}

try:
args = TOOLS[act.tool].args.model_validate(act.args)
observation = TOOLS[act.tool].run(**args.model_dump())
except (ValidationError, TypeError) as e:
observation = {"error": str(e)[:300]}
messages += [{"role": "assistant", "content": raw},
{"role": "user", "content": f"Tool result: {json.dumps(observation)}"}]

return {"answer": None, "cited": [], "steps": MAX_STEPS, "error": "budget_exhausted"}


if __name__ == "__main__":
print(json.dumps(run("I was charged twice for order A-1182. Can you refund the duplicate?"), indent=2))

Smoke test (2026-10-23): with a seven-record kb/chunks.jsonl (two SOPs, one example, three tool cards, one retired SOP), CHAT_MODEL=gemma4:e4b and EMBED_MODEL=embeddinggemma-2:270m on Ollama 0.40.8, the run called lookup_order, then issue_refund for 39.90, and returned "Refunded the duplicate charge of EUR 39.90 (refund r-88)." citing [1] in 3 steps. The run used MIN_SCORE=0.3. The threshold is the part that does not transfer between embedders. With that embedder and no query prompt, the matching question scored 0.83 against the refund SOP, and an unrelated one ("What is your return policy for opened electronics?") still scored 0.65. A threshold of 0.45 would never abstain. Calibrate MIN_SCORE on in-scope and out-of-scope golden questions for the embedder you deploy, and use the model's query and document prompts (see the Local RAG Lab, Step 4).

The code abstains before calling the model when no SOP clears the threshold, offers only the top-5 tool IDs as an enum, validates tool arguments and returns errors to the model as observations, checks citations against the sources it showed, and stops at MAX_STEPS.

Known gaps versus production: no write-tool approval gate (add an interrupt before issue_refund for amounts above a limit), no tracing (emit OTel GenAI chat and execute_tool spans per Agentic AI Platforms), no ACL filter in search, and dense-only retrieval. Each is a line or a function away; none changes the control flow above.

Thinking mode: the example does not set it. The switch depends on the runtime and the model (as of 2026-10, from the vendor docs and source):

Runtime and APIQwen3.8: thinking offgpt-oss: lowest effort
vLLM, OpenAI-compatible /v1/chat/completionsextra_body={"chat_template_kwargs": {"enable_thinking": False}}. Top-level reasoning_effort="none" also sets enable_thinking=False when no explicit value is passed (vLLM protocol)reasoning_effort="low"; gpt-oss documents only low, medium, and high
Ollama, OpenAI-compatible /v1/chat/completionsreasoning_effort="none"reasoning_effort="low"; Ollama maps "none" to no thinking output for gpt-oss (Ollama OpenAI compatibility)
Ollama native /api/chatthink: falsethink: "low"; allowed values come from /api/show (Ollama thinking)

Ollama's OpenAI-compatible /v1/chat/completions does not accept a think field; use reasoning_effort there. The Ollama docs list think only on the native API and on /v1/responses. Verify the switch on your deployed model before relying on it in the loop (section 7).


7. Model choice for small agents​

Tool-calling reliability decides whether this architecture works, so choose on that and on latency, not on general leaderboards. Measure on your own tool set with the scripted scenarios in section 8.

7.1 Candidates (as of 2026-10)​

ModelSize and architectureLicenceNotes
Qwen3.8-27BDense 27B, vision-language; 262K native context, extensible to about 1M; Hugging Face 14 Aug 2026Apache-2.0Smallest Qwen 3.8 open model on the Qwen HF org (no 2B to 4B variant as of 2026-10). The other open models are Qwen3.8-Flash-Next (125B total, 6B active MoE, about 180 GB of weights) and Qwen3.8-2.4T-A95B. Thinking is on by default; reasoning_effort takes xhigh (default), medium, or low. Simon Willison reports it is strong for coding agents but overthinks at its default xhigh effort. The Ollama library lists qwen3.8:27b at 18 GB
Gemma 4Five sizes: E2B (2.3B effective), E4B (4.5B effective), 12B Unified (dense, 11.95B), 26B A4B (MoE, 25.2B total), 31B dense (30.7B); released 2 Apr 2026Apache-2.0 (model card license tag and license)E2B and E4B for edge; 26B A4B for low active-parameter cost. Model card: Gemma 4 model card
gpt-oss-20bMoE, 21B total, 3.6B active; MXFP4 weights; runs within 16 GB (model card)Apache-2.0Function calling and Structured Outputs are listed in the gpt-oss README. Reasoning effort is low, medium, or high. The Ollama tag gpt-oss:20b is 14 GB

Naming caveat: the request named "Qwen3.8 small"; the smallest Qwen 3.8 open model is the 27B dense model, and the Flash-Next MoE is small only in active parameters.

7.2 Evaluations to read​

EvalWhat it measuresRead it for
Berkeley Function Calling Leaderboard (BFCL) V4Function calling across single, parallel, multi-turn, and agentic categories; page last updated 2026-04-12Whether the model calls functions correctly; use the multi-turn and agentic columns for agents
tau-bench (introduces pass^k) and tau2-bench (code: sierra-research/tau2-bench)Tool-agent-user interaction under domain policies. The tau2 paper adds a telecom dual-control domain, where the user also acts on shared state. The repository now ships τ³-bench (v1.0.1, July 2026) with a banking_knowledge domain; scores from tau2-bench before 1.0.1 are not comparable with later versionsMulti-turn tool use under a policy. The tau2 paper reports significant performance drops when agents move from no-user to dual-control

Use pass^k, not pass@1, for agent decisions: pass^k is the probability that all k independent runs succeed, which is the number a customer experiences. Re-run the public benchmarks against your quantized build; published numbers are for the reference weights.

7.3 Selection checklist​

  • Structured-output validity on your action schema: 200 runs, target above 99 % valid JSON after one repair. A model that fails this fails the architecture regardless of benchmark rank.
  • Tool selection from a 5-card candidate set, including distractor cards.
  • Abstention: give it a question the SOP does not cover and check it returns NOT_FOUND.
  • Thinking on or off per step: run the scenario suite with thinking off for routing and argument filling, and with it on only for steps that need reasoning. Measure the latency and accuracy difference; do not assume it.
  • Context use: test at the prompt sizes you will send (SOP, two examples, five tool cards, and tool results), not at a short demo prompt.

7.4 Quantization​

Quantization changes tool-calling behaviour more than it changes chat quality, and the small models are the most sensitive. The rule is to measure, not to assume. Run the selection checklist on the exact quantized file you will deploy, and compare it with the unquantized reference on the same suite. Pick the smallest quantization that keeps the structured-output and tool-selection scores within your threshold. This page does not cite a published quantization study for tool calling; the measurement is the source of truth.


8. Evaluation​

Evaluate the layers separately. A failure in retrieval looks like a failure in the model; the per-step metrics tell them apart.

MetricDefinitionTarget to start with (set your own after a baseline)
Task successScripted scenario ends in the correct final state (refund created with the right amount, or NOT_FOUND where expected)Gate in CI; report pass^3
Retrieval hit rate per stepThe record needed for this step is in the top-k shown to the model; measured per step type (SOP, example, tool card)Recall@k against golden labels; track per kind
Tool-call validityShare of actions that pass the schema and argument validation on the first attemptAbove 99 % for structured output; report the repair rate
Tool selection accuracyCorrect tool chosen given the offered cardsMeasured on distractor-heavy cases
Abstention correctnessOn unanswerable questions, NOT_FOUND returned; on answerable ones, no false NOT_FOUNDReport both; the trade-off is set by MIN_SCORE
Citation validityCited IDs exist and the cited record supports the numeric claims100 % by code check; sample for semantic support
Budget exhaustion rateRequests ending in budget_exhaustedBelow 1 % in the regression suite; alert above it

Golden trajectories​

A golden trajectory is a recorded sequence of expected steps for a scenario, with the correct final state:

evals/golden.jsonl (one case)
{"id": "dup-refund-001", "input": "I was charged twice for order A-1182", "fixture": "orders/A-1182.json", "expected_sop": "sop-refund-duplicate-charge", "must_call": ["lookup_order", "issue_refund"], "must_not_call": ["cancel_order"], "expect_final": {"refund_amount_eur": 39.90}, "max_steps": 4}
{"id": "no-such-policy-002", "input": "Can I expense a gym membership?", "fixture": "none", "expected_sop": null, "must_call": [], "must_not_call": ["issue_refund"], "expect_final": {"answer": "NOT_FOUND"}, "max_steps": 2}

Grade outcomes first: compare the final state of the fixture environment, not the agent's claim. Use must_call and must_not_call as constraints, not as an exact sequence, so valid alternative paths pass. Seed the fixture for each run and reset it afterwards.

Score each case on four booleans: final state matches, must_call and must_not_call respected, steps <= max_steps, and the expected SOP appeared in the retrieved IDs. Sources of cases: start with 20–30 scenarios drawn from real tickets (with PII removed), add the adversarial ones (ambiguous request, duplicate records, a retired SOP still in the index, a prompt injection inside a resolved ticket), and grow the set each time production shows a new failure. Review the failing cases weekly with the feedback loop in section 9.


9. Best practices​

  • Put facts and procedures in the index; the prompt holds the role, the output contract, and the rules.
  • Keep the model's decision to one action per turn from an enumerated list; check arguments in code.
  • Retrieve before the first call; expose at most one budgeted follow-up retrieval tool.
  • Write SOPs as numbered steps with a "when" line, and test each record by retrieving it with its own title.
  • Derive every filter in code (ACL, status, kind, product) and validate everything the model emits.
  • Measure on the quantized, deployed build; review every memory write; trace every step with the IDs the golden set uses.

10. Pitfalls to design around​

  • Top-k tool retrieval misses the one tool needed. Keep a fixed core set per workflow and measure tool-card recall per scenario; a missed tool is an invisible failure unless you check.
  • Examples teach the wrong pattern. A resolved ticket with a bad fix becomes a template for every later request. Review examples like code.
  • Stale SOPs. Retrieval returns the highest-similarity record, which can be an old version. Index only reviewed records and retire superseded versions in the same change.
  • Dense retrieval on identifiers. Order IDs and error codes can be semantically close and literally different. Keep BM25 in the stack.
  • Thresholds that do not transfer. A MIN_SCORE calibrated on one embedding model is wrong for another; recalibrate on each change.
  • Thinking that eats the budget or overthinks routing. Measure per step (section 7.3).
  • Schema that the engine silently weakens. Test that invalid output is actually rejected.
  • Prompt injection in retrieved content. Keep retrieved text in the data section, keep tool checks independent of the prompt, and include injection cases in the suite.
  • Model writes its own memory. Keep the write path behind a review status.
  • Quantization regressions invisible in chat quality. Tool-call validity catches them; chat samples do not.

Definition of done​

  • Knowledge stored as SOPs, tool cards, examples, glossary, FAQ, and resolved tickets, each with the standard envelope and a reviewed status
  • SOP and examples injected before the first model call; at most one follow-up retrieval tool, budgeted
  • Top-k tool cards per turn; action schema enumerates offered tool names
  • Structured output enabled and tested on the deployed runtime; invalid output is rejected in test
  • Tool arguments validated in code; write tools behind an approval rule for consequential amounts
  • Citations checked in code; NOT_FOUND path tested with an unanswerable question
  • Abstention threshold calibrated on the golden set for the deployed embedding model
  • Step, tool-call, and timeout budgets enforced outside the model; budget exhaustion routes to a human
  • Memory write-back goes through a review queue; no direct writes to the retrievable index
  • Scripted scenario suite with golden trajectories runs in CI; pass^3, tool-call validity, retrieval hit rate per step, and abstention correctness reported
  • Model choice made on tool-calling measurements on the quantized deployment build
  • Traces emitted per step; a failing production case can be replayed as a golden case

Sources​