GraphRAG and Entity Extraction with Local Models
Service: Sovereign & Private AI, RAG Systems · Typical shape: a 1–2 week spike to prove the graph beats vector retrieval on your question set, then a productionised extraction and query tier
This page is the implementation companion to RAG Systems. Read Lab: Local RAG first; this page assumes you already have a chunker, an embedder, and a vector index. The agent side is covered in RAG for Agents and Agentic AI Platforms. Model selection for the extractor and the answerer follows Model Landscape, and the serving tier follows Inference Engineering.
The approach is simple to state: during ingestion, an LLM turns each chunk into entities and relations; the graph is stored with a pointer from every edge back to the chunks that produced it; at query time you walk the graph to find context that a similarity search would miss. Almost all of the difficulty sits in extraction quality, entity resolution, and cost. This page covers those three in that order.
1. When graph RAG beats vector RAG
Vector RAG retrieves passages that look like the question. GraphRAG retrieves passages that are connected to the entities the question names, and it can answer questions that need a summary across the whole corpus. The two behave differently on three question types:
| Question type | Example | Vector RAG | Graph RAG | Verdict |
|---|---|---|---|---|
| Single-hop lookup | "What is the per diem for Berlin?" | Strong: one passage contains the answer | Adds cost, little gain | Vector |
| Multi-hop | "Which suppliers of Project X also supplied the 2024 audit findings?" | Retrieves the two sides separately, often misses the join | Walks Project X → supplier → audit finding | Graph |
| Global or sensemaking | "What are the main risk themes across the 2,000 incident reports?" | Returns the 10 most similar reports, not the corpus | Community summaries answer over the whole graph | Graph (global search) |
| Aggregation | "How many contracts name vendor V?" | Unreliable: counts only what it retrieves | Exact with a COUNT over edges | Graph, or a structured query |
| Fact lookup with paraphrase | "When did we move the DR site?" | Good with hybrid search | Good only if extraction captured the event | Vector or hybrid |
| Entity-centric corpus (people, parts, contracts, regulations) | "Everything we know about component C-17" | Scattered hits | One neighbourhood query | Graph |
Graph RAG does not pay off when:
- The corpus is mostly prose with few recurring named entities (novels, meeting notes without people or systems), so extraction yields a sparse graph.
- Questions are single-hop and answered by one passage. The graph adds indexing cost and a second failure mode.
- Entities are ambiguous and no resolution step is possible (for example, "Anna" across unrelated departments with no identifiers).
- The corpus changes faster than you can afford to re-extract. Incremental extraction is possible but must be designed in.
- The team cannot run an evaluation set. Without one you cannot tell a good graph from a plausible-looking one.
Evidence is mixed and benchmark-dependent. The GraphRAG-Bench paper "When to use Graphs in RAG" (arXiv 2506.05690) states that recent studies report GraphRAG frequently underperforms vanilla RAG on many real-world tasks, and sets out to measure when graph structure does help (arXiv 2506.05690). Treat any vendor benchmark, including the ones in this page's Sources, as a hypothesis to re-measure on your corpus.
Cost note: indexing is an LLM call per chunk
Graph construction is paid up front, per chunk, and it is the dominant cost:
extraction calls = C × (1 + g) C = chunks, g = gleaning/re-prompt passes
input tokens = C × (p + t) p = prompt tokens, t = chunk tokens
output tokens = C × o × (1 + g) o = output tokens per call
Worked example with assumed numbers (replace with your measurements): 10,000 chunks, 600 tokens each, a 400-token extraction prompt, 200 output tokens, one pass (g = 0):
| Quantity | Value |
|---|---|
| Extraction calls | 10,000 |
| Input tokens | 10,000 × (400 + 600) = 10 M |
| Output tokens | 10,000 × 200 = 2 M |
Two consequences follow. Re-indexing after a prompt or schema change costs the whole corpus again, so version the prompt and schema alongside the graph. And gleaning passes multiply cost by (1 + g) for a recall gain you must measure against your hand-labelled sample (section 7) before you accept it.
Microsoft's LazyGraphRAG moves LLM work to query time: its indexing cost is stated as identical to vector RAG and 0.1% of full GraphRAG indexing, with query-time cost roughly 700× lower than GraphRAG global search for comparable quality (Microsoft Research, Nov 2024). These are vendor-reported figures. If indexing cost is the blocker, evaluate this option before paying for full extraction.
2. Entity and relation extraction with local models
Three approaches, in the order to try them:
| Approach | What it is | Strengths | Weaknesses | Use when |
|---|---|---|---|---|
| A. Schema-first LLM extraction | Local instruct model, JSON-schema constrained output mapped to Pydantic models | Typed output, provenance easy, works for any relation set | One LLM call per chunk; drift in entity naming | Default. Start here |
| B. Small encoder NER pre-pass | GLiNER (zero-shot span NER) and GLiREL (zero-shot relation classification) on CPU or a small GPU | Very cheap, deterministic, fast; labels given at inference time | Shallow; relations limited to the candidate labels; no coreference | High volume, fixed entity types, or as a candidate generator that the LLM then verifies |
| C. Few-shot LLM extraction | Prompt with worked examples, free-form or loosely structured output | No schema work to start | Output parsing failures, inconsistent types | Prototyping only; move to A once the types stabilise |
2.1 Schema-first extraction (approach A)
Define the schema once and use it for three things: the constrained decoder, the validation of the response, and the graph write. Use the OpenAI-compatible response_format with json_schema. vLLM accepts it on its OpenAI-compatible server; the guided_* request fields were removed in v0.12.0, and the native API takes the json key under structured_outputs (vLLM docs). Ollama's /v1/chat/completions maps response_format with type: "json_schema" onto its native format field, as the Ollama source shows (Ollama OpenAI compatibility, ollama/openai/openai.go).
Use temperature=0 for extraction. Constrained decoding guarantees syntax, not correctness: the model can still invent entities or misattribute relations, which is why section 7 measures precision and recall.
2.2 Small encoder NER and relation pre-pass (approach B)
GLiNER labels spans from a label list you pass at inference time, so the entity types can change without retraining. The current quickstart uses the gliner package and a community checkpoint (GLiNER):
pip install gliner
from gliner import GLiNER
model = GLiNER.from_pretrained("gliner-community/gliner_small-v2.5")
LABELS = ["person", "organization", "product", "location", "regulation"]
def spans(text: str) -> list[dict]:
return model.predict_entities(text, LABELS, threshold=0.5)
Fastino's GLiNER2 is a separate package (gliner2, not gliner). Its latest PyPI release is 2.1.0 (9 Oct 2026). It covers entity extraction and a schema-driven extraction interface, and it needs the local extra for inference (GLiNER2):
pip install "gliner2[local]"
from gliner2 import AutoExtractor
model = AutoExtractor.from_pretrained("fastino/gliner2.5-base-v1")
result = model.extract_entities(
"Apple CEO Tim Cook announced iPhone 15 in Cupertino yesterday.",
["company", "person", "product", "location"],
)
GLiREL is the relation counterpart. It classifies relations between entity spans you already have, so it needs the NER output as input (GLiREL):
pip install glirel
python -m spacy download en_core_web_sm
from glirel import GLiREL
import spacy
rel_model = GLiREL.from_pretrained("jackboyla/glirel-large-v0")
nlp = spacy.load("en_core_web_sm")
def relations(text: str, ner: list[list], labels: list[str]) -> list[dict]:
tokens = [t.text for t in nlp(text)]
# ner entries: [start_token, end_token_inclusive, type, surface_text]
return rel_model.predict_relations(tokens, labels, threshold=0.0, ner=ner, top_k=1)
Use the pre-pass as a candidate generator: pass its entity spans into the LLM prompt as hints ("these spans were detected; you may use them or add others"), or let the LLM verify the candidate triples. Do not write pre-pass output straight to the graph; its precision on your domain is unknown until measured.
2.3 Few-shot extraction (approach C)
Few-shot prompts are useful for a day of prototyping and then become a liability: they drift between model versions and make parsing fragile. Keep the examples, but move the output contract into the schema (2.1).
2.4 Runnable schema-first extractor
This script takes chunks with ids, calls a local OpenAI-compatible endpoint (vLLM or Ollama), validates each response against Pydantic models, and attaches the chunk id to every entity and relation. It is the minimal production shape: schema, validation, provenance, and a count of failures.
import json
import sys
from typing import Literal
from openai import OpenAI
from pydantic import BaseModel, ValidationError
BASE_URL = "http://localhost:8000/v1" # vLLM; Ollama: http://localhost:11434/v1
MODEL = "Qwen/Qwen3.5-9B" # must match the served model name
client = OpenAI(base_url=BASE_URL, api_key="local") # key is required by the client, not checked locally
EntityType = Literal["Person", "Organization", "Product", "Location", "Regulation", "Concept"]
RelationType = Literal["OWNS", "PART_OF", "DEPENDS_ON", "REGULATES", "WORKS_FOR", "LOCATED_IN", "RELATED_TO"]
class Entity(BaseModel):
name: str
type: EntityType
class Relation(BaseModel):
source: str
relation: RelationType
target: str
evidence: str # verbatim span from the chunk, used for audit and for the eval
class Extraction(BaseModel):
entities: list[Entity]
relations: list[Relation]
SYSTEM = (
"Extract entities and relations from the passage. Use only the listed types. "
"Names must appear verbatim in the passage. Evidence must be a verbatim quote. "
"Return nothing for facts the passage does not state."
)
def extract(text: str) -> Extraction | None:
resp = client.chat.completions.create(
model=MODEL,
temperature=0,
messages=[{"role": "system", "content": SYSTEM}, {"role": "user", "content": text}],
response_format={
"type": "json_schema",
"json_schema": {"name": "extraction", "schema": Extraction.model_json_schema()},
},
# Qwen3.5 thinks by default; extraction does not need it (vLLM; on Ollama use reasoning_effort="none")
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
try:
return Extraction.model_validate_json(resp.choices[0].message.content)
except ValidationError:
return None # real failure point: count and route to a retry queue
def run(chunks: list[dict]) -> tuple[list[dict], list[dict], int]:
ents, rels, failed = [], [], 0
for c in chunks:
ext = extract(c["text"])
if ext is None:
failed += 1
continue
ents += [{**e.model_dump(), "chunk_id": c["chunk_id"], "doc_id": c["doc_id"]} for e in ext.entities]
rels += [{**r.model_dump(), "chunk_id": c["chunk_id"], "doc_id": c["doc_id"]} for r in ext.relations]
return ents, rels, failed
if __name__ == "__main__":
chunks = [json.loads(line) for line in open(sys.argv[1], encoding="utf-8")]
ents, rels, failed = run(chunks)
print(json.dumps({"entities": len(ents), "relations": len(rels), "failed_chunks": failed}))
Input is JSON Lines with chunk_id, doc_id, and text. Run it with python graphrag/extract.py chunks.jsonl. The evidence field is the provenance anchor: the verbatim quote lets you check each edge against its source without re-running the model. Remove the evidence requirement only after the precision check in section 7 passes without it.
Two notes on the snippet. (The Local RAG Lab, Step 7, runs the same pattern end to end on Ollama with gemma4:e4b, including the measured cost of constrained decoding on a CPU.) First, Literal types become enum in the JSON schema, so the decoder cannot emit an out-of-set type. Second, the RelationType set is deliberately small. Every relation type you add multiplies the ways the model can mislabel an edge.
2.5 Entity resolution and deduplication
Extraction produces Acme GmbH, Acme, ACME Gmbh, and Acme Kft. as four nodes unless you resolve them. Resolution runs in three layers, cheapest first:
- Normalisation. Unicode NFKD, strip diacritics for matching (keep the original for display), lowercase, collapse whitespace, remove legal suffixes from a per-language list (
gmbh,kft,ltd,inc). Store the result asname_norm. - Alias table. A curated mapping
alias_norm -> canonical_id, maintained by hand for high-value entities and filled from identifiers (VAT numbers, product SKUs, employee ids) when present. Aliases beat any similarity score. - Embedding similarity. Embed
nameplus type and a short description. Merge candidates of the same type above a cosine threshold, and send the borderline band to the LLM or to a human. Pick the threshold on your labelled sample, not by convention.
import unicodedata
import re
LEGAL = {"gmbh", "kft", "ltd", "inc", "llc", "zrt", "bt", "nyrt"}
def norm(name: str) -> str:
s = unicodedata.normalize("NFKD", name).encode("ascii", "ignore").decode().lower()
s = re.sub(r"[^\w\s]", " ", s)
return " ".join(w for w in s.split() if w not in LEGAL)
def resolve(ents: list[dict], aliases: dict[str, str]) -> dict[str, str]:
"""Map every (type, name_norm) to a canonical id. Aliases win over normalisation."""
canon: dict[str, str] = {}
for e in ents:
key = norm(e["name"])
canon.setdefault(f'{e["type"]}:{key}', aliases.get(key, f'{e["type"]}:{key}'))
return canon
Rules that prevent the common resolution failures:
- Never merge across types.
Applethe company andapplethe fruit are different nodes; the type is part of the key. - Keep the surface forms: store every observed
nameon the canonical node asaliases, so answers can cite the name the source used. - Resolution is not idempotent under incremental indexing. Store the mapping and re-run it on new entities only, or every merge decision becomes a migration.
2.6 Provenance
Every node and edge carries chunk_ids (and doc_id, version). When an edge is merged from several chunks, the list grows; the edge's evidence keeps one quote per chunk. Three consequences:
- Answers can cite
[doc_id#chunk_id]for every fact, including facts that came from graph traversal, not from retrieved text. - A document deletion or a new version removes or re-points only the edges whose
chunk_idsreference that document. Without this you cannot honour deletions, which is a data-protection requirement in most customer deployments. - Evaluation can check each edge against its quote automatically (section 7).
3. Graph construction and storage options
| Option | Type and licence (as of 2026-10) | Vector index | Python entry point | Use when |
|---|---|---|---|---|
| Neo4j | Server; Community and Enterprise editions (check terms) | Native vector indexes in the same database | neo4j-graphrag 1.22.0 (1 Oct 2026) | Default for production graph RAG. Largest ecosystem, mature drivers, built-in KG builder |
| Memgraph | Server; BSL 1.1 for the community code, Memgraph Enterprise License for enterprise features (Memgraph) | Built-in vector indexes, hybrid graph and similarity queries | Neo4j-compatible Bolt driver and Cypher | Low-latency in-memory workloads; check BSL terms with legal before a commercial deployment |
| FalkorDB | Server; SSPL v1 (FalkorDB) | Not confirmed on the repository page; check the docs | Redis protocol, Cypher | Redis-shaped infrastructure already exists; SSPL restricts offering it as a managed service |
| Apache AGE on PostgreSQL | Postgres extension, Apache Software Foundation project (AGE) | No native vector index; the AGE site describes pgvector integration as a proposal | Cypher via cypher() in SQL | One-database requirement and a team that runs Postgres well. Keep embeddings in pgvector and join in SQL |
| Kuzu | Archived; the repository notice names v0.11.3 as the last release, and the last master commit is 10 Oct 2025 (Kuzu) | Full-text and vector indices in last release (v0.11.3) | kuzu 0.11.3 (frozen) | Do not start new work on it. Successor: Ladybug (MIT, pip install ladybug 0.21.2 on PyPI, formerly Kuzu per its README) (Ladybug) |
| SQLite + NetworkX | Open source (check licence) | None; use sqlite-vec from the lab | networkx, sqlite3 | Corpora up to tens of thousands of chunks, single-process, prototypes and offline laptops |
Selection rules:
- Start with SQLite + NetworkX for the spike if the graph fits in memory. Its operations are plain Python, so the first evaluation loop is fast. Migrate once you need concurrent writes, multi-hop Cypher at scale, or ACL-filtered traversal.
- For a customer deployment on private infrastructure, default to Neo4j unless the customer already runs Postgres and the graph is modest, in which case AGE plus pgvector keeps the operational footprint to one database.
- Check licences before procurement: Neo4j Enterprise features, Memgraph BSL, and FalkorDB SSPL each have terms that affect managed offerings.
The builder takes an LLM, a Neo4j driver, an embedder, and a schema with node_types, relationship_types, and patterns. Pass additional_node_types: False explicitly so nodes whose type is outside the schema are dropped. In neo4j-graphrag 1.22.0 this flag defaults to False only when node_types is non-empty, so an empty list silently turns it on. For a local endpoint, pass base_url to OpenAILLM (neo4j-graphrag 1.22.0; other kwargs go to the openai.OpenAI client):
import neo4j
from neo4j_graphrag.llm import OpenAILLM
llm = OpenAILLM(
model_name="Qwen/Qwen3.5-9B",
base_url="http://localhost:8000/v1",
api_key="local",
model_params={"temperature": 0, "response_format": {"type": "json_object"}},
)
driver = neo4j.GraphDatabase.driver("neo4j://localhost:7687", auth=("neo4j", "password"))
read_session = driver.session(default_access_mode=neo4j.READ_ACCESS) # for generated or user-facing queries
default_access_mode is a routing hint for clusters, not a permission boundary. Enforce read-only with a database role that holds only read grants.
4. Frameworks
Frameworks handle extraction, community detection, and query routing so you do not have to. The trade-off is control: each one fixes its own prompts, storage, and query modes. Compare them on your corpus before committing, and check the framework's version in your lockfile, because most of them change prompts and storage formats between releases.
The examples below use Qwen/Qwen3.5-9B as the extraction model. Qwen3.5 thinks by default, and most frameworks do not pass chat_template_kwargs, so turn thinking off on the server instead: vllm serve Qwen/Qwen3.5-9B --default-chat-template-kwargs '{"enable_thinking": false}' (request values still take precedence; vLLM v0.32.0 CLI args).
| Framework | Latest version (as of 2026-10) | Graph store | Query modes | Local OpenAI-compatible support | Note |
|---|---|---|---|---|---|
| Microsoft GraphRAG | 3.3.0 (8 Oct 2026); Python 3.11–3.13 (PyPI) | Parquet files (default), community reports | Local, global, DRIFT, basic (docs) | Yes, api_base on each model block (section below) | Heaviest indexing; the reference for community summaries |
| LightRAG | lightrag-hku 1.5.7 on PyPI (2 Sep 2026); repo at HKUDS | Pluggable (JSON files, Neo4j, Postgres, and others) | Local, global, hybrid, naive, mix | Yes, openai and ollama bindings in the examples | Lighter than GraphRAG; a single LLM for extract and query is the simplest setup |
| nano-graphrag | 0.0.8.2 on PyPI (19 Oct 2024; no newer release); small reimplementation of GraphRAG's core | In-memory / file | Local, global | Yes, OpenAI-compatible function hooks | Read the code, it is short. Good for understanding the pipeline |
| HippoRAG 2 | 2.0.0a5 per the repository README; PyPI's latest is 2.0.0a4 (24 Jun 2025) | Local files plus embeddings | Personalised PageRank over an OpenIE graph | Yes: llm_base_url and embedding_base_url (a5 also has embedding_provider and *_api_key_env) | Strong on multi-hop; pre-release, pin the version |
| LlamaIndex PropertyGraphIndex | llama-index-core 0.14.25 on PyPI | Neo4j and others via property-graph store | Property-graph retrievers, vector and keyword | Yes, through LlamaIndex LLM and embedding classes | Use when the app is already on LlamaIndex |
| Cognee | 1.6.5 (16 Oct 2026) | Pluggable (graph + vector) | remember, recall, improve, forget | Yes, LLM_PROVIDER="custom" with LLM_ENDPOINT (section 4.6) | Agent memory platform |
| Graphiti | graphiti-core 0.31.1 (15 Oct 2026) | Neo4j (default), FalkorDB, Neptune, Kuzu (upstream archived) | Temporal, hybrid search over episodes | Yes, OpenAIGenericClient with LLMConfig(base_url=...) | Temporal knowledge graph for agents; see section 6 |
4.1 Microsoft GraphRAG
GraphRAG builds a hierarchy of communities over the extracted graph, writes an LLM summary per community, and answers global questions by map-reduce over those summaries. Its local search starts from entities and expands their neighbourhood. Configuration is YAML in settings.yaml, with separate chat and embedding model blocks. The model block keys are model_provider, model, auth_method, api_key, and api_base (GraphRAG 3.3.0 config docs and the graphrag-llm 3.3.0 ModelConfig). GraphRAG passes <model_provider>/<model> to LiteLLM with api_base as the endpoint, so a local server is an openai provider plus a base URL (GraphRAG models config, yaml reference):
completion_models:
default_completion_model:
model_provider: openai
model: Qwen/Qwen3.5-9B # served model name
auth_method: api_key
api_key: ${LOCAL_LLM_KEY} # any non-empty value for vLLM/Ollama
api_base: http://localhost:8000/v1 # Ollama: http://localhost:11434/v1
embedding_models:
default_embedding_model:
model_provider: openai
model: Qwen/Qwen3-Embedding-0.6B
auth_method: api_key
api_key: ${LOCAL_LLM_KEY}
api_base: http://localhost:8001/v1
GraphRAG's docs warn that models reached through proxies such as Ollama often return malformed JSON, and that the model must return the structured output GraphRAG expects. Test the served model on a sample before indexing.
Operational rules for GraphRAG on local models:
- Run
graphrag init --root <dir> --forcebetween minor version bumps, as the README advises. It overwrites your configuration and prompts, so back them up first. Then re-index. Do not hand-edit generated prompts across versions. - Use
max_tokensonly as a budget cap, not to control response length: GraphRAG's docs say its prompts control response length. Check the defaults against the local model's context window. - Index a sample corpus first and time each stage. Community summarisation is the largest stage on most corpora.
4.2 LightRAG
LightRAG builds a graph with LLM extraction and answers with entity, relation, and hybrid retrieval. Its examples wire a local Ollama model through lightrag.llm.ollama and an OpenAI-compatible server through lightrag.llm.openai. The OpenAI-compatible completion function openai_complete_if_cache accepts base_url and api_key (checked in lightrag-hku 1.5.7). openai_embed is an EmbeddingFunc; its unwrapped .func accepts base_url and api_key too. The sketch uses the calls from the OpenAI demo: initialize_storages(), ainsert(), and aquery(param=QueryParam(...)):
import asyncio
from functools import partial
from lightrag import LightRAG, QueryParam
from lightrag.llm.openai import openai_complete_if_cache, openai_embed
from lightrag.utils import EmbeddingFunc
BASE = "http://localhost:8000/v1" # vLLM; for Ollama use lightrag.llm.ollama instead
async def llm(prompt, system_prompt=None, history_messages=[], **kw):
return await openai_complete_if_cache(
"Qwen/Qwen3.5-9B", prompt, system_prompt=system_prompt,
history_messages=history_messages, base_url=BASE, api_key="local", **kw,
)
embed = EmbeddingFunc(
embedding_dim=1024, max_token_size=8192,
func=partial(openai_embed.func, model="Qwen/Qwen3-Embedding-0.6B", base_url=BASE, api_key="local"),
)
async def main():
rag = LightRAG(working_dir="./rag_store", llm_model_func=llm, embedding_func=embed)
await rag.initialize_storages()
await rag.ainsert(open("corpus.md", encoding="utf-8").read())
print(await rag.aquery("Which suppliers link Project X to the 2024 audit?", param=QueryParam(mode="hybrid")))
asyncio.run(main())
The embedding_dim must match the served embedding model. A mismatch corrupts the vector store silently on some backends, so check the dimension with one test embedding before indexing.
4.3 nano-graphrag
nano-graphrag is a compact reimplementation of GraphRAG's core loop, and it is the fastest way to see what the larger frameworks do. Its LLM and embedding functions are plain async callables, so pointing it at a local endpoint is the same OpenAI-client wrapper as in 4.2. Use it to learn the pipeline, to prototype a custom extraction prompt, or as the basis for an in-house pipeline. Do not treat it as a drop-in replacement for the production frameworks without reading its current version.
4.4 HippoRAG 2
HippoRAG 2 extracts an OpenIE-style graph and retrieves by personalised PageRank from the query's entities, which suits multi-hop questions. Its constructor takes separate LLM and embedding endpoints, which makes a split serving setup (small model for indexing, larger for answers) easy to express (HippoRAG):
from hipporag import HippoRAG
hipporag = HippoRAG(
save_dir="outputs",
llm_model_name="Qwen/Qwen3.5-9B", # must equal the served model name
llm_base_url="http://localhost:8000/v1",
llm_api_key_env="LOCAL_LLM_KEY",
embedding_model_name="Qwen/Qwen3-Embedding-0.6B",
embedding_provider="openai",
embedding_base_url="http://localhost:8001/v1",
embedding_api_key_env="LOCAL_EMB_KEY",
)
hipporag.index(docs=docs)
results = hipporag.rag_qa(queries=["Which suppliers link Project X to the 2024 audit?"])
The snippet follows the 2.0.0a5 README. PyPI's latest is 2.0.0a4 (24 Jun 2025), and its constructor has no embedding_provider or *_api_key_env arguments (checked in the wheel), so the snippet fails on a plain pip install hipporag. Install a5 from the repository and pin the commit. The README states that persisted indexes are bound to their endpoint and model identity; changing the embedding model requires a re-index.
4.5 LlamaIndex PropertyGraphIndex
Use PropertyGraphIndex when the application is already built on LlamaIndex. Its SchemaLLMPathExtractor takes possible_entities, possible_relations, and a kg_validation_schema that maps allowed relations per entity type; the documented pattern pairs it with a Neo4jPropertyGraphStore and embed_kg_nodes=True (LlamaIndex PropertyGraph guide). Set strict=True so triplets outside the validation schema are dropped rather than kept.
4.6 Cognee and Graphiti
Both are agent-memory systems with a graph under the hood. Section 6 covers Graphiti as agent memory. Cognee's README documents remember and recall as its primary calls. Its .env.template defines the custom-endpoint variables below. The model prefix (openai/) and endpoint pattern follow the template's own examples; this page has not run them end to end:
LLM_PROVIDER="custom"
LLM_MODEL="openai/Qwen/Qwen3.5-9B"
LLM_ENDPOINT="http://localhost:8000/v1"
LLM_API_KEY="local"
EMBEDDING_PROVIDER="custom"
EMBEDDING_MODEL="openai/Qwen/Qwen3-Embedding-0.6B"
EMBEDDING_ENDPOINT="http://localhost:8001/v1"
EMBEDDING_API_KEY="local"
EMBEDDING_DIMENSIONS=1024
The template says EMBEDDING_DIMENSIONS must be set explicitly for custom models, or the vector store shape can mismatch. Use Cognee when the requirement is "persistent memory for agents" rather than "answer questions over a corpus".
5. Query time
Four patterns, chosen by question type. Route between them with a classifier or let the agent choose (section 6).
| Pattern | Starts from | Retrieves | Good for | Cost per query |
|---|---|---|---|---|
| Local search | Entities matched in the question | Entity → neighbourhood (1–2 hops) → source chunks via chunk_ids | Entity-centric and multi-hop facts | Low: graph queries plus a few chunks |
| Global search | Community summaries | Map over summaries, reduce to an answer | "Main themes across the corpus" | High: many LLM calls over summaries |
| Hybrid (vector + BM25 + graph) | Question embedding and terms | Dense and BM25 top-k fused with RRF, then graph expansion of the top entities | Default for mixed questions | Medium |
| Text-to-Cypher | Natural language | LLM-generated Cypher executed on the graph | Ad hoc aggregation with a stable schema | Medium, with the risks below |
5.1 Local search
- Match entities in the question. Use the same normalisation as section 2.5 on the question's noun phrases, plus embedding match against entity names. Cap the matches (top 5).
- Expand one or two hops, with a relation-type allow-list and a row limit.
- Collect
chunk_idsfrom the traversed edges, rank them by how many matched entities they touch, and fetch the chunk text from the document store. - Put the chunks and the traversed triples in the prompt, each with its
[doc_id#chunk_id]id.
5.2 Global search
Global search needs community summaries. Build communities (Leiden is the usual choice; GraphRAG uses it), summarise each community at each level, and answer by mapping the question over the summaries and reducing the partial answers. Cost scales with the number of communities, not with the question, so cache answers to frequent global questions and rebuild summaries only for communities whose members changed.
5.3 Hybrid retrieval
Run dense and BM25 retrieval in parallel, fuse with Reciprocal Rank Fusion (k = 60, as in the lab), then add graph neighbours of the top-ranked entities as extra candidates before reranking. Keep graph expansion as an additive step: it should widen recall, and the reranker should decide what enters the prompt. Measure whether the graph candidates survive reranking; if they never do, the graph is not contributing and the index cost is not justified.
5.4 Text-to-Cypher: risks and guarded templates
Free-form text-to-Cypher is a prompt-injection and data-exposure path. The LLM writes a query from user text, so a crafted question can request writes, unbounded traversals, or properties outside the caller's ACL. Rules:
- Prefer parameterised templates that the LLM fills by choosing a template id and parameters. Free-form generation is a fallback for analysts, not for end users.
- Execute in a read-only session. The Neo4j Python driver takes
default_access_mode=neo4j.READ_ACCESSper session, but that is a routing hint, so enforce read-only with a database role that has only read grants. - Enforce a row limit and a timeout in the driver, and reject queries that contain
CALL,LOAD CSV,apoc., orDELETEbefore execution. - Apply the ACL as a parameter that every template must bind (
$groups), never as a string concatenated into the query.
Template for local search, bound and capped:
MATCH (e:Entity {name_norm: $name})-[r]-(n:Entity)
WHERE type(r) IN $allowed_rels
AND any(g IN $groups WHERE g IN r.acl_groups)
RETURN n.name AS neighbour, type(r) AS relation, r.chunk_ids AS chunk_ids
LIMIT $limit
Pass allowed_rels and limit from code. Do not let the model choose them. If you must allow generated Cypher, run it through an EXPLAIN check that rejects plans without an index seek on the start node.
6. Agentic setup
An agent with a graph and a vector store as tools answers questions by deciding which retrieval to run next, reading the result, and stopping when it has enough evidence. The tools are the interface; keep them narrow and cite everything they return.
6.1 Tools
| Tool | Backed by | Returns | Notes |
|---|---|---|---|
search_entities(query, k) | Graph entity index (name + embedding) | Entity ids, types, names, degree | Entry point for entity questions |
get_neighbors(entity_id, rel_types, limit) | Graph (Cypher template, section 5.4) | Triples with chunk_ids | Bounded by allow-list and limit |
get_chunk(chunk_id) | Document store | Text, doc_id, section path, version | The only source of quotable text |
vector_search(query, k, filters) | Vector index with ACL filter | Chunk ids, scores, snippets | Fallback for prose questions |
Use function tools for a small, fixed set. Use an MCP server when the same tools must serve several agent clients, when the tools live in another team's service, or when you need the protocol's standard transport and auth. The reference MCP memory server (@modelcontextprotocol/server-memory, run with npx -y @modelcontextprotocol/server-memory) exposes a local knowledge graph of entities, relations, and observations with create_entities, search_nodes, open_nodes, and read_graph (MCP memory server). It is a memory store for conversations, not a corpus index with ACLs; do not expose the corpus graph through it.
The MCP Python SDK is at 2.3.0 (2 Oct 2026, Python 3.10+) (PyPI). Check its server API for your version before writing a server; this page does not reproduce it.
6.2 Minimal tool definitions
Plain OpenAI-style function tools, which work against the same local endpoint as the extractor:
TOOLS = [
{"type": "function", "function": {
"name": "search_entities",
"description": "Find graph entities whose names match the query. Use before get_neighbors.",
"parameters": {"type": "object", "properties": {
"query": {"type": "string"},
"k": {"type": "integer", "minimum": 1, "maximum": 10},
}, "required": ["query"]},
}},
{"type": "function", "function": {
"name": "get_neighbors",
"description": "Return relations of an entity with the chunk ids that support them.",
"parameters": {"type": "object", "properties": {
"entity_id": {"type": "string"},
"rel_types": {"type": "array", "items": {"type": "string"}},
"limit": {"type": "integer", "minimum": 1, "maximum": 50},
}, "required": ["entity_id"]},
}},
{"type": "function", "function": {
"name": "get_chunk",
"description": "Return the text of one chunk by id, with doc_id and section. Only source of quotable text.",
"parameters": {"type": "object", "properties": {"chunk_id": {"type": "string"}},
"required": ["chunk_id"]},
}},
{"type": "function", "function": {
"name": "vector_search",
"description": "Semantic search over chunks. Use for prose questions or when entity search returns nothing.",
"parameters": {"type": "object", "properties": {
"query": {"type": "string"}, "k": {"type": "integer", "minimum": 1, "maximum": 20},
}, "required": ["query"]},
}},
]
The server side of each tool applies the ACL filter from the caller's identity, not from the model's arguments. Treat any groups or doc_id parameter the model sends as untrusted input and drop it.
6.3 Loop, budget, citations, stop
Loop rules:
- Budget. Cap tool calls per question (start with 8), total retrieved tokens (start with 12k), and wall-clock time. Enforce in the loop, not in the prompt. When any budget is spent, the agent answers with what it has or says it cannot answer.
- Stop. Stop when the answer's claims are all covered by retrieved chunks, or when the last two tool calls added no new chunk ids (a plateau). A repeated identical call is a stop signal, not a retry.
- Citations. Every factual sentence cites
[doc_id#chunk_id]. The loop validates the citations after generation: each id must be in the set returned byget_chunkduring this run. A citation outside that set fails the answer, which triggers one regeneration and then a refusal. - Abstention. If no chunk supports the question after the budget, return "not found in the indexed sources" and the list of searches tried. Do not let the model answer from parametric memory.
- Logging. Record every tool call, arguments, returned ids, and token counts. This is the trace you need for section 7.
6.4 Graphiti as agent long-term memory
Graphiti stores episodes (conversations, documents, events) and derives temporal entities and facts, so each fact has validity intervals and provenance back to the episode. It is aimed at agent memory that changes over time, not at corpus question answering. Its constructor takes the graph database connection, and it accepts a custom LLM client and embedder. For a local OpenAI-compatible endpoint it provides OpenAIGenericClient with LLMConfig(base_url=...) and an OpenAIEmbedder with its own base_url. Its MCP server in the repository defaults to FalkorDB and also supports Neo4j (Graphiti):
from graphiti_core import Graphiti
from graphiti_core.llm_client.config import LLMConfig
from graphiti_core.llm_client.openai_generic_client import OpenAIGenericClient
from graphiti_core.embedder.openai import OpenAIEmbedder, OpenAIEmbedderConfig
llm = OpenAIGenericClient(config=LLMConfig(
api_key="local", base_url="http://localhost:8000/v1", model="Qwen/Qwen3.5-9B",
))
embedder = OpenAIEmbedder(config=OpenAIEmbedderConfig(
api_key="local", base_url="http://localhost:8001/v1",
embedding_model="Qwen/Qwen3-Embedding-0.6B", embedding_dim=1024,
))
graphiti = Graphiti("bolt://localhost:7687", "neo4j", "password", llm_client=llm, embedder=embedder)
The import paths, the positional Graphiti(uri, user, password, ...) form, and the LLMConfig, OpenAIEmbedderConfig fields (embedding_dim is inherited from EmbedderConfig) match graphiti-core 0.31.1. The newer form passes graph_driver= instead. Use Graphiti for memory that must answer "what was true on date X" and keep the corpus index separate. Mixing corpus facts and conversation facts in one graph makes deletion and access control much harder.
7. Evaluation
Measure each stage separately. A good answer can hide a bad graph, and a good graph can sit behind bad retrieval.
| Layer | Metric | How | Pass bar (starting point, tune to the use case) |
|---|---|---|---|
| Entity extraction | Precision, recall, F1 by type | Hand-label 100–200 chunks stratified by document type; two labellers; measure agreement (Cohen's kappa) first | Precision ≥ 0.85 for the top three types before indexing the corpus |
| Relation extraction | Precision, recall | Same sample, triples labelled with their evidence span | Precision ≥ 0.75; below this, drop the relation type or constrain it |
| Entity resolution | Pairwise precision and recall of merges | Labelled pairs from the candidate band (section 2.5) | Zero cross-type merges; measure merge errors on a sample each release |
| Provenance | Share of edges whose evidence appears verbatim in the cited chunk | Automatic string check, plus a manual sample | 100% verbatim (or fail the extraction batch) |
| Retrieval | recall@k on the graph-needed question subset | Golden set labelled with the chunk ids that answer each question | Beats vector-only recall@10 on the multi-hop slice, or do not ship the graph |
| Answers | Answer correctness; faithfulness (claims supported by cited chunks) | Judge model or human review; sample every release | Faithfulness target set per use case; any unsupported claim is a bug |
| Multi-hop QA | Exact match and F1 on a multi-hop question set | Use public multi-hop datasets (for example HotpotQA, MuSiQue, 2WikiMultiHopQA) plus your own domain set | Report per hop count (1, 2, 3+) |
Practical rules:
- Build the golden set from real user questions. Label each question with its type (single-hop, multi-hop, global, aggregation); the graph only earns its cost on the non-single-hop types.
- Run ablations: vector only, graph only, hybrid. The graph must win on the questions it is meant for, not on average.
- Graph quality checks on every build: node and edge counts by type, share of isolated nodes, share of entities with one mention (often extraction noise), top-degree hub entities (a hub that is a generic word such as "policy" signals a bad type rule), and the number of cross-type merges. Track these as build artefacts and alert on large swings.
- Record the extractor model, prompt version, schema version, and embedding model on the index. A metric change without those four fields cannot be attributed.
8. Hardware and cost
Extraction is throughput-bound and the answer step is latency-bound. Size and schedule them separately.
| Stage | Model class | Serving | Bottleneck |
|---|---|---|---|
| Extraction (per chunk) | Small instruct model (7–8B class) or the NER pre-pass | vLLM with batching; GLiNER/GLiREL on CPU or a small GPU | Tokens per second across the batch |
| Community summaries (GraphRAG) | Small or medium model | vLLM batch job, offline | Total tokens, one-off per build |
| Answers and agent loop | Larger instruct model | vLLM online serving | Time to first token and concurrency |
| Embeddings | 0.6B–4B embedder | TEI or vLLM pooling (see the RAG page) | Batch size |
Guidance:
- Use a small model for extraction and a larger one for answers. Extraction runs once per chunk and must be cheap; answers run per question and benefit from reasoning capacity. Validate the small model with the section 7 extraction metrics; do not assume it.
- Batch extraction on vLLM. Submit the chunk set as concurrent requests, let continuous batching fill the GPU, and enable prefix caching when every request shares a long system prompt (the schema and instructions). The schema-constrained decoding in vLLM adds per-token cost; measure it with and without the schema on your hardware.
- Measure before tuning. Time one build stage by stage, and optimise the largest one. For a typical build the LLM extraction and community summarisation dominate; the graph store and the pre-pass are rarely the bottleneck.
- Plan the budget as tokens. Multiply the cost formulas in section 1 by the measured tokens per second of your configuration to get wall-clock time for a corpus. Re-measure after each model or prompt change.
Best practices
- Start with the question set, not the framework. Write 50 real questions, label their type, and run vector RAG on them. Build the graph only for the types that vector RAG misses.
- Constrain the schema. A small, closed entity and relation vocabulary, enforced by the decoder and by
additional_node_types: False-style settings, beats a large open one. - Keep the graph and the chunks as separate stores. The graph points to chunk ids; the chunk store owns the text, versions, and ACLs. Deletions and re-indexing then touch one table each.
- Version everything in the index. Prompt, schema, extractor model, embedding model, and framework version. A rebuild must be reproducible.
- Resolve entities deliberately. Aliases and type-scoped keys first, similarity second, humans for the band in between.
- Make every edge auditable.
chunk_idsand a verbatimevidencespan on every relation. - Bound every traversal. Allow-listed relation types, a hop limit, and a row limit in every query template.
- Use the graph as an additive candidate source. Rerank graph and vector candidates together and measure which survive.
- Cite from the store, not from the model. The agent may only quote text returned by
get_chunkduring the run.
Pitfalls to design around
- Extraction drift across chunks. The same entity appears as
ACME,Acme Ltd., andthe company. Design normalisation and aliasing before the first build, not after the graph is full of duplicates. - Hub entities. Generic nodes (
the policy,the system) connect everything and flood traversal. Exclude them by type and by degree at query time. - Relation labels that mean nothing.
RELATED_TOis a sink. Each relation type needs a definition in the prompt and a sample where it applies. - Silent schema-violating output. A decoder can produce valid JSON with an empty
entitieslist for a dense chunk. Count empty extractions per document and review the outliers. - Re-indexing by accident. Changing a prompt or a model re-runs extraction on the corpus. Gate prompt changes behind an index version, and budget the rebuild before merging.
- Embedding dimension and model mismatch. Frameworks may store vectors without checking the model. Check dimension and model id at index load and fail closed.
- Text-to-Cypher as an input filter. A generated query can read data outside the caller's scope. Bind the ACL as a parameter in every template and run read-only.
- Deletions that do not reach the graph. A deleted document must remove its chunks and every edge whose
chunk_idsreference it, and the entities left with no support. Test this path with a fixture. - Agent loops that never stop. Enforce call, token, and time budgets in code, and treat repeated calls as a stop signal.
- Pre-release frameworks in production. HippoRAG 2 is pre-release on its README; pin exact versions and re-run the evaluation before each upgrade.
- Archived or restrictively licensed stores. Kuzu is archived (use Ladybug, or another store); check Memgraph BSL and FalkorDB SSPL terms against the customer's offering model.
Definition of done
- Question set labelled by type; vector-only baseline recorded with recall@10 on each type.
- Extraction schema defined in code, validated against Pydantic, and closed (no open entity or relation types).
- Extraction precision and recall measured on a hand-labelled sample, meeting the section 7 bars for the types used.
- Entity resolution rules implemented and tested, including the cross-type merge test.
- Every node and edge carries
chunk_idsanddoc_id; evidence check passes at 100% verbatim. - Graph store chosen with its licence checked; read-only query role for the application.
- Query templates parameterised, bounded, and ACL-bound; text-to-Cypher disabled for end users or behind the guard.
- Hybrid retrieval ablation shows the graph improves the multi-hop slice; otherwise the graph is removed from the pipeline.
- Agent tools implemented with server-side ACL, budgets enforced in code, and citation validation on every answer.
- Deletion and re-indexing tested end to end with a fixture corpus.
- Index metadata records extractor model, prompt and schema version, embedding model, and framework version.
- Build timing recorded per stage, with a cost estimate for the full corpus from measured tokens per second.
Sources
- Microsoft GraphRAG releases (v3.3.0 marked latest): https://github.com/microsoft/graphrag/releases
- graphrag on PyPI (3.3.0, 8 Oct 2026, Python 3.11–3.13): https://pypi.org/project/graphrag/
- GraphRAG README (
graphrag init --forceguidance): https://github.com/microsoft/graphrag - Microsoft GraphRAG documentation (search methods, init guidance): https://microsoft.github.io/graphrag/
- GraphRAG models configuration: https://microsoft.github.io/graphrag/config/models/
- GraphRAG model config source (
graphrag-llm3.3.0,api_base): https://github.com/microsoft/graphrag/tree/main/packages/graphrag-llm - LazyGraphRAG (Microsoft Research, 25 Nov 2024; cost figures are vendor-reported): https://www.microsoft.com/en-us/research/blog/lazygraphrag-setting-a-new-standard-for-quality-and-cost/
- When to use Graphs in RAG (introduces GraphRAG-Bench), arXiv 2506.05690: https://arxiv.org/abs/2506.05690
- LightRAG repository (
lightrag-hku1.5.7 on PyPI): https://github.com/HKUDS/LightRAG - LightRAG OpenAI demo: https://github.com/HKUDS/LightRAG/blob/main/examples/lightrag_openai_demo.py
- LightRAG Ollama demo: https://github.com/HKUDS/LightRAG/blob/main/examples/lightrag_ollama_demo.py
- HippoRAG (2.0.0a5 per README): https://github.com/OSU-NLP-Group/HippoRAG
- LlamaIndex PropertyGraph index guide: https://developers.llamaindex.ai/python/framework/module_guides/indexing/lpg_index_guide/
- Cognee repository (1.6.5 on PyPI): https://github.com/topoteretes/cognee
- Cognee environment template (custom endpoint variables): https://github.com/topoteretes/cognee/blob/main/.env.template
- Graphiti repository (graphiti-core 0.31.1 on PyPI): https://github.com/getzep/graphiti
- neo4j-graphrag on PyPI (1.22.0): https://pypi.org/project/neo4j-graphrag/
- Neo4j Python driver (
default_access_mode): https://pypi.org/project/neo4j/ - neo4j-graphrag KG builder guide: https://neo4j.com/docs/neo4j-graphrag-python/current/user_guide_kg_builder.html
- Memgraph repository (BSL community license, built-in vector indexes): https://github.com/memgraph/memgraph
- FalkorDB repository (SSPL v1): https://github.com/FalkorDB/FalkorDB
- Apache AGE: https://age.apache.org/
- Kuzu repository (archived 10 Oct 2025): https://github.com/kuzudb/kuzu
- Ladybug repository (successor; MIT): https://github.com/ladybugdb/ladybug
- GLiNER: https://github.com/urchade/GLiNER
- GLiNER2 (Fastino): https://github.com/fastino-ai/GLiNER2
- GLiREL: https://github.com/jackboyla/GLiREL
- vLLM structured outputs: https://docs.vllm.ai/en/latest/features/structured_outputs/
- Ollama structured outputs: https://docs.ollama.com/capabilities/structured-outputs
- Ollama OpenAI compatibility: https://docs.ollama.com/api/openai-compatibility
- MCP memory server reference: https://github.com/modelcontextprotocol/servers/tree/main/src/memory
- MCP Python SDK on PyPI (2.3.0): https://pypi.org/project/mcp/
- Anthropic, Contextual Retrieval (cited in RAG Systems): https://www.anthropic.com/news/contextual-retrieval
- Related pages: Lab: Local RAG, RAG for Agents, RAG Systems, Agentic AI Platforms, Model Landscape, Inference Engineering