Ugrás a fő tartalomhoz

GraphRAG and Entity Extraction with Local Models

Advanced

Service: Sovereign & Private AI, RAG Systems · Typical shape: a 1–2 week spike to prove the graph beats vector retrieval on your question set, then a productionised extraction and query tier

This page is the implementation companion to RAG Systems. Read Lab: Local RAG first; this page assumes you already have a chunker, an embedder, and a vector index. The agent side is covered in RAG for Agents and Agentic AI Platforms. Model selection for the extractor and the answerer follows Model Landscape, and the serving tier follows Inference Engineering.

The approach is simple to state: during ingestion, an LLM turns each chunk into entities and relations; the graph is stored with a pointer from every edge back to the chunks that produced it; at query time you walk the graph to find context that a similarity search would miss. Almost all of the difficulty sits in extraction quality, entity resolution, and cost. This page covers those three in that order.


1. When graph RAG beats vector RAG​

Vector RAG retrieves passages that look like the question. GraphRAG retrieves passages that are connected to the entities the question names, and it can answer questions that need a summary across the whole corpus. The two behave differently on three question types:

Question typeExampleVector RAGGraph RAGVerdict
Single-hop lookup"What is the per diem for Berlin?"Strong: one passage contains the answerAdds cost, little gainVector
Multi-hop"Which suppliers of Project X also supplied the 2024 audit findings?"Retrieves the two sides separately, often misses the joinWalks Project X → supplier → audit findingGraph
Global or sensemaking"What are the main risk themes across the 2,000 incident reports?"Returns the 10 most similar reports, not the corpusCommunity summaries answer over the whole graphGraph (global search)
Aggregation"How many contracts name vendor V?"Unreliable: counts only what it retrievesExact with a COUNT over edgesGraph, or a structured query
Fact lookup with paraphrase"When did we move the DR site?"Good with hybrid searchGood only if extraction captured the eventVector or hybrid
Entity-centric corpus (people, parts, contracts, regulations)"Everything we know about component C-17"Scattered hitsOne neighbourhood queryGraph

Graph RAG does not pay off when:

  • The corpus is mostly prose with few recurring named entities (novels, meeting notes without people or systems), so extraction yields a sparse graph.
  • Questions are single-hop and answered by one passage. The graph adds indexing cost and a second failure mode.
  • Entities are ambiguous and no resolution step is possible (for example, "Anna" across unrelated departments with no identifiers).
  • The corpus changes faster than you can afford to re-extract. Incremental extraction is possible but must be designed in.
  • The team cannot run an evaluation set. Without one you cannot tell a good graph from a plausible-looking one.

Evidence is mixed and benchmark-dependent. The GraphRAG-Bench paper "When to use Graphs in RAG" (arXiv 2506.05690) states that recent studies report GraphRAG frequently underperforms vanilla RAG on many real-world tasks, and sets out to measure when graph structure does help (arXiv 2506.05690). Treat any vendor benchmark, including the ones in this page's Sources, as a hypothesis to re-measure on your corpus.

Cost note: indexing is an LLM call per chunk​

Graph construction is paid up front, per chunk, and it is the dominant cost:

extraction calls = C × (1 + g) C = chunks, g = gleaning/re-prompt passes
input tokens = C × (p + t) p = prompt tokens, t = chunk tokens
output tokens = C × o × (1 + g) o = output tokens per call

Worked example with assumed numbers (replace with your measurements): 10,000 chunks, 600 tokens each, a 400-token extraction prompt, 200 output tokens, one pass (g = 0):

QuantityValue
Extraction calls10,000
Input tokens10,000 × (400 + 600) = 10 M
Output tokens10,000 × 200 = 2 M

Two consequences follow. Re-indexing after a prompt or schema change costs the whole corpus again, so version the prompt and schema alongside the graph. And gleaning passes multiply cost by (1 + g) for a recall gain you must measure against your hand-labelled sample (section 7) before you accept it.

Microsoft's LazyGraphRAG moves LLM work to query time: its indexing cost is stated as identical to vector RAG and 0.1% of full GraphRAG indexing, with query-time cost roughly 700× lower than GraphRAG global search for comparable quality (Microsoft Research, Nov 2024). These are vendor-reported figures. If indexing cost is the blocker, evaluate this option before paying for full extraction.


2. Entity and relation extraction with local models​

Three approaches, in the order to try them:

ApproachWhat it isStrengthsWeaknessesUse when
A. Schema-first LLM extractionLocal instruct model, JSON-schema constrained output mapped to Pydantic modelsTyped output, provenance easy, works for any relation setOne LLM call per chunk; drift in entity namingDefault. Start here
B. Small encoder NER pre-passGLiNER (zero-shot span NER) and GLiREL (zero-shot relation classification) on CPU or a small GPUVery cheap, deterministic, fast; labels given at inference timeShallow; relations limited to the candidate labels; no coreferenceHigh volume, fixed entity types, or as a candidate generator that the LLM then verifies
C. Few-shot LLM extractionPrompt with worked examples, free-form or loosely structured outputNo schema work to startOutput parsing failures, inconsistent typesPrototyping only; move to A once the types stabilise

2.1 Schema-first extraction (approach A)​

Define the schema once and use it for three things: the constrained decoder, the validation of the response, and the graph write. Use the OpenAI-compatible response_format with json_schema. vLLM accepts it on its OpenAI-compatible server; the guided_* request fields were removed in v0.12.0, and the native API takes the json key under structured_outputs (vLLM docs). Ollama's /v1/chat/completions maps response_format with type: "json_schema" onto its native format field, as the Ollama source shows (Ollama OpenAI compatibility, ollama/openai/openai.go).

Use temperature=0 for extraction. Constrained decoding guarantees syntax, not correctness: the model can still invent entities or misattribute relations, which is why section 7 measures precision and recall.

2.2 Small encoder NER and relation pre-pass (approach B)​

GLiNER labels spans from a label list you pass at inference time, so the entity types can change without retraining. The current quickstart uses the gliner package and a community checkpoint (GLiNER):

pip install gliner
graphrag/gliner_prepass.py
from gliner import GLiNER

model = GLiNER.from_pretrained("gliner-community/gliner_small-v2.5")
LABELS = ["person", "organization", "product", "location", "regulation"]

def spans(text: str) -> list[dict]:
return model.predict_entities(text, LABELS, threshold=0.5)

Fastino's GLiNER2 is a separate package (gliner2, not gliner). Its latest PyPI release is 2.1.0 (9 Oct 2026). It covers entity extraction and a schema-driven extraction interface, and it needs the local extra for inference (GLiNER2):

pip install "gliner2[local]"
graphrag/gliner2_prepass.py
from gliner2 import AutoExtractor

model = AutoExtractor.from_pretrained("fastino/gliner2.5-base-v1")
result = model.extract_entities(
"Apple CEO Tim Cook announced iPhone 15 in Cupertino yesterday.",
["company", "person", "product", "location"],
)

GLiREL is the relation counterpart. It classifies relations between entity spans you already have, so it needs the NER output as input (GLiREL):

pip install glirel
python -m spacy download en_core_web_sm
graphrag/glirel_prepass.py
from glirel import GLiREL
import spacy

rel_model = GLiREL.from_pretrained("jackboyla/glirel-large-v0")
nlp = spacy.load("en_core_web_sm")

def relations(text: str, ner: list[list], labels: list[str]) -> list[dict]:
tokens = [t.text for t in nlp(text)]
# ner entries: [start_token, end_token_inclusive, type, surface_text]
return rel_model.predict_relations(tokens, labels, threshold=0.0, ner=ner, top_k=1)

Use the pre-pass as a candidate generator: pass its entity spans into the LLM prompt as hints ("these spans were detected; you may use them or add others"), or let the LLM verify the candidate triples. Do not write pre-pass output straight to the graph; its precision on your domain is unknown until measured.

2.3 Few-shot extraction (approach C)​

Few-shot prompts are useful for a day of prototyping and then become a liability: they drift between model versions and make parsing fragile. Keep the examples, but move the output contract into the schema (2.1).

2.4 Runnable schema-first extractor​

This script takes chunks with ids, calls a local OpenAI-compatible endpoint (vLLM or Ollama), validates each response against Pydantic models, and attaches the chunk id to every entity and relation. It is the minimal production shape: schema, validation, provenance, and a count of failures.

graphrag/extract.py
import json
import sys
from typing import Literal

from openai import OpenAI
from pydantic import BaseModel, ValidationError

BASE_URL = "http://localhost:8000/v1" # vLLM; Ollama: http://localhost:11434/v1
MODEL = "Qwen/Qwen3.5-9B" # must match the served model name
client = OpenAI(base_url=BASE_URL, api_key="local") # key is required by the client, not checked locally

EntityType = Literal["Person", "Organization", "Product", "Location", "Regulation", "Concept"]
RelationType = Literal["OWNS", "PART_OF", "DEPENDS_ON", "REGULATES", "WORKS_FOR", "LOCATED_IN", "RELATED_TO"]

class Entity(BaseModel):
name: str
type: EntityType

class Relation(BaseModel):
source: str
relation: RelationType
target: str
evidence: str # verbatim span from the chunk, used for audit and for the eval

class Extraction(BaseModel):
entities: list[Entity]
relations: list[Relation]

SYSTEM = (
"Extract entities and relations from the passage. Use only the listed types. "
"Names must appear verbatim in the passage. Evidence must be a verbatim quote. "
"Return nothing for facts the passage does not state."
)

def extract(text: str) -> Extraction | None:
resp = client.chat.completions.create(
model=MODEL,
temperature=0,
messages=[{"role": "system", "content": SYSTEM}, {"role": "user", "content": text}],
response_format={
"type": "json_schema",
"json_schema": {"name": "extraction", "schema": Extraction.model_json_schema()},
},
# Qwen3.5 thinks by default; extraction does not need it (vLLM; on Ollama use reasoning_effort="none")
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
try:
return Extraction.model_validate_json(resp.choices[0].message.content)
except ValidationError:
return None # real failure point: count and route to a retry queue

def run(chunks: list[dict]) -> tuple[list[dict], list[dict], int]:
ents, rels, failed = [], [], 0
for c in chunks:
ext = extract(c["text"])
if ext is None:
failed += 1
continue
ents += [{**e.model_dump(), "chunk_id": c["chunk_id"], "doc_id": c["doc_id"]} for e in ext.entities]
rels += [{**r.model_dump(), "chunk_id": c["chunk_id"], "doc_id": c["doc_id"]} for r in ext.relations]
return ents, rels, failed

if __name__ == "__main__":
chunks = [json.loads(line) for line in open(sys.argv[1], encoding="utf-8")]
ents, rels, failed = run(chunks)
print(json.dumps({"entities": len(ents), "relations": len(rels), "failed_chunks": failed}))

Input is JSON Lines with chunk_id, doc_id, and text. Run it with python graphrag/extract.py chunks.jsonl. The evidence field is the provenance anchor: the verbatim quote lets you check each edge against its source without re-running the model. Remove the evidence requirement only after the precision check in section 7 passes without it.

Two notes on the snippet. (The Local RAG Lab, Step 7, runs the same pattern end to end on Ollama with gemma4:e4b, including the measured cost of constrained decoding on a CPU.) First, Literal types become enum in the JSON schema, so the decoder cannot emit an out-of-set type. Second, the RelationType set is deliberately small. Every relation type you add multiplies the ways the model can mislabel an edge.

2.5 Entity resolution and deduplication​

Extraction produces Acme GmbH, Acme, ACME Gmbh, and Acme Kft. as four nodes unless you resolve them. Resolution runs in three layers, cheapest first:

  1. Normalisation. Unicode NFKD, strip diacritics for matching (keep the original for display), lowercase, collapse whitespace, remove legal suffixes from a per-language list (gmbh, kft, ltd, inc). Store the result as name_norm.
  2. Alias table. A curated mapping alias_norm -> canonical_id, maintained by hand for high-value entities and filled from identifiers (VAT numbers, product SKUs, employee ids) when present. Aliases beat any similarity score.
  3. Embedding similarity. Embed name plus type and a short description. Merge candidates of the same type above a cosine threshold, and send the borderline band to the LLM or to a human. Pick the threshold on your labelled sample, not by convention.
graphrag/resolve.py
import unicodedata
import re

LEGAL = {"gmbh", "kft", "ltd", "inc", "llc", "zrt", "bt", "nyrt"}

def norm(name: str) -> str:
s = unicodedata.normalize("NFKD", name).encode("ascii", "ignore").decode().lower()
s = re.sub(r"[^\w\s]", " ", s)
return " ".join(w for w in s.split() if w not in LEGAL)

def resolve(ents: list[dict], aliases: dict[str, str]) -> dict[str, str]:
"""Map every (type, name_norm) to a canonical id. Aliases win over normalisation."""
canon: dict[str, str] = {}
for e in ents:
key = norm(e["name"])
canon.setdefault(f'{e["type"]}:{key}', aliases.get(key, f'{e["type"]}:{key}'))
return canon

Rules that prevent the common resolution failures:

  • Never merge across types. Apple the company and apple the fruit are different nodes; the type is part of the key.
  • Keep the surface forms: store every observed name on the canonical node as aliases, so answers can cite the name the source used.
  • Resolution is not idempotent under incremental indexing. Store the mapping and re-run it on new entities only, or every merge decision becomes a migration.

2.6 Provenance​

Every node and edge carries chunk_ids (and doc_id, version). When an edge is merged from several chunks, the list grows; the edge's evidence keeps one quote per chunk. Three consequences:

  • Answers can cite [doc_id#chunk_id] for every fact, including facts that came from graph traversal, not from retrieved text.
  • A document deletion or a new version removes or re-points only the edges whose chunk_ids reference that document. Without this you cannot honour deletions, which is a data-protection requirement in most customer deployments.
  • Evaluation can check each edge against its quote automatically (section 7).

3. Graph construction and storage options​

OptionType and licence (as of 2026-10)Vector indexPython entry pointUse when
Neo4jServer; Community and Enterprise editions (check terms)Native vector indexes in the same databaseneo4j-graphrag 1.22.0 (1 Oct 2026)Default for production graph RAG. Largest ecosystem, mature drivers, built-in KG builder
MemgraphServer; BSL 1.1 for the community code, Memgraph Enterprise License for enterprise features (Memgraph)Built-in vector indexes, hybrid graph and similarity queriesNeo4j-compatible Bolt driver and CypherLow-latency in-memory workloads; check BSL terms with legal before a commercial deployment
FalkorDBServer; SSPL v1 (FalkorDB)Not confirmed on the repository page; check the docsRedis protocol, CypherRedis-shaped infrastructure already exists; SSPL restricts offering it as a managed service
Apache AGE on PostgreSQLPostgres extension, Apache Software Foundation project (AGE)No native vector index; the AGE site describes pgvector integration as a proposalCypher via cypher() in SQLOne-database requirement and a team that runs Postgres well. Keep embeddings in pgvector and join in SQL
KuzuArchived; the repository notice names v0.11.3 as the last release, and the last master commit is 10 Oct 2025 (Kuzu)Full-text and vector indices in last release (v0.11.3)kuzu 0.11.3 (frozen)Do not start new work on it. Successor: Ladybug (MIT, pip install ladybug 0.21.2 on PyPI, formerly Kuzu per its README) (Ladybug)
SQLite + NetworkXOpen source (check licence)None; use sqlite-vec from the labnetworkx, sqlite3Corpora up to tens of thousands of chunks, single-process, prototypes and offline laptops

Selection rules:

  1. Start with SQLite + NetworkX for the spike if the graph fits in memory. Its operations are plain Python, so the first evaluation loop is fast. Migrate once you need concurrent writes, multi-hop Cypher at scale, or ACL-filtered traversal.
  2. For a customer deployment on private infrastructure, default to Neo4j unless the customer already runs Postgres and the graph is modest, in which case AGE plus pgvector keeps the operational footprint to one database.
  3. Check licences before procurement: Neo4j Enterprise features, Memgraph BSL, and FalkorDB SSPL each have terms that affect managed offerings.

The builder takes an LLM, a Neo4j driver, an embedder, and a schema with node_types, relationship_types, and patterns. Pass additional_node_types: False explicitly so nodes whose type is outside the schema are dropped. In neo4j-graphrag 1.22.0 this flag defaults to False only when node_types is non-empty, so an empty list silently turns it on. For a local endpoint, pass base_url to OpenAILLM (neo4j-graphrag 1.22.0; other kwargs go to the openai.OpenAI client):

graphrag/neo4j_kg.py
import neo4j
from neo4j_graphrag.llm import OpenAILLM

llm = OpenAILLM(
model_name="Qwen/Qwen3.5-9B",
base_url="http://localhost:8000/v1",
api_key="local",
model_params={"temperature": 0, "response_format": {"type": "json_object"}},
)
driver = neo4j.GraphDatabase.driver("neo4j://localhost:7687", auth=("neo4j", "password"))
read_session = driver.session(default_access_mode=neo4j.READ_ACCESS) # for generated or user-facing queries

default_access_mode is a routing hint for clusters, not a permission boundary. Enforce read-only with a database role that holds only read grants.


4. Frameworks​

Frameworks handle extraction, community detection, and query routing so you do not have to. The trade-off is control: each one fixes its own prompts, storage, and query modes. Compare them on your corpus before committing, and check the framework's version in your lockfile, because most of them change prompts and storage formats between releases.

The examples below use Qwen/Qwen3.5-9B as the extraction model. Qwen3.5 thinks by default, and most frameworks do not pass chat_template_kwargs, so turn thinking off on the server instead: vllm serve Qwen/Qwen3.5-9B --default-chat-template-kwargs '{"enable_thinking": false}' (request values still take precedence; vLLM v0.32.0 CLI args).

FrameworkLatest version (as of 2026-10)Graph storeQuery modesLocal OpenAI-compatible supportNote
Microsoft GraphRAG3.3.0 (8 Oct 2026); Python 3.11–3.13 (PyPI)Parquet files (default), community reportsLocal, global, DRIFT, basic (docs)Yes, api_base on each model block (section below)Heaviest indexing; the reference for community summaries
LightRAGlightrag-hku 1.5.7 on PyPI (2 Sep 2026); repo at HKUDSPluggable (JSON files, Neo4j, Postgres, and others)Local, global, hybrid, naive, mixYes, openai and ollama bindings in the examplesLighter than GraphRAG; a single LLM for extract and query is the simplest setup
nano-graphrag0.0.8.2 on PyPI (19 Oct 2024; no newer release); small reimplementation of GraphRAG's coreIn-memory / fileLocal, globalYes, OpenAI-compatible function hooksRead the code, it is short. Good for understanding the pipeline
HippoRAG 22.0.0a5 per the repository README; PyPI's latest is 2.0.0a4 (24 Jun 2025)Local files plus embeddingsPersonalised PageRank over an OpenIE graphYes: llm_base_url and embedding_base_url (a5 also has embedding_provider and *_api_key_env)Strong on multi-hop; pre-release, pin the version
LlamaIndex PropertyGraphIndexllama-index-core 0.14.25 on PyPINeo4j and others via property-graph storeProperty-graph retrievers, vector and keywordYes, through LlamaIndex LLM and embedding classesUse when the app is already on LlamaIndex
Cognee1.6.5 (16 Oct 2026)Pluggable (graph + vector)remember, recall, improve, forgetYes, LLM_PROVIDER="custom" with LLM_ENDPOINT (section 4.6)Agent memory platform
Graphitigraphiti-core 0.31.1 (15 Oct 2026)Neo4j (default), FalkorDB, Neptune, Kuzu (upstream archived)Temporal, hybrid search over episodesYes, OpenAIGenericClient with LLMConfig(base_url=...)Temporal knowledge graph for agents; see section 6

4.1 Microsoft GraphRAG​

GraphRAG builds a hierarchy of communities over the extracted graph, writes an LLM summary per community, and answers global questions by map-reduce over those summaries. Its local search starts from entities and expands their neighbourhood. Configuration is YAML in settings.yaml, with separate chat and embedding model blocks. The model block keys are model_provider, model, auth_method, api_key, and api_base (GraphRAG 3.3.0 config docs and the graphrag-llm 3.3.0 ModelConfig). GraphRAG passes <model_provider>/<model> to LiteLLM with api_base as the endpoint, so a local server is an openai provider plus a base URL (GraphRAG models config, yaml reference):

settings.yaml (excerpt)
completion_models:
default_completion_model:
model_provider: openai
model: Qwen/Qwen3.5-9B # served model name
auth_method: api_key
api_key: ${LOCAL_LLM_KEY} # any non-empty value for vLLM/Ollama
api_base: http://localhost:8000/v1 # Ollama: http://localhost:11434/v1

embedding_models:
default_embedding_model:
model_provider: openai
model: Qwen/Qwen3-Embedding-0.6B
auth_method: api_key
api_key: ${LOCAL_LLM_KEY}
api_base: http://localhost:8001/v1

GraphRAG's docs warn that models reached through proxies such as Ollama often return malformed JSON, and that the model must return the structured output GraphRAG expects. Test the served model on a sample before indexing.

Operational rules for GraphRAG on local models:

  • Run graphrag init --root <dir> --force between minor version bumps, as the README advises. It overwrites your configuration and prompts, so back them up first. Then re-index. Do not hand-edit generated prompts across versions.
  • Use max_tokens only as a budget cap, not to control response length: GraphRAG's docs say its prompts control response length. Check the defaults against the local model's context window.
  • Index a sample corpus first and time each stage. Community summarisation is the largest stage on most corpora.

4.2 LightRAG​

LightRAG builds a graph with LLM extraction and answers with entity, relation, and hybrid retrieval. Its examples wire a local Ollama model through lightrag.llm.ollama and an OpenAI-compatible server through lightrag.llm.openai. The OpenAI-compatible completion function openai_complete_if_cache accepts base_url and api_key (checked in lightrag-hku 1.5.7). openai_embed is an EmbeddingFunc; its unwrapped .func accepts base_url and api_key too. The sketch uses the calls from the OpenAI demo: initialize_storages(), ainsert(), and aquery(param=QueryParam(...)):

graphrag/lightrag_local.py
import asyncio
from functools import partial

from lightrag import LightRAG, QueryParam
from lightrag.llm.openai import openai_complete_if_cache, openai_embed
from lightrag.utils import EmbeddingFunc

BASE = "http://localhost:8000/v1" # vLLM; for Ollama use lightrag.llm.ollama instead

async def llm(prompt, system_prompt=None, history_messages=[], **kw):
return await openai_complete_if_cache(
"Qwen/Qwen3.5-9B", prompt, system_prompt=system_prompt,
history_messages=history_messages, base_url=BASE, api_key="local", **kw,
)

embed = EmbeddingFunc(
embedding_dim=1024, max_token_size=8192,
func=partial(openai_embed.func, model="Qwen/Qwen3-Embedding-0.6B", base_url=BASE, api_key="local"),
)

async def main():
rag = LightRAG(working_dir="./rag_store", llm_model_func=llm, embedding_func=embed)
await rag.initialize_storages()
await rag.ainsert(open("corpus.md", encoding="utf-8").read())
print(await rag.aquery("Which suppliers link Project X to the 2024 audit?", param=QueryParam(mode="hybrid")))

asyncio.run(main())

The embedding_dim must match the served embedding model. A mismatch corrupts the vector store silently on some backends, so check the dimension with one test embedding before indexing.

4.3 nano-graphrag​

nano-graphrag is a compact reimplementation of GraphRAG's core loop, and it is the fastest way to see what the larger frameworks do. Its LLM and embedding functions are plain async callables, so pointing it at a local endpoint is the same OpenAI-client wrapper as in 4.2. Use it to learn the pipeline, to prototype a custom extraction prompt, or as the basis for an in-house pipeline. Do not treat it as a drop-in replacement for the production frameworks without reading its current version.

4.4 HippoRAG 2​

HippoRAG 2 extracts an OpenIE-style graph and retrieves by personalised PageRank from the query's entities, which suits multi-hop questions. Its constructor takes separate LLM and embedding endpoints, which makes a split serving setup (small model for indexing, larger for answers) easy to express (HippoRAG):

graphrag/hipporag_local.py
from hipporag import HippoRAG

hipporag = HippoRAG(
save_dir="outputs",
llm_model_name="Qwen/Qwen3.5-9B", # must equal the served model name
llm_base_url="http://localhost:8000/v1",
llm_api_key_env="LOCAL_LLM_KEY",
embedding_model_name="Qwen/Qwen3-Embedding-0.6B",
embedding_provider="openai",
embedding_base_url="http://localhost:8001/v1",
embedding_api_key_env="LOCAL_EMB_KEY",
)
hipporag.index(docs=docs)
results = hipporag.rag_qa(queries=["Which suppliers link Project X to the 2024 audit?"])

The snippet follows the 2.0.0a5 README. PyPI's latest is 2.0.0a4 (24 Jun 2025), and its constructor has no embedding_provider or *_api_key_env arguments (checked in the wheel), so the snippet fails on a plain pip install hipporag. Install a5 from the repository and pin the commit. The README states that persisted indexes are bound to their endpoint and model identity; changing the embedding model requires a re-index.

4.5 LlamaIndex PropertyGraphIndex​

Use PropertyGraphIndex when the application is already built on LlamaIndex. Its SchemaLLMPathExtractor takes possible_entities, possible_relations, and a kg_validation_schema that maps allowed relations per entity type; the documented pattern pairs it with a Neo4jPropertyGraphStore and embed_kg_nodes=True (LlamaIndex PropertyGraph guide). Set strict=True so triplets outside the validation schema are dropped rather than kept.

4.6 Cognee and Graphiti​

Both are agent-memory systems with a graph under the hood. Section 6 covers Graphiti as agent memory. Cognee's README documents remember and recall as its primary calls. Its .env.template defines the custom-endpoint variables below. The model prefix (openai/) and endpoint pattern follow the template's own examples; this page has not run them end to end:

cognee .env (custom OpenAI-compatible endpoint, unverified end to end)
LLM_PROVIDER="custom"
LLM_MODEL="openai/Qwen/Qwen3.5-9B"
LLM_ENDPOINT="http://localhost:8000/v1"
LLM_API_KEY="local"
EMBEDDING_PROVIDER="custom"
EMBEDDING_MODEL="openai/Qwen/Qwen3-Embedding-0.6B"
EMBEDDING_ENDPOINT="http://localhost:8001/v1"
EMBEDDING_API_KEY="local"
EMBEDDING_DIMENSIONS=1024

The template says EMBEDDING_DIMENSIONS must be set explicitly for custom models, or the vector store shape can mismatch. Use Cognee when the requirement is "persistent memory for agents" rather than "answer questions over a corpus".


5. Query time​

Four patterns, chosen by question type. Route between them with a classifier or let the agent choose (section 6).

PatternStarts fromRetrievesGood forCost per query
Local searchEntities matched in the questionEntity → neighbourhood (1–2 hops) → source chunks via chunk_idsEntity-centric and multi-hop factsLow: graph queries plus a few chunks
Global searchCommunity summariesMap over summaries, reduce to an answer"Main themes across the corpus"High: many LLM calls over summaries
Hybrid (vector + BM25 + graph)Question embedding and termsDense and BM25 top-k fused with RRF, then graph expansion of the top entitiesDefault for mixed questionsMedium
Text-to-CypherNatural languageLLM-generated Cypher executed on the graphAd hoc aggregation with a stable schemaMedium, with the risks below
  1. Match entities in the question. Use the same normalisation as section 2.5 on the question's noun phrases, plus embedding match against entity names. Cap the matches (top 5).
  2. Expand one or two hops, with a relation-type allow-list and a row limit.
  3. Collect chunk_ids from the traversed edges, rank them by how many matched entities they touch, and fetch the chunk text from the document store.
  4. Put the chunks and the traversed triples in the prompt, each with its [doc_id#chunk_id] id.

Global search needs community summaries. Build communities (Leiden is the usual choice; GraphRAG uses it), summarise each community at each level, and answer by mapping the question over the summaries and reducing the partial answers. Cost scales with the number of communities, not with the question, so cache answers to frequent global questions and rebuild summaries only for communities whose members changed.

5.3 Hybrid retrieval​

Run dense and BM25 retrieval in parallel, fuse with Reciprocal Rank Fusion (k = 60, as in the lab), then add graph neighbours of the top-ranked entities as extra candidates before reranking. Keep graph expansion as an additive step: it should widen recall, and the reranker should decide what enters the prompt. Measure whether the graph candidates survive reranking; if they never do, the graph is not contributing and the index cost is not justified.

5.4 Text-to-Cypher: risks and guarded templates​

Free-form text-to-Cypher is a prompt-injection and data-exposure path. The LLM writes a query from user text, so a crafted question can request writes, unbounded traversals, or properties outside the caller's ACL. Rules:

  • Prefer parameterised templates that the LLM fills by choosing a template id and parameters. Free-form generation is a fallback for analysts, not for end users.
  • Execute in a read-only session. The Neo4j Python driver takes default_access_mode=neo4j.READ_ACCESS per session, but that is a routing hint, so enforce read-only with a database role that has only read grants.
  • Enforce a row limit and a timeout in the driver, and reject queries that contain CALL, LOAD CSV, apoc., or DELETE before execution.
  • Apply the ACL as a parameter that every template must bind ($groups), never as a string concatenated into the query.

Template for local search, bound and capped:

MATCH (e:Entity {name_norm: $name})-[r]-(n:Entity)
WHERE type(r) IN $allowed_rels
AND any(g IN $groups WHERE g IN r.acl_groups)
RETURN n.name AS neighbour, type(r) AS relation, r.chunk_ids AS chunk_ids
LIMIT $limit

Pass allowed_rels and limit from code. Do not let the model choose them. If you must allow generated Cypher, run it through an EXPLAIN check that rejects plans without an index seek on the start node.


6. Agentic setup​

An agent with a graph and a vector store as tools answers questions by deciding which retrieval to run next, reading the result, and stopping when it has enough evidence. The tools are the interface; keep them narrow and cite everything they return.

6.1 Tools​

ToolBacked byReturnsNotes
search_entities(query, k)Graph entity index (name + embedding)Entity ids, types, names, degreeEntry point for entity questions
get_neighbors(entity_id, rel_types, limit)Graph (Cypher template, section 5.4)Triples with chunk_idsBounded by allow-list and limit
get_chunk(chunk_id)Document storeText, doc_id, section path, versionThe only source of quotable text
vector_search(query, k, filters)Vector index with ACL filterChunk ids, scores, snippetsFallback for prose questions

Use function tools for a small, fixed set. Use an MCP server when the same tools must serve several agent clients, when the tools live in another team's service, or when you need the protocol's standard transport and auth. The reference MCP memory server (@modelcontextprotocol/server-memory, run with npx -y @modelcontextprotocol/server-memory) exposes a local knowledge graph of entities, relations, and observations with create_entities, search_nodes, open_nodes, and read_graph (MCP memory server). It is a memory store for conversations, not a corpus index with ACLs; do not expose the corpus graph through it.

The MCP Python SDK is at 2.3.0 (2 Oct 2026, Python 3.10+) (PyPI). Check its server API for your version before writing a server; this page does not reproduce it.

6.2 Minimal tool definitions​

Plain OpenAI-style function tools, which work against the same local endpoint as the extractor:

graphrag/agent_tools.py
TOOLS = [
{"type": "function", "function": {
"name": "search_entities",
"description": "Find graph entities whose names match the query. Use before get_neighbors.",
"parameters": {"type": "object", "properties": {
"query": {"type": "string"},
"k": {"type": "integer", "minimum": 1, "maximum": 10},
}, "required": ["query"]},
}},
{"type": "function", "function": {
"name": "get_neighbors",
"description": "Return relations of an entity with the chunk ids that support them.",
"parameters": {"type": "object", "properties": {
"entity_id": {"type": "string"},
"rel_types": {"type": "array", "items": {"type": "string"}},
"limit": {"type": "integer", "minimum": 1, "maximum": 50},
}, "required": ["entity_id"]},
}},
{"type": "function", "function": {
"name": "get_chunk",
"description": "Return the text of one chunk by id, with doc_id and section. Only source of quotable text.",
"parameters": {"type": "object", "properties": {"chunk_id": {"type": "string"}},
"required": ["chunk_id"]},
}},
{"type": "function", "function": {
"name": "vector_search",
"description": "Semantic search over chunks. Use for prose questions or when entity search returns nothing.",
"parameters": {"type": "object", "properties": {
"query": {"type": "string"}, "k": {"type": "integer", "minimum": 1, "maximum": 20},
}, "required": ["query"]},
}},
]

The server side of each tool applies the ACL filter from the caller's identity, not from the model's arguments. Treat any groups or doc_id parameter the model sends as untrusted input and drop it.

6.3 Loop, budget, citations, stop​

Loop rules:

  • Budget. Cap tool calls per question (start with 8), total retrieved tokens (start with 12k), and wall-clock time. Enforce in the loop, not in the prompt. When any budget is spent, the agent answers with what it has or says it cannot answer.
  • Stop. Stop when the answer's claims are all covered by retrieved chunks, or when the last two tool calls added no new chunk ids (a plateau). A repeated identical call is a stop signal, not a retry.
  • Citations. Every factual sentence cites [doc_id#chunk_id]. The loop validates the citations after generation: each id must be in the set returned by get_chunk during this run. A citation outside that set fails the answer, which triggers one regeneration and then a refusal.
  • Abstention. If no chunk supports the question after the budget, return "not found in the indexed sources" and the list of searches tried. Do not let the model answer from parametric memory.
  • Logging. Record every tool call, arguments, returned ids, and token counts. This is the trace you need for section 7.

6.4 Graphiti as agent long-term memory​

Graphiti stores episodes (conversations, documents, events) and derives temporal entities and facts, so each fact has validity intervals and provenance back to the episode. It is aimed at agent memory that changes over time, not at corpus question answering. Its constructor takes the graph database connection, and it accepts a custom LLM client and embedder. For a local OpenAI-compatible endpoint it provides OpenAIGenericClient with LLMConfig(base_url=...) and an OpenAIEmbedder with its own base_url. Its MCP server in the repository defaults to FalkorDB and also supports Neo4j (Graphiti):

graphrag/graphiti_memory.py
from graphiti_core import Graphiti
from graphiti_core.llm_client.config import LLMConfig
from graphiti_core.llm_client.openai_generic_client import OpenAIGenericClient
from graphiti_core.embedder.openai import OpenAIEmbedder, OpenAIEmbedderConfig

llm = OpenAIGenericClient(config=LLMConfig(
api_key="local", base_url="http://localhost:8000/v1", model="Qwen/Qwen3.5-9B",
))
embedder = OpenAIEmbedder(config=OpenAIEmbedderConfig(
api_key="local", base_url="http://localhost:8001/v1",
embedding_model="Qwen/Qwen3-Embedding-0.6B", embedding_dim=1024,
))
graphiti = Graphiti("bolt://localhost:7687", "neo4j", "password", llm_client=llm, embedder=embedder)

The import paths, the positional Graphiti(uri, user, password, ...) form, and the LLMConfig, OpenAIEmbedderConfig fields (embedding_dim is inherited from EmbedderConfig) match graphiti-core 0.31.1. The newer form passes graph_driver= instead. Use Graphiti for memory that must answer "what was true on date X" and keep the corpus index separate. Mixing corpus facts and conversation facts in one graph makes deletion and access control much harder.


7. Evaluation​

Measure each stage separately. A good answer can hide a bad graph, and a good graph can sit behind bad retrieval.

LayerMetricHowPass bar (starting point, tune to the use case)
Entity extractionPrecision, recall, F1 by typeHand-label 100–200 chunks stratified by document type; two labellers; measure agreement (Cohen's kappa) firstPrecision ≥ 0.85 for the top three types before indexing the corpus
Relation extractionPrecision, recallSame sample, triples labelled with their evidence spanPrecision ≥ 0.75; below this, drop the relation type or constrain it
Entity resolutionPairwise precision and recall of mergesLabelled pairs from the candidate band (section 2.5)Zero cross-type merges; measure merge errors on a sample each release
ProvenanceShare of edges whose evidence appears verbatim in the cited chunkAutomatic string check, plus a manual sample100% verbatim (or fail the extraction batch)
Retrievalrecall@k on the graph-needed question subsetGolden set labelled with the chunk ids that answer each questionBeats vector-only recall@10 on the multi-hop slice, or do not ship the graph
AnswersAnswer correctness; faithfulness (claims supported by cited chunks)Judge model or human review; sample every releaseFaithfulness target set per use case; any unsupported claim is a bug
Multi-hop QAExact match and F1 on a multi-hop question setUse public multi-hop datasets (for example HotpotQA, MuSiQue, 2WikiMultiHopQA) plus your own domain setReport per hop count (1, 2, 3+)

Practical rules:

  • Build the golden set from real user questions. Label each question with its type (single-hop, multi-hop, global, aggregation); the graph only earns its cost on the non-single-hop types.
  • Run ablations: vector only, graph only, hybrid. The graph must win on the questions it is meant for, not on average.
  • Graph quality checks on every build: node and edge counts by type, share of isolated nodes, share of entities with one mention (often extraction noise), top-degree hub entities (a hub that is a generic word such as "policy" signals a bad type rule), and the number of cross-type merges. Track these as build artefacts and alert on large swings.
  • Record the extractor model, prompt version, schema version, and embedding model on the index. A metric change without those four fields cannot be attributed.

8. Hardware and cost​

Extraction is throughput-bound and the answer step is latency-bound. Size and schedule them separately.

StageModel classServingBottleneck
Extraction (per chunk)Small instruct model (7–8B class) or the NER pre-passvLLM with batching; GLiNER/GLiREL on CPU or a small GPUTokens per second across the batch
Community summaries (GraphRAG)Small or medium modelvLLM batch job, offlineTotal tokens, one-off per build
Answers and agent loopLarger instruct modelvLLM online servingTime to first token and concurrency
Embeddings0.6B–4B embedderTEI or vLLM pooling (see the RAG page)Batch size

Guidance:

  • Use a small model for extraction and a larger one for answers. Extraction runs once per chunk and must be cheap; answers run per question and benefit from reasoning capacity. Validate the small model with the section 7 extraction metrics; do not assume it.
  • Batch extraction on vLLM. Submit the chunk set as concurrent requests, let continuous batching fill the GPU, and enable prefix caching when every request shares a long system prompt (the schema and instructions). The schema-constrained decoding in vLLM adds per-token cost; measure it with and without the schema on your hardware.
  • Measure before tuning. Time one build stage by stage, and optimise the largest one. For a typical build the LLM extraction and community summarisation dominate; the graph store and the pre-pass are rarely the bottleneck.
  • Plan the budget as tokens. Multiply the cost formulas in section 1 by the measured tokens per second of your configuration to get wall-clock time for a corpus. Re-measure after each model or prompt change.

Best practices​

  • Start with the question set, not the framework. Write 50 real questions, label their type, and run vector RAG on them. Build the graph only for the types that vector RAG misses.
  • Constrain the schema. A small, closed entity and relation vocabulary, enforced by the decoder and by additional_node_types: False-style settings, beats a large open one.
  • Keep the graph and the chunks as separate stores. The graph points to chunk ids; the chunk store owns the text, versions, and ACLs. Deletions and re-indexing then touch one table each.
  • Version everything in the index. Prompt, schema, extractor model, embedding model, and framework version. A rebuild must be reproducible.
  • Resolve entities deliberately. Aliases and type-scoped keys first, similarity second, humans for the band in between.
  • Make every edge auditable. chunk_ids and a verbatim evidence span on every relation.
  • Bound every traversal. Allow-listed relation types, a hop limit, and a row limit in every query template.
  • Use the graph as an additive candidate source. Rerank graph and vector candidates together and measure which survive.
  • Cite from the store, not from the model. The agent may only quote text returned by get_chunk during the run.

Pitfalls to design around​

  • Extraction drift across chunks. The same entity appears as ACME, Acme Ltd., and the company. Design normalisation and aliasing before the first build, not after the graph is full of duplicates.
  • Hub entities. Generic nodes (the policy, the system) connect everything and flood traversal. Exclude them by type and by degree at query time.
  • Relation labels that mean nothing. RELATED_TO is a sink. Each relation type needs a definition in the prompt and a sample where it applies.
  • Silent schema-violating output. A decoder can produce valid JSON with an empty entities list for a dense chunk. Count empty extractions per document and review the outliers.
  • Re-indexing by accident. Changing a prompt or a model re-runs extraction on the corpus. Gate prompt changes behind an index version, and budget the rebuild before merging.
  • Embedding dimension and model mismatch. Frameworks may store vectors without checking the model. Check dimension and model id at index load and fail closed.
  • Text-to-Cypher as an input filter. A generated query can read data outside the caller's scope. Bind the ACL as a parameter in every template and run read-only.
  • Deletions that do not reach the graph. A deleted document must remove its chunks and every edge whose chunk_ids reference it, and the entities left with no support. Test this path with a fixture.
  • Agent loops that never stop. Enforce call, token, and time budgets in code, and treat repeated calls as a stop signal.
  • Pre-release frameworks in production. HippoRAG 2 is pre-release on its README; pin exact versions and re-run the evaluation before each upgrade.
  • Archived or restrictively licensed stores. Kuzu is archived (use Ladybug, or another store); check Memgraph BSL and FalkorDB SSPL terms against the customer's offering model.

Definition of done​

  • Question set labelled by type; vector-only baseline recorded with recall@10 on each type.
  • Extraction schema defined in code, validated against Pydantic, and closed (no open entity or relation types).
  • Extraction precision and recall measured on a hand-labelled sample, meeting the section 7 bars for the types used.
  • Entity resolution rules implemented and tested, including the cross-type merge test.
  • Every node and edge carries chunk_ids and doc_id; evidence check passes at 100% verbatim.
  • Graph store chosen with its licence checked; read-only query role for the application.
  • Query templates parameterised, bounded, and ACL-bound; text-to-Cypher disabled for end users or behind the guard.
  • Hybrid retrieval ablation shows the graph improves the multi-hop slice; otherwise the graph is removed from the pipeline.
  • Agent tools implemented with server-side ACL, budgets enforced in code, and citation validation on every answer.
  • Deletion and re-indexing tested end to end with a fixture corpus.
  • Index metadata records extractor model, prompt and schema version, embedding model, and framework version.
  • Build timing recorded per stage, with a cost estimate for the full corpus from measured tokens per second.

Sources​