Ugrás a fő tartalomhoz

RAG Systems on Private Infrastructure

Basic Advanced

Service: RAG Systems on Private Infrastructure · Typical shape: 4–10 week build after a PoC, 2–4 engineers, deployed on-premise, OpenStack, or STACKIT

Retrieval-Augmented Generation answers questions from the customer's own documents, with citations, without any data leaving their infrastructure. Answer quality is decided mostly by retrieval — parsing, chunking, embeddings, hybrid search, reranking, and permissions — so that is where this page spends its time. For a laptop-sized version you can build end to end, do the Local RAG Lab.


What we deliver​

  • An ingestion pipeline that parses the customer's real formats (scanned PDFs, Office files, HTML, wikis, tickets) into structured, chunked, embedded records with source metadata and ACLs.
  • A retrieval service: hybrid search (BM25 + dense), metadata and permission filters, reranking, and query rewriting, behind one API.
  • A generation service that answers only from retrieved context, cites sources, and declines when the context does not support an answer.
  • Incremental indexing that keeps the index fresh as sources change, including deletions.
  • An evaluation suite (retrieval recall@k, faithfulness, answer correctness) running in CI, and dashboards in production.
  • Everything as code: Helm charts, OpenTofu modules, CI pipelines — see How We Build.

Reference architecture​

All components run inside the customer's boundary: GPU nodes for the LLM, a smaller GPU or CPU pool for embeddings and reranking, and a database tier for the index. See Private & Sovereign AI for the platform and Serving at Scale for the inference tier.


Implementation steps​

1. Ingestion and parsing​

Parsing quality caps everything downstream: a table flattened into word soup cannot be retrieved correctly no matter how good the embeddings are.

ParserStrengthsLicenceUse when
Docling (v2.140, as of 2026-10)Layout analysis, table structure, reading order, OCR, unified DoclingDocument for PDF/DOCX/PPTX/XLSX/HTML/images; built-in chunkers; optional Granite-Docling VLM pipelineMITDefault. Runs fully offline on CPU or GPU
Unstructured (OSS 0.27)Very broad format coverage (email, EPUB, many Office variants), element typingApache-2.0Long-tail formats, email archives
Marker (v2.0)High-quality PDF → Markdown, equationsCode Apache-2.0; model weights under a modified OpenRAIL-M with commercial thresholds (free below USD 5M funding/revenue)Check the weights licence before customer use
MinerU (4.0)Strong on complex scientific/Chinese layoutsApache-2.0 plus terms: commercial licence above 100M MAU or USD 20M monthly revenue; attribution for online servicesReview the licence terms first
ingest/parse.py
from docling.document_converter import DocumentConverter
from docling.chunking import HybridChunker
from docling_core.transforms.chunker.tokenizer.huggingface import HuggingFaceTokenizer
from transformers import AutoTokenizer

EMBED_MODEL_ID = "Qwen/Qwen3-Embedding-0.6B"
tokenizer = HuggingFaceTokenizer(tokenizer=AutoTokenizer.from_pretrained(EMBED_MODEL_ID), max_tokens=512)
chunker = HybridChunker(tokenizer=tokenizer, merge_peers=True)
converter = DocumentConverter()

def parse(path: str, meta: dict) -> list[dict]:
doc = converter.convert(source=path).document
return [
{
"text": chunk.text,
"embed_text": chunker.contextualize(chunk=chunk), # prepends headings/captions
"headings": chunk.meta.headings,
**meta, # source_uri, doc_id, version, acl_groups, lang, updated_at
}
for chunk in chunker.chunk(dl_doc=doc)
]

Practical rules:

  • Keep the parsed document (Docling JSON) in object storage next to the original. Re-chunking or re-embedding then never requires re-parsing.
  • Capture metadata at ingestion: source_uri, doc_id, version/hash, title, section path, page, lang, updated_at, acl_groups. You cannot filter or cite on what you did not store.
  • Route scanned documents through OCR explicitly and record OCR confidence; low-confidence pages go to a review queue.

2. Chunking strategy​

StrategyHowWhen
Structure-aware (default)Split on document structure (headings, list items, tables), then merge/split to a token budget; prepend heading pathMost business documents. Docling's HybridChunker does this: hierarchical split, token-aware split of oversized chunks, merge of small peers under the same heading, table headers repeated per split
Contextual retrievalAn LLM writes 50–100 tokens situating each chunk in its document; prepend to the chunk before embedding and BM25 indexingDocuments where chunks lose meaning out of context ("the company's revenue grew 3 %"). Anthropic measured a 49 % reduction in failed retrievals with contextual embeddings + contextual BM25, and 67 % with reranking added (Anthropic, 2024)
Late chunkingEmbed the whole (long) document once with a long-context embedding model, then pool token embeddings per chunkLong, coherent documents where cross-chunk references matter; needs a long-context embedder and custom pooling (Günther et al.)
Parent–childRetrieve small chunks, pass their parent section to the LLMPrecise matching but the answer needs surrounding context
Fixed-size with overlapN tokens, M overlapBaseline only

Contextual retrieval on private infrastructure uses the local LLM with prefix caching (vLLM --enable-prefix-caching) so the document is processed once per document, not once per chunk:

ingest/contextualise.py
CONTEXT_PROMPT = """<document>
{doc}
</document>
Here is the chunk we want to situate within the whole document:
<chunk>
{chunk}
</chunk>
Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk. Answer only with the succinct context and nothing else."""

def contextualise(client, model: str, doc_md: str, chunks: list[dict]) -> list[dict]:
for c in chunks:
r = client.chat.completions.create(
model=model, temperature=0, max_tokens=120,
messages=[{"role": "user", "content": CONTEXT_PROMPT.format(doc=doc_md, chunk=c["text"])}],
)
c["embed_text"] = f'{r.choices[0].message.content}\n\n{c["embed_text"]}'
return chunks

Start at 300–600 tokens per chunk (rule of thumb), then tune chunk size against recall@k on your golden set — it is an empirical parameter, not a constant.

3. Embedding model​

Choose by measured recall on the customer's corpus and languages, using the MTEB leaderboard only to build the shortlist. MTEB versions (English v1, v2, multilingual MMTEB) are not comparable to each other; compare within one leaderboard view.

Model (as of 2026-10)SizeLicenceNotes
Qwen3-Embedding 0.6B / 4B / 8B0.6–8BApache-2.08B scored 70.58 on MTEB multilingual at release (June 2025, #1 then); 32k context; Matryoshka dims; instruction-prefixed queries. 0.6B is our default for cost/quality
Harrier-OSS-v1 270M / 0.6B / 27B0.27–27BMITMicrosoft, 2026; model card reports MTEB v2 66.5 / 69.0 / 74.3; 32k context; Hungarian listed among supported languages
BGE-M3568MMITDense + sparse + multi-vector from one model; 100+ languages; mature, cheap on CPU
EmbeddingGemma 2 (Sep 2026)740M (270M text)Apache-2.0Multimodal (text, code, image, video, audio); 768-d with MRL down to 128; 8K context; phone and laptop class
granite-embedding-311m-multilingual-r2311MApache-2.0IBM, 2026; enterprise-friendly licence
jina-embeddings-v5-text-small677MCC-BY-NC-4.0Strong scores; non-commercial weights — needs a commercial licence for customer work

Hungarian: all models above claim broad multilingual coverage, but Hungarian is a low-resource, agglutinative language. Build a 50–100 query Hungarian slice of the golden set and measure recall@10 per candidate; also test cross-lingual retrieval (Hungarian question, English document) if the corpus is mixed.

Serve embeddings with Text Embeddings Inference (v1.9, supports Qwen3 and Gemma3 embedders and XLM-RoBERTa rerankers) or vLLM pooling models:

docker run --gpus all -p 8080:80 -v $PWD/models:/data \
ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 --model-id Qwen/Qwen3-Embedding-0.6B
docker run --gpus all -p 8081:80 -v $PWD/models:/data \
ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 --model-id BAAI/bge-reranker-v2-m3

Record the embedding model ID and revision on every vector. Changing the model means a full re-embed into a new index and an alias switch.

4. Vector database​

pgvectorQdrantWeaviateOpenSearch
Version (as of 2026-10)0.8.71.191.403.9
LicencePostgreSQLApache-2.0BSD-3 core; some modules under the Weaviate LicenseApache-2.0
Hybrid searchCombine with Postgres FTS (or ParadeDB pg_search, AGPL) and RRF in SQLNative: dense + sparse prefetch, RRF / DBSF fusion, multi-stage rerankNative hybrid query (BM25 + vector, fusion + alpha)Native hybrid query with normalization or RRF (score-ranker-processor, since 2.19)
FilteringSQL WHERE; iterative index scans (0.8+) avoid empty results after filteringPayload indexes, filter-aware HNSWPre-filtering with inverted indexEfficient k-NN filtering, document-level security
Multi-tenancySchemas / RLS / partitionsPayload-based tenancy, shard keysNative multi-tenancyIndex per tenant or DLS
OpsYour existing Postgres (backups, HA, RLS)Single binary, Helm chart, snapshotsHelm, modulesCluster ops; built-in BM25 analyzers include Hungarian
Choose whenModest corpus (rule of thumb: < ~10M chunks), team already runs Postgres, ACLs via RLSDefault for dedicated vector workloads and large corporaBuilt-in modules/multi-tenancy are wantedCustomer already runs OpenSearch/Elastic, or keyword search quality is critical

Our default: pgvector when the corpus is modest and Postgres is already operated; Qdrant otherwise.

5. Hybrid search with reciprocal rank fusion​

Dense retrieval misses exact identifiers (contract numbers, error codes, product SKUs); BM25 misses paraphrases. Run both and fuse with RRF, score(d) = Σ 1 / (k + rank_i(d)) with k ≈ 60 (Cormack et al., 2009).

retrieval/hybrid.sql (pgvector + Postgres FTS, Hungarian stemming)
CREATE TABLE chunks (
id bigserial PRIMARY KEY,
doc_id text NOT NULL,
content text NOT NULL,
embedding halfvec(1024) NOT NULL,
acl_groups text[] NOT NULL,
lang text NOT NULL,
updated_at timestamptz NOT NULL,
tsv tsvector GENERATED ALWAYS AS (to_tsvector('hungarian', content)) STORED
);
CREATE INDEX ON chunks USING hnsw (embedding halfvec_cosine_ops);
CREATE INDEX ON chunks USING gin (tsv);
CREATE INDEX ON chunks USING gin (acl_groups);

SET hnsw.iterative_scan = relaxed_order; -- keep enough rows after ACL filtering

WITH dense AS (
SELECT id, row_number() OVER (ORDER BY d) AS r
FROM (
SELECT id, embedding <=> $1::halfvec AS d -- kNN ORDER BY + LIMIT keeps the HNSW index usable
FROM chunks WHERE acl_groups && $3::text[]
ORDER BY embedding <=> $1::halfvec LIMIT 50
) knn
), sparse AS (
SELECT id, row_number() OVER (ORDER BY ts_rank_cd(tsv, q) DESC) AS r
FROM chunks, websearch_to_tsquery('hungarian', $2) q
WHERE tsv @@ q AND acl_groups && $3::text[]
ORDER BY ts_rank_cd(tsv, q) DESC LIMIT 50
)
SELECT id, sum(1.0 / (60 + r)) AS rrf
FROM (SELECT * FROM dense UNION ALL SELECT * FROM sparse) u
GROUP BY id ORDER BY rrf DESC LIMIT 40;
retrieval/qdrant_hybrid.py
from qdrant_client import QdrantClient, models

client = QdrantClient(url="http://qdrant:6333")

def hybrid(dense_vec, sparse_vec, user_groups: list[str], limit: int = 40):
acl = models.Filter(must=[models.FieldCondition(key="acl_groups", match=models.MatchAny(any=user_groups))])
return client.query_points(
collection_name="chunks",
prefetch=[
models.Prefetch(query=sparse_vec, using="sparse", filter=acl, limit=50),
models.Prefetch(query=dense_vec, using="dense", filter=acl, limit=50),
],
query=models.RrfQuery(rrf=models.Rrf()),
limit=limit,
).points

RRF is robust and needs no score normalisation; score-based fusion can edge it out when tuned on judged data (OpenSearch measured RRF ~3.9 % lower NDCG@10 on average across six BEIR datasets than their tuned score-based hybrid). Pick with your eval set, not by default.

6. Reranking​

Retrieve wide (top 40–150), rerank with a cross-encoder, pass the top 5–20 to the LLM. Anthropic's contextual-retrieval experiments retrieved 150 and kept 20 after reranking.

Reranker (as of 2026-10)LicenceNotes
bge-reranker-v2-m3Apache-2.0Multilingual, XLM-RoBERTa base; runs on TEI; default
Qwen3-Reranker 0.6B / 4B / 8BApache-2.0LLM-based, stronger, more expensive; serve with vLLM
mxbai-rerank-large-v2Apache-2.0Good English/multilingual trade-off
jina-reranker-v3CC-BY-NC-4.0Non-commercial weights
curl -s http://reranker:8081/rerank -H 'Content-Type: application/json' \
-d '{"query": "Mi a felmondási idő?", "texts": ["...chunk 1...", "...chunk 2..."]}'

Budget the latency: a cross-encoder over 50 chunks of 512 tokens is typically tens to a few hundred milliseconds on a GPU (rule of thumb; measure it in your load test).

7. Query understanding​

  • Rewrite conversational follow-ups into standalone queries using the chat history ("And for contractors?" → "What is the notice period for contractors?").
  • Multi-query / decomposition for compound questions: generate 2–4 sub-queries, retrieve for each, fuse with RRF.
  • HyDE (embed a hypothetical answer) only if it measurably improves recall on your set; it adds an LLM call per query.
  • Self-querying filters: extract structured filters (date range, document type, department) with a JSON-schema constrained LLM call and apply them as metadata filters.
  • Language handling: detect query language; for mixed corpora, either rely on a cross-lingual embedder or translate the query and search both.

8. Permission-aware retrieval​

The LLM must never see a chunk the user is not allowed to read. Enforce it in retrieval, not in the prompt.

  • Sync ACLs from the source system (SharePoint, Confluence, file-share NTFS ACLs) into acl_groups at ingestion and on permission-change events; re-sync on a schedule as a safety net.
  • Apply ACLs as a pre-filter inside the search (Qdrant filter, SQL WHERE, Postgres RLS, OpenSearch DLS). Post-filtering after top-k leaks through ranking and returns empty results.
  • Resolve groups server-side from the validated token; never accept groups from the client.
  • Use separate indexes or tenants for hard isolation between customers or legal entities.
  • Test it: the eval suite includes "user A asks about document only B can read" cases that must return no such content.

9. Generation, grounding, and citations​

prompts/answer.txt
You answer questions using only the numbered sources below.
- Cite every factual sentence with its source number, e.g. [2].
- If the sources do not contain the answer, say that the documents do not cover it and suggest who to ask.
- Answer in the language of the question.

Sources:
{% for s in sources %}[{{ loop.index }}] {{ s.title }} ({{ s.source_uri }}, p.{{ s.page }})
{{ s.text }}
{% endfor %}
  • Return citations as structured data (source_id, uri, page, quoted span) alongside the answer so the UI can render links and highlights.
  • Validate citations after generation: every cited ID must exist in the provided sources; optionally check that the quoted span appears in the chunk.
  • Order sources by rerank score and keep total context well within the model's effective window; more context is not automatically better.

10. Evaluation​

Evaluate retrieval and generation separately; most quality problems are retrieval problems.

LayerMetricTooling
Retrievalrecall@k (did a relevant chunk make the top k?), MRR, nDCG@kOwn script over the golden set (below)
RetrievalContext precision / context recallRAGAS, promptfoo context-recall
GenerationFaithfulness (claims supported by context), response relevancy, noise sensitivityRAGAS, DeepEval, promptfoo context-faithfulness
End to endAnswer correctness vs. reference, citation validity, correct refusalLLM judge with rubric + code checks (see GenAI PoC)
SecurityACL leakage tests, prompt injection in documentsDedicated test cases
evals/retrieval_recall.py
import json
from statistics import mean

def recall_at_k(golden_path: str, search, k: int = 10) -> dict:
rows = [json.loads(l) for l in open(golden_path, encoding="utf-8")]
hits, rr = [], []
for row in rows:
ids = [h.doc_id for h in search(row["query"], groups=row["groups"])[:k]]
relevant = set(row["relevant_doc_ids"])
hits.append(any(i in relevant for i in ids))
rr.append(next((1 / (i + 1) for i, d in enumerate(ids) if d in relevant), 0.0))
return {f"recall@{k}": mean(hits), "mrr": mean(rr), "n": len(rows)}

Gate CI on recall@k and faithfulness thresholds; break results down by tag (language, document type, difficulty).

11. Freshness and incremental indexing​

  • Prefer change feeds (Microsoft Graph delta queries, Confluence webhooks, database CDC) over full rescans; fall back to scheduled scans comparing content hashes.
  • Make ingestion idempotent: key chunks by (doc_id, version, chunk_no); on update, upsert new chunks then delete the old version's chunks in one transaction or batch.
  • Propagate deletions and permission changes with priority; a deleted or restricted document must disappear from results within the agreed SLA.
  • Use blue/green indexes with aliases for re-embedding or chunking changes; switch the alias after the eval suite passes on the new index.
  • Store updated_at and expose it in citations; optionally boost recency for time-sensitive corpora.

12. GraphRAG and agentic RAG — when warranted​

ApproachUse whenCost
Classic hybrid RAG (above)Fact lookup, policy Q&A, most enterprise searchBaseline
GraphRAG (Microsoft GraphRAG v3.3, LightRAG v1.5)Global questions across the corpus ("main themes across all incident reports"), entity-relationship questionsLLM-heavy indexing (entity/relation extraction, community summaries); re-indexing cost on change (Edge et al., 2024)
Agentic RAGMulti-hop questions, combining retrieval with SQL/APIs, iterative searchMore LLM calls and latency; needs step limits and tracing — see Agentic AI Platforms

Start with hybrid RAG; add graph or agentic retrieval only for the question classes where the eval shows classic retrieval cannot reach the bar.

13. Operations​

  • Dashboards: query volume, p50/p95 latency per stage (rewrite, retrieve, rerank, generate), empty-result rate, "no answer" rate, thumbs-down rate, cost per query.
  • Index health: chunk count per source, ingestion lag, failed parses, embedding-model version distribution.
  • Feedback loop: sample thumbs-down and "no answer" traces weekly into the golden set.
  • Backups: the source of truth is the documents plus parsed JSON in object storage; the index must be rebuildable from them by a pipeline run.

Best practices​

  • Optimise retrieval first; measure recall@k before touching the prompt.
  • Hybrid + rerank is the default stack; contextual retrieval is the next lever for chunk-context problems (Anthropic).
  • Choose embedders by measured recall on the customer's language mix, not by leaderboard rank; check the weights licence.
  • Treat ACLs as a retrieval filter, tested in CI.
  • Version everything that changes vectors: parser, chunker config, embedding model, contextualisation prompt.
  • Treat retrieved documents as untrusted input: they can carry prompt injections (OWASP LLM08: Vector and Embedding Weaknesses, LLM01: Prompt Injection).

Pitfalls to design around​

  • Tables and scans → use a layout-aware parser with table structure and OCR, and sample-check parsed output for every document type.
  • Exact identifiers → keep BM25 in the stack and add identifier fields as keyword-indexed metadata.
  • Filtered HNSW returning too few rows → enable pgvector iterative scans or use filter-aware engines; verify with ACL-heavy test users.
  • Silent model drift → pin embedder revisions; re-embed via a new index, never in place.
  • Stale or deleted content → monitor ingestion lag and deletion propagation as SLOs.
  • Licence surprises → check weights licences (Marker, MinerU, Jina, Gemma) during design.

Definition of done​

  • All agreed source systems ingested with connectors and change detection
  • Parsed output sample-checked for every document type (tables, scans, multi-column)
  • Every chunk carries source URI, version, page/section, language, updated_at, and ACL groups
  • Embedding model chosen by measured recall on the customer's languages; model ID and revision stored
  • Hybrid search + reranker live; fusion and top-k tuned on the golden set
  • ACL pre-filtering enforced in the index and covered by leakage tests
  • Answers cite sources; citation validity checked automatically; refusal when unsupported
  • Eval suite (recall@k, faithfulness, correctness, ACL tests) runs in CI with thresholds
  • Incremental indexing with deletion propagation; index rebuildable from object storage
  • Dashboards and alerts for latency per stage, ingestion lag, and feedback
  • Everything deployed via IaC and Helm; runbook handed over

Sources​