GenAI Proof of Concept
Service: GenAI Proof of Concept (PoC) · Typical shape: 1–6 week time-box, 2–3 engineers plus a customer domain expert, fixed scope and a written definition of done
A PoC answers one question with evidence: can this use case reach a quality bar the business accepts, at a cost and latency it can live with, on data and infrastructure it actually has? We answer it with an evaluation suite, not with a demo. The demo is a by-product; the eval numbers, the trace data, and a production-shaped repository are the deliverables.
What we deliver
| Deliverable | Content |
|---|---|
| Success criteria | Signed-off metrics with thresholds (e.g. "≥ 85 % answer correctness on the golden set, p95 latency ≤ 4 s, ≤ €0.02 per request") |
| Golden dataset | 50–300 versioned examples with expected outputs or grading rubrics, owned jointly with the customer's domain expert |
| Baseline | The simplest credible approach measured against the golden set (a single prompt, or the customer's current process) |
| Iteration log | Every experiment with its eval score, cost, and latency, linked to traces |
| Working prototype | A UI the stakeholders can use, backed by an OpenAI-compatible model endpoint |
| Production-shaped repo | Template repository with IaC, CI running the eval suite, containerised services |
| Handover report | Results vs. criteria, architecture, cost model, risks, and a production plan |
Reference architecture
The PoC stack is deliberately the production stack in miniature. Everything talks to an OpenAI-compatible API through a gateway, so the same code runs against a hosted model, a sovereign-cloud endpoint (e.g. STACKIT AI Model Serving), or a self-hosted vLLM server — swapping is a config change, not a rewrite.
| Layer | Default choice | Alternatives | Why |
|---|---|---|---|
| Model access | OpenAI-compatible endpoint via LiteLLM proxy | Direct SDK with base_url | One API for hosted, sovereign, and local models; per-key budgets and usage logs |
| Local / private serving | vLLM OpenAI-compatible server | Ollama, llama.cpp server for laptops | Same API as hosted providers; see Inference Engineering |
| UI | Open WebUI for chat-shaped use cases | Gradio, Streamlit for form/workflow-shaped use cases | Zero frontend work for chat; Gradio/Streamlit when the use case is "upload → extract → review" |
| Tracing | Langfuse (self-hostable) | Arize Phoenix (OpenTelemetry/OpenInference native) | Traces, cost, latency, datasets, and scores in one place from day one |
| Evals | promptfoo for prompt/model matrices in CI | DeepEval (pytest-style), Inspect (agentic/multi-step), RAGAS (RAG metrics) | See the decision table below |
Current versions (as of 2026-10): LiteLLM 1.106, vLLM 0.32, Open WebUI 0.12.3, Gradio 6.30, Streamlit 1.66, Langfuse server 4.63 / Python SDK 5.0, Phoenix 20.23, promptfoo 0.124 (npm), DeepEval 4.3, Inspect AI 0.3.281. Pin them in the repo; check the release pages before you start.
Implementation steps
Week-by-week plan
Scale the plan to the time-box. The phases stay the same; only their length changes.
| Phase | 1-week PoC | 3-week PoC | 6-week PoC | Exit criterion |
|---|---|---|---|---|
| 0. Frame | Day 1 | Days 1–2 | Week 1 | Success metrics signed off; data access granted; model access decided (hosted / sovereign / local) |
| 1. Golden set + baseline | Day 2 | Days 3–5 | Week 2 | ≥ 50 labelled examples; baseline scored; tracing live |
| 2. Iterate | Days 3–4 | Week 2 – mid week 3 | Weeks 3–5 | Each experiment scored; best configuration identified |
| 3. Harden + measure | Day 4 | Mid week 3 | Week 5 | Cost and latency measured under realistic load; failure analysis done |
| 4. Handover | Day 5 | End of week 3 | Week 6 | Report delivered; demo to stakeholders; production plan agreed |
1. Frame the question and the metrics (day 1)
Write the success criteria into docs/success-criteria.md and have the customer sponsor approve them before any prompt is written. Use three metric families:
| Family | Example metrics | How measured |
|---|---|---|
| Quality | Correctness, faithfulness to sources, extraction F1, format validity, refusal correctness | Golden set + graders (code, LLM-judge, human) |
| Operational | p50/p95 latency, time-to-first-token, throughput, error rate | Traces + load test |
| Economic | Cost per request, cost per successful task, projected monthly cost at expected volume | Token counts × price, or GPU-hours for self-hosted |
Every metric gets a threshold and a direction. "Good answers" is not a metric; "≥ 85 % of golden-set answers graded correct by the rubric, with judge–human agreement ≥ 0.8 on a 30-item calibration sample" is.
2. Build the golden dataset (days 2–5)
- Source inputs from real traffic: historical tickets, real documents, actual emails. Synthetic examples fill gaps (edge cases, adversarial inputs), never the core.
- Cover the distribution: common cases, hard cases, out-of-scope inputs the system must refuse or route, and Hungarian-language inputs when users write in Hungarian.
- Store it versioned in the repo (
evals/golden.jsonl) and mirror it into Langfuse/Phoenix datasets for experiment comparison. - Have the domain expert write or approve every expected output. Record who labelled what.
{"id": "inv-001", "input": "Mennyi a 2026/114-es számla végösszege?", "target": "1 270 000 HUF", "tags": ["hu", "extraction", "easy"]}
{"id": "inv-002", "input": "Which invoices from Q3 are overdue?", "target": "2026/087, 2026/093", "tags": ["en", "aggregation", "hard"]}
{"id": "oos-001", "input": "Write me a poem about the CFO.", "target": "REFUSE_OUT_OF_SCOPE", "tags": ["out-of-scope"]}
3. Stand up the stack with tracing from day one
model_list:
- model_name: poc-default
litellm_params:
model: hosted_vllm/${MODEL_ID}
api_base: http://vllm:8000/v1
- model_name: poc-hosted
litellm_params:
model: openai/${HOSTED_MODEL_ID}
api_base: os.environ/HOSTED_API_BASE
api_key: os.environ/HOSTED_API_KEY
litellm_settings:
success_callback: ["langfuse"]
failure_callback: ["langfuse"]
services:
vllm:
image: vllm/vllm-openai:v0.32.0
command: ["--model", "${MODEL_ID}", "--max-model-len", "32768"]
deploy: { resources: { reservations: { devices: [{ capabilities: ["gpu"] }] } } }
litellm:
image: ghcr.io/berriai/litellm:1.106.0 # pinned release; main-stable is deprecated
command: ["--config", "/app/config.yaml"]
volumes: ["./litellm/config.yaml:/app/config.yaml:ro"]
env_file: .env
ports: ["4000:4000"]
open-webui:
image: ghcr.io/open-webui/open-webui:v0.12.3
environment:
OPENAI_API_BASE_URL: http://litellm:4000/v1
OPENAI_API_KEY: ${LITELLM_KEY}
ports: ["3000:8080"]
The application calls the gateway with the stock OpenAI SDK and traces every call:
import os
from langfuse import observe
from langfuse.openai import OpenAI # drop-in wrapper: traces tokens, cost, latency
client = OpenAI(base_url=os.environ["LLM_BASE_URL"], api_key=os.environ["LLM_API_KEY"])
@observe()
def answer(question: str, context: str) -> str:
resp = client.chat.completions.create(
model=os.getenv("LLM_MODEL", "poc-default"),
temperature=0,
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
],
)
return resp.choices[0].message.content
For Langfuse in a PoC, docker compose on one VM is fine; it needs PostgreSQL, ClickHouse, Redis/Valkey, and S3-compatible storage. Use the Helm chart when the PoC turns into production (self-hosting docs).
4. Measure the baseline
Run the simplest credible system against the golden set before any optimisation: one prompt, one model, no retrieval tricks. This number is the yardstick for every later claim of improvement.
description: invoice-assistant PoC
prompts:
- file://prompts/v1_baseline.txt
- file://prompts/v2_structured.txt
providers:
- id: openai:chat:poc-default
config: { apiBaseUrl: http://localhost:4000/v1, apiKeyEnvar: LITELLM_KEY, temperature: 0 }
- id: openai:chat:poc-hosted
config: { apiBaseUrl: http://localhost:4000/v1, apiKeyEnvar: LITELLM_KEY, temperature: 0 }
defaultTest:
options:
provider: # the judge: pin it, and keep it different from the model under test
id: openai:chat:judge
config: { apiBaseUrl: http://localhost:4000/v1, apiKeyEnvar: LITELLM_KEY, temperature: 0 }
assert:
- type: latency
threshold: 4000
- type: cost
threshold: 0.02
tests: file://golden_promptfoo.yaml
- vars: { question: "Mennyi a 2026/114-es számla végösszege?" }
assert:
- type: contains
value: "1 270 000"
- vars: { question: "Write me a poem about the CFO." }
assert:
- type: llm-rubric
value: Politely declines and explains the assistant only answers invoice questions.
npx promptfoo@0.124.1 eval -c evals/promptfooconfig.yaml --output results/$(git rev-parse --short HEAD).json
npx promptfoo@0.124.1 view
Note that model-graded assertions use the grader set in defaultTest.options.provider, not the providers under test (promptfoo OpenAI provider docs).
5. Iterate, one variable at a time
Order experiments by expected gain per hour of effort:
- Prompt and output format — explicit instructions, examples, structured output (JSON schema), reasoning before answer.
- Context — what data the model sees: retrieval quality, chunking, metadata (see RAG Systems).
- Model choice — compare 2–3 candidates across size classes; include at least one model that can run on the target infrastructure (Model Landscape).
- Decomposition — prompt chaining or routing when a single call cannot hit the bar (see Agentic AI Platforms).
- Fine-tuning — only after the above plateau and only if the eval shows a gap fine-tuning can close (Fine-Tuning).
Log every run in docs/iteration-log.md: hypothesis, change, commit hash, score per metric, cost, latency, decision.
6. Measure cost and latency properly
- Hosted APIs: cost = input tokens × input price + output tokens × output price (+ cached-token discounts). Take token counts from traces, not estimates.
- Self-hosted: cost per request = (GPU-hour price × GPUs) / sustained requests per hour at your latency SLO. Measure the denominator with a load test, not a single request.
- Measure TTFT, inter-token latency, and end-to-end latency separately; streaming UIs care about TTFT, batch jobs about throughput.
- Load-test at the expected concurrency with realistic prompt lengths. vLLM ships a benchmarking CLI (
vllm bench serve); for any endpoint, a small Locust or k6 script against the gateway is enough.
from statistics import quantiles
def cost_report(traces, eur_per_m_in: float, eur_per_m_out: float, monthly_requests: int):
cost = [(t.input_tokens * eur_per_m_in + t.output_tokens * eur_per_m_out) / 1e6 for t in traces]
lat = sorted(t.latency_s for t in traces)
p50, p95 = quantiles(lat, n=100)[49], quantiles(lat, n=100)[94]
per_req = sum(cost) / len(cost)
return {"eur_per_request": per_req, "eur_per_month": per_req * monthly_requests, "p50_s": p50, "p95_s": p95}
7. Hand over
Present results against the signed-off criteria, then hand over the repository and the report (contents below).
Eval tooling: which tool for which job
| Tool | Best for | Shape | Notes |
|---|---|---|---|
| promptfoo | Prompt × model matrices, CI gates, red-teaming | YAML config, Node CLI | Deterministic (contains, is-json, latency, cost) and model-graded (llm-rubric, factuality, context-faithfulness, g-eval) assertions |
| DeepEval | Python teams who want evals as pytest tests | deepeval test run | G-Eval, faithfulness, answer relevancy, agent/tool metrics |
| RAGAS | RAG-specific metrics | Python library | Context precision/recall, faithfulness, response relevancy, noise sensitivity |
| Inspect | Multi-step and agentic tasks, sandboxed tool use | Python tasks, inspect eval CLI | From the UK AI Security Institute; strong log viewer; OpenAI-compatible providers via openai-api/<name>/<model> |
| Langfuse / Phoenix experiments | Comparing runs on datasets with trace links | UI + SDK | Use alongside one of the above for CI |
Default: promptfoo in CI for the regression gate, plus DeepEval or Inspect when graders need real Python logic or multi-step environments.
import json, pytest
from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, SingleTurnParams
from app.llm import answer
correctness = GEval(
name="Correctness",
criteria="The actual output states the same facts as the expected output; extra unsupported facts are wrong.",
evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.EXPECTED_OUTPUT],
threshold=0.8,
)
golden = [json.loads(l) for l in open("evals/golden.jsonl", encoding="utf-8")]
@pytest.mark.parametrize("ex", golden, ids=[g["id"] for g in golden])
def test_golden(ex):
assert_test(LLMTestCase(input=ex["input"], actual_output=answer(ex["input"], ""), expected_output=ex["target"]), [correctness])
from inspect_ai import Task, task
from inspect_ai.dataset import json_dataset
from inspect_ai.scorer import model_graded_fact
from inspect_ai.solver import generate, system_message
@task
def invoice_qa():
return Task(
dataset=json_dataset("evals/golden.jsonl"), # uses the input/target fields
solver=[system_message(open("app/prompts/system.txt").read()), generate()],
scorer=model_graded_fact(),
)
export LIEBER_BASE_URL=http://localhost:4000/v1 LIEBER_API_KEY=$LITELLM_KEY
inspect eval evals/inspect_task.py --model openai-api/lieber/poc-default
LLM-as-judge: use it, calibrate it
LLM judges agree with human preferences at roughly human–human levels on open-ended chat tasks, but they show position bias, verbosity bias, and self-enhancement bias (Zheng et al., 2023). Design for it:
- Prefer code graders wherever the answer is checkable (exact match, regex, JSON schema, numeric tolerance, SQL result equality).
- Give the judge a reference answer and a rubric, and ask for a short rationale before a categorical verdict (pass/fail or a 1–4 scale), not a free 1–10 score.
- In pairwise comparisons, swap positions and count only consistent verdicts.
- Use a judge from a different model family than the model under test; pin the judge model and version.
- Calibrate: have the domain expert grade 30–50 items; report judge–human agreement (accuracy or Cohen's κ) in the handover report. Re-calibrate when you change the judge or the rubric.
- Run each item multiple times when outputs are non-deterministic and report variance, not a single number (Anthropic: Demystifying evals for AI agents).
Production-ready shortcuts
The PoC repository is generated from our template so the production build starts from it instead of rewriting it (How We Build).
.
├── app/ # FastAPI service, prompts as files, no notebooks in the runtime path
├── evals/ # golden.jsonl, promptfoo config, DeepEval/Inspect tests, cost model
├── infra/ # OpenTofu: GPU node pool or model endpoint, object storage, Langfuse
├── deploy/ # Helm chart / Kustomize overlays
├── docs/ # success-criteria.md, iteration-log.md, handover-report.md
├── compose.yaml # local stack: vllm, litellm, open-webui, langfuse
└── .gitlab-ci.yml # lint, unit tests, eval gate, image build
eval:
stage: test
image: node:22
script:
- npx promptfoo@0.124.1 eval -c evals/promptfooconfig.yaml --output eval.json
artifacts: { paths: [eval.json], when: always }
rules:
- changes: [app/**/*, evals/**/*, prompts/**/*]
- Prompts are code: files in the repo, reviewed in merge requests, eval-gated in CI.
- Config over constants: model name, base URL, temperature, and top-k come from environment variables, so production swaps them without code changes.
- IaC from day one: even a single GPU VM or a managed model endpoint is created with OpenTofu/Terraform, so the production environment is a new workspace, not a ticket.
- Secrets never in the repo; use the platform secret store (Vault, cloud secret manager, sealed secrets).
- Data handling documented: where customer data lives during the PoC, who can access it, and when it is deleted.
Handover report contents
- Executive summary — the question, the answer, and the recommendation (go / go with conditions / reframe), in half a page.
- Success criteria vs. results — table of every metric, threshold, and measured value with confidence notes.
- Approach — final architecture diagram, model(s), prompts, retrieval setup, decisions and why.
- Iteration history — the experiments that moved the numbers, from the iteration log.
- Failure analysis — categorised remaining errors with examples and the proposed remedy for each.
- Cost and latency model — measured unit costs, projected monthly cost at expected volume, hosted vs. self-hosted comparison.
- Data findings — quality, coverage, and access issues found; what must be fixed before production.
- Security and compliance notes — data flows, residency, model licence, prompt-injection exposure.
- Production plan — scope, architecture deltas, effort estimate, and next step (typically AI Productification).
- Appendix — how to run the repo, the eval suite, and the dashboards.
Best practices
- Evals before prompts. The golden set and metrics exist before the first serious prompt; otherwise every change is judged by vibes (Anthropic: Demystifying evals).
- Start small and real. 20–50 tasks drawn from real cases is a good start; grow the set with every failure you find.
- Separate capability and regression evals. Capability evals track what you are trying to improve; regression evals (near 100 % pass) guard what already works.
- Trace everything from day one. Debugging, cost numbers, and dataset curation all come from traces.
- Keep the model swappable. The OpenAI-compatible boundary keeps hosted-vs-sovereign-vs-local a business decision, not a re-architecture.
- Time-box hard. If the bar is not reached at the end of phase 2, the report says what would be needed to reach it; the deadline holds.
Pitfalls to design around
- Demo-driven scope creep → keep a "parking lot" list in the repo and review it only at handover.
- A golden set written by engineers → have the domain expert own labels; engineers own the harness.
- Judge drift → pin the judge model and rubric version; recalibrate after any change.
- Single-run numbers → report means over multiple runs and per-tag breakdowns (language, difficulty).
- Latency measured on an idle server → load-test at target concurrency before quoting p95.
- Notebook-only code → notebooks are for exploration; the runtime path lives in
app/from week one.
Definition of done
- Success criteria with thresholds signed off by the customer sponsor
- Golden dataset (≥ 50 items) versioned in the repo, labels approved by the domain expert
- Baseline scored and recorded
- Tracing live for every LLM call (tokens, cost, latency)
- Iteration log with commit hashes and scores for every experiment
- LLM judge calibrated against human grades; agreement reported
- Cost per request and p50/p95 latency measured under realistic load
- Eval suite runs in CI and gates merges
- Infrastructure defined as code; stack starts with one command
- Handover report delivered and walked through with stakeholders
- Customer data handled and deleted according to the agreed data plan
Sources
- Anthropic — Demystifying evals for AI agents (tasks, graders, pass@k vs. pass^k, 20–50 starting tasks)
- Anthropic — Building effective agents
- Zheng et al. — Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- promptfoo — Configuration guide, Assertions and metrics, OpenAI provider / custom base URL, npm package
- DeepEval — Getting started, PyPI
- Inspect — Documentation, Model providers
- RAGAS — Documentation
- Langfuse — Self-hosting, Observability get started, Releases
- Arize Phoenix — Documentation, OpenInference
- LiteLLM — Proxy quick start, PyPI
- vLLM — OpenAI-compatible server, PyPI
- Open WebUI — Documentation, Releases
- Gradio — PyPI; Streamlit — PyPI