Skip to main content

GenAI Proof of Concept

Basic Advanced

Service: GenAI Proof of Concept (PoC) · Typical shape: 1–6 week time-box, 2–3 engineers plus a customer domain expert, fixed scope and a written definition of done

A PoC answers one question with evidence: can this use case reach a quality bar the business accepts, at a cost and latency it can live with, on data and infrastructure it actually has? We answer it with an evaluation suite, not with a demo. The demo is a by-product; the eval numbers, the trace data, and a production-shaped repository are the deliverables.


What we deliver​

DeliverableContent
Success criteriaSigned-off metrics with thresholds (e.g. "≥ 85 % answer correctness on the golden set, p95 latency ≤ 4 s, ≤ €0.02 per request")
Golden dataset50–300 versioned examples with expected outputs or grading rubrics, owned jointly with the customer's domain expert
BaselineThe simplest credible approach measured against the golden set (a single prompt, or the customer's current process)
Iteration logEvery experiment with its eval score, cost, and latency, linked to traces
Working prototypeA UI the stakeholders can use, backed by an OpenAI-compatible model endpoint
Production-shaped repoTemplate repository with IaC, CI running the eval suite, containerised services
Handover reportResults vs. criteria, architecture, cost model, risks, and a production plan

Reference architecture​

The PoC stack is deliberately the production stack in miniature. Everything talks to an OpenAI-compatible API through a gateway, so the same code runs against a hosted model, a sovereign-cloud endpoint (e.g. STACKIT AI Model Serving), or a self-hosted vLLM server — swapping is a config change, not a rewrite.

LayerDefault choiceAlternativesWhy
Model accessOpenAI-compatible endpoint via LiteLLM proxyDirect SDK with base_urlOne API for hosted, sovereign, and local models; per-key budgets and usage logs
Local / private servingvLLM OpenAI-compatible serverOllama, llama.cpp server for laptopsSame API as hosted providers; see Inference Engineering
UIOpen WebUI for chat-shaped use casesGradio, Streamlit for form/workflow-shaped use casesZero frontend work for chat; Gradio/Streamlit when the use case is "upload → extract → review"
TracingLangfuse (self-hostable)Arize Phoenix (OpenTelemetry/OpenInference native)Traces, cost, latency, datasets, and scores in one place from day one
Evalspromptfoo for prompt/model matrices in CIDeepEval (pytest-style), Inspect (agentic/multi-step), RAGAS (RAG metrics)See the decision table below

Current versions (as of 2026-10): LiteLLM 1.106, vLLM 0.32, Open WebUI 0.12.3, Gradio 6.30, Streamlit 1.66, Langfuse server 4.63 / Python SDK 5.0, Phoenix 20.23, promptfoo 0.124 (npm), DeepEval 4.3, Inspect AI 0.3.281. Pin them in the repo; check the release pages before you start.


Implementation steps​

Week-by-week plan​

Scale the plan to the time-box. The phases stay the same; only their length changes.

Phase1-week PoC3-week PoC6-week PoCExit criterion
0. FrameDay 1Days 1–2Week 1Success metrics signed off; data access granted; model access decided (hosted / sovereign / local)
1. Golden set + baselineDay 2Days 3–5Week 2≥ 50 labelled examples; baseline scored; tracing live
2. IterateDays 3–4Week 2 – mid week 3Weeks 3–5Each experiment scored; best configuration identified
3. Harden + measureDay 4Mid week 3Week 5Cost and latency measured under realistic load; failure analysis done
4. HandoverDay 5End of week 3Week 6Report delivered; demo to stakeholders; production plan agreed

1. Frame the question and the metrics (day 1)​

Write the success criteria into docs/success-criteria.md and have the customer sponsor approve them before any prompt is written. Use three metric families:

FamilyExample metricsHow measured
QualityCorrectness, faithfulness to sources, extraction F1, format validity, refusal correctnessGolden set + graders (code, LLM-judge, human)
Operationalp50/p95 latency, time-to-first-token, throughput, error rateTraces + load test
EconomicCost per request, cost per successful task, projected monthly cost at expected volumeToken counts × price, or GPU-hours for self-hosted

Every metric gets a threshold and a direction. "Good answers" is not a metric; "≥ 85 % of golden-set answers graded correct by the rubric, with judge–human agreement ≥ 0.8 on a 30-item calibration sample" is.

2. Build the golden dataset (days 2–5)​

  • Source inputs from real traffic: historical tickets, real documents, actual emails. Synthetic examples fill gaps (edge cases, adversarial inputs), never the core.
  • Cover the distribution: common cases, hard cases, out-of-scope inputs the system must refuse or route, and Hungarian-language inputs when users write in Hungarian.
  • Store it versioned in the repo (evals/golden.jsonl) and mirror it into Langfuse/Phoenix datasets for experiment comparison.
  • Have the domain expert write or approve every expected output. Record who labelled what.
evals/golden.jsonl
{"id": "inv-001", "input": "Mennyi a 2026/114-es számla végösszege?", "target": "1 270 000 HUF", "tags": ["hu", "extraction", "easy"]}
{"id": "inv-002", "input": "Which invoices from Q3 are overdue?", "target": "2026/087, 2026/093", "tags": ["en", "aggregation", "hard"]}
{"id": "oos-001", "input": "Write me a poem about the CFO.", "target": "REFUSE_OUT_OF_SCOPE", "tags": ["out-of-scope"]}

3. Stand up the stack with tracing from day one​

litellm/config.yaml
model_list:
- model_name: poc-default
litellm_params:
model: hosted_vllm/${MODEL_ID}
api_base: http://vllm:8000/v1
- model_name: poc-hosted
litellm_params:
model: openai/${HOSTED_MODEL_ID}
api_base: os.environ/HOSTED_API_BASE
api_key: os.environ/HOSTED_API_KEY
litellm_settings:
success_callback: ["langfuse"]
failure_callback: ["langfuse"]
compose.yaml
services:
vllm:
image: vllm/vllm-openai:v0.32.0
command: ["--model", "${MODEL_ID}", "--max-model-len", "32768"]
deploy: { resources: { reservations: { devices: [{ capabilities: ["gpu"] }] } } }
litellm:
image: ghcr.io/berriai/litellm:1.106.0 # pinned release; main-stable is deprecated
command: ["--config", "/app/config.yaml"]
volumes: ["./litellm/config.yaml:/app/config.yaml:ro"]
env_file: .env
ports: ["4000:4000"]
open-webui:
image: ghcr.io/open-webui/open-webui:v0.12.3
environment:
OPENAI_API_BASE_URL: http://litellm:4000/v1
OPENAI_API_KEY: ${LITELLM_KEY}
ports: ["3000:8080"]

The application calls the gateway with the stock OpenAI SDK and traces every call:

app/llm.py
import os
from langfuse import observe
from langfuse.openai import OpenAI # drop-in wrapper: traces tokens, cost, latency

client = OpenAI(base_url=os.environ["LLM_BASE_URL"], api_key=os.environ["LLM_API_KEY"])

@observe()
def answer(question: str, context: str) -> str:
resp = client.chat.completions.create(
model=os.getenv("LLM_MODEL", "poc-default"),
temperature=0,
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
],
)
return resp.choices[0].message.content

For Langfuse in a PoC, docker compose on one VM is fine; it needs PostgreSQL, ClickHouse, Redis/Valkey, and S3-compatible storage. Use the Helm chart when the PoC turns into production (self-hosting docs).

4. Measure the baseline​

Run the simplest credible system against the golden set before any optimisation: one prompt, one model, no retrieval tricks. This number is the yardstick for every later claim of improvement.

evals/promptfooconfig.yaml
description: invoice-assistant PoC
prompts:
- file://prompts/v1_baseline.txt
- file://prompts/v2_structured.txt
providers:
- id: openai:chat:poc-default
config: { apiBaseUrl: http://localhost:4000/v1, apiKeyEnvar: LITELLM_KEY, temperature: 0 }
- id: openai:chat:poc-hosted
config: { apiBaseUrl: http://localhost:4000/v1, apiKeyEnvar: LITELLM_KEY, temperature: 0 }
defaultTest:
options:
provider: # the judge: pin it, and keep it different from the model under test
id: openai:chat:judge
config: { apiBaseUrl: http://localhost:4000/v1, apiKeyEnvar: LITELLM_KEY, temperature: 0 }
assert:
- type: latency
threshold: 4000
- type: cost
threshold: 0.02
tests: file://golden_promptfoo.yaml
evals/golden_promptfoo.yaml
- vars: { question: "Mennyi a 2026/114-es számla végösszege?" }
assert:
- type: contains
value: "1 270 000"
- vars: { question: "Write me a poem about the CFO." }
assert:
- type: llm-rubric
value: Politely declines and explains the assistant only answers invoice questions.
npx promptfoo@0.124.1 eval -c evals/promptfooconfig.yaml --output results/$(git rev-parse --short HEAD).json
npx promptfoo@0.124.1 view

Note that model-graded assertions use the grader set in defaultTest.options.provider, not the providers under test (promptfoo OpenAI provider docs).

5. Iterate, one variable at a time​

Order experiments by expected gain per hour of effort:

  1. Prompt and output format — explicit instructions, examples, structured output (JSON schema), reasoning before answer.
  2. Context — what data the model sees: retrieval quality, chunking, metadata (see RAG Systems).
  3. Model choice — compare 2–3 candidates across size classes; include at least one model that can run on the target infrastructure (Model Landscape).
  4. Decomposition — prompt chaining or routing when a single call cannot hit the bar (see Agentic AI Platforms).
  5. Fine-tuning — only after the above plateau and only if the eval shows a gap fine-tuning can close (Fine-Tuning).

Log every run in docs/iteration-log.md: hypothesis, change, commit hash, score per metric, cost, latency, decision.

6. Measure cost and latency properly​

  • Hosted APIs: cost = input tokens × input price + output tokens × output price (+ cached-token discounts). Take token counts from traces, not estimates.
  • Self-hosted: cost per request = (GPU-hour price × GPUs) / sustained requests per hour at your latency SLO. Measure the denominator with a load test, not a single request.
  • Measure TTFT, inter-token latency, and end-to-end latency separately; streaming UIs care about TTFT, batch jobs about throughput.
  • Load-test at the expected concurrency with realistic prompt lengths. vLLM ships a benchmarking CLI (vllm bench serve); for any endpoint, a small Locust or k6 script against the gateway is enough.
evals/cost_model.py
from statistics import quantiles

def cost_report(traces, eur_per_m_in: float, eur_per_m_out: float, monthly_requests: int):
cost = [(t.input_tokens * eur_per_m_in + t.output_tokens * eur_per_m_out) / 1e6 for t in traces]
lat = sorted(t.latency_s for t in traces)
p50, p95 = quantiles(lat, n=100)[49], quantiles(lat, n=100)[94]
per_req = sum(cost) / len(cost)
return {"eur_per_request": per_req, "eur_per_month": per_req * monthly_requests, "p50_s": p50, "p95_s": p95}

7. Hand over​

Present results against the signed-off criteria, then hand over the repository and the report (contents below).


Eval tooling: which tool for which job​

ToolBest forShapeNotes
promptfooPrompt × model matrices, CI gates, red-teamingYAML config, Node CLIDeterministic (contains, is-json, latency, cost) and model-graded (llm-rubric, factuality, context-faithfulness, g-eval) assertions
DeepEvalPython teams who want evals as pytest testsdeepeval test runG-Eval, faithfulness, answer relevancy, agent/tool metrics
RAGASRAG-specific metricsPython libraryContext precision/recall, faithfulness, response relevancy, noise sensitivity
InspectMulti-step and agentic tasks, sandboxed tool usePython tasks, inspect eval CLIFrom the UK AI Security Institute; strong log viewer; OpenAI-compatible providers via openai-api/<name>/<model>
Langfuse / Phoenix experimentsComparing runs on datasets with trace linksUI + SDKUse alongside one of the above for CI

Default: promptfoo in CI for the regression gate, plus DeepEval or Inspect when graders need real Python logic or multi-step environments.

evals/test_answers.py
import json, pytest
from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, SingleTurnParams
from app.llm import answer

correctness = GEval(
name="Correctness",
criteria="The actual output states the same facts as the expected output; extra unsupported facts are wrong.",
evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.EXPECTED_OUTPUT],
threshold=0.8,
)
golden = [json.loads(l) for l in open("evals/golden.jsonl", encoding="utf-8")]

@pytest.mark.parametrize("ex", golden, ids=[g["id"] for g in golden])
def test_golden(ex):
assert_test(LLMTestCase(input=ex["input"], actual_output=answer(ex["input"], ""), expected_output=ex["target"]), [correctness])
evals/inspect_task.py
from inspect_ai import Task, task
from inspect_ai.dataset import json_dataset
from inspect_ai.scorer import model_graded_fact
from inspect_ai.solver import generate, system_message

@task
def invoice_qa():
return Task(
dataset=json_dataset("evals/golden.jsonl"), # uses the input/target fields
solver=[system_message(open("app/prompts/system.txt").read()), generate()],
scorer=model_graded_fact(),
)
export LIEBER_BASE_URL=http://localhost:4000/v1 LIEBER_API_KEY=$LITELLM_KEY
inspect eval evals/inspect_task.py --model openai-api/lieber/poc-default

LLM-as-judge: use it, calibrate it​

LLM judges agree with human preferences at roughly human–human levels on open-ended chat tasks, but they show position bias, verbosity bias, and self-enhancement bias (Zheng et al., 2023). Design for it:

  • Prefer code graders wherever the answer is checkable (exact match, regex, JSON schema, numeric tolerance, SQL result equality).
  • Give the judge a reference answer and a rubric, and ask for a short rationale before a categorical verdict (pass/fail or a 1–4 scale), not a free 1–10 score.
  • In pairwise comparisons, swap positions and count only consistent verdicts.
  • Use a judge from a different model family than the model under test; pin the judge model and version.
  • Calibrate: have the domain expert grade 30–50 items; report judge–human agreement (accuracy or Cohen's κ) in the handover report. Re-calibrate when you change the judge or the rubric.
  • Run each item multiple times when outputs are non-deterministic and report variance, not a single number (Anthropic: Demystifying evals for AI agents).

Production-ready shortcuts​

The PoC repository is generated from our template so the production build starts from it instead of rewriting it (How We Build).

Repository layout
.
├── app/ # FastAPI service, prompts as files, no notebooks in the runtime path
├── evals/ # golden.jsonl, promptfoo config, DeepEval/Inspect tests, cost model
├── infra/ # OpenTofu: GPU node pool or model endpoint, object storage, Langfuse
├── deploy/ # Helm chart / Kustomize overlays
├── docs/ # success-criteria.md, iteration-log.md, handover-report.md
├── compose.yaml # local stack: vllm, litellm, open-webui, langfuse
└── .gitlab-ci.yml # lint, unit tests, eval gate, image build
.gitlab-ci.yml (eval gate)
eval:
stage: test
image: node:22
script:
- npx promptfoo@0.124.1 eval -c evals/promptfooconfig.yaml --output eval.json
artifacts: { paths: [eval.json], when: always }
rules:
- changes: [app/**/*, evals/**/*, prompts/**/*]
  • Prompts are code: files in the repo, reviewed in merge requests, eval-gated in CI.
  • Config over constants: model name, base URL, temperature, and top-k come from environment variables, so production swaps them without code changes.
  • IaC from day one: even a single GPU VM or a managed model endpoint is created with OpenTofu/Terraform, so the production environment is a new workspace, not a ticket.
  • Secrets never in the repo; use the platform secret store (Vault, cloud secret manager, sealed secrets).
  • Data handling documented: where customer data lives during the PoC, who can access it, and when it is deleted.

Handover report contents​

  1. Executive summary — the question, the answer, and the recommendation (go / go with conditions / reframe), in half a page.
  2. Success criteria vs. results — table of every metric, threshold, and measured value with confidence notes.
  3. Approach — final architecture diagram, model(s), prompts, retrieval setup, decisions and why.
  4. Iteration history — the experiments that moved the numbers, from the iteration log.
  5. Failure analysis — categorised remaining errors with examples and the proposed remedy for each.
  6. Cost and latency model — measured unit costs, projected monthly cost at expected volume, hosted vs. self-hosted comparison.
  7. Data findings — quality, coverage, and access issues found; what must be fixed before production.
  8. Security and compliance notes — data flows, residency, model licence, prompt-injection exposure.
  9. Production plan — scope, architecture deltas, effort estimate, and next step (typically AI Productification).
  10. Appendix — how to run the repo, the eval suite, and the dashboards.

Best practices​

  • Evals before prompts. The golden set and metrics exist before the first serious prompt; otherwise every change is judged by vibes (Anthropic: Demystifying evals).
  • Start small and real. 20–50 tasks drawn from real cases is a good start; grow the set with every failure you find.
  • Separate capability and regression evals. Capability evals track what you are trying to improve; regression evals (near 100 % pass) guard what already works.
  • Trace everything from day one. Debugging, cost numbers, and dataset curation all come from traces.
  • Keep the model swappable. The OpenAI-compatible boundary keeps hosted-vs-sovereign-vs-local a business decision, not a re-architecture.
  • Time-box hard. If the bar is not reached at the end of phase 2, the report says what would be needed to reach it; the deadline holds.

Pitfalls to design around​

  • Demo-driven scope creep → keep a "parking lot" list in the repo and review it only at handover.
  • A golden set written by engineers → have the domain expert own labels; engineers own the harness.
  • Judge drift → pin the judge model and rubric version; recalibrate after any change.
  • Single-run numbers → report means over multiple runs and per-tag breakdowns (language, difficulty).
  • Latency measured on an idle server → load-test at target concurrency before quoting p95.
  • Notebook-only code → notebooks are for exploration; the runtime path lives in app/ from week one.

Definition of done​

  • Success criteria with thresholds signed off by the customer sponsor
  • Golden dataset (≥ 50 items) versioned in the repo, labels approved by the domain expert
  • Baseline scored and recorded
  • Tracing live for every LLM call (tokens, cost, latency)
  • Iteration log with commit hashes and scores for every experiment
  • LLM judge calibrated against human grades; agreement reported
  • Cost per request and p50/p95 latency measured under realistic load
  • Eval suite runs in CI and gates merges
  • Infrastructure defined as code; stack starts with one command
  • Handover report delivered and walked through with stakeholders
  • Customer data handled and deleted according to the agreed data plan

Sources​