AI Productification — From PoC to Production
Service: AI Productification · Typical shape: 6–12 weeks after a successful GenAI PoC; 2–4 engineers (platform, backend, ML/evals), then a run phase with an agreed on-call model
What we deliver
A PoC proves that a use case works on a representative slice of data. Productification makes it operable: every request is routed, traced, evaluated, rate-limited, costed, and rolled back without a redeploy. The deliverables:
| Deliverable | Concrete artefact |
|---|---|
| LLM gateway | LiteLLM proxy or Agent Router (formerly Envoy AI Gateway) config in Git, with keys, budgets, fallbacks |
| Prompt and config registry | Versioned prompts in Git (or Langfuse prompt management) referenced by ID + version |
| Eval suite and CI gate | Golden dataset, scorers, thresholds; pipeline fails on regression |
| Progressive delivery | Feature flags (OpenFeature), shadow traffic, canary by percentage |
| Observability | OpenTelemetry traces with GenAI attributes, dashboards for TTFT, latency, error rate, tokens, cost |
| Guardrails | Input and output validation, structured outputs, safety classifier, PII redaction |
| Security review | OWASP Top 10 for LLM Applications 2025 mapping with tests |
| Data pipelines | Ingestion and refresh jobs (for RAG: see RAG Systems), feedback capture |
| Compliance pack | EU AI Act role and risk classification, Article 50 transparency, logging and oversight design |
| Runbooks | Incident playbooks for provider outage, quality regression, cost spike, prompt injection |
Everything assumes the shared standards in How We Build: IaC, GitOps, Kubernetes, and CI/CD from day one.
Reference architecture
Design rules behind the picture:
- The application never calls a model provider directly. All traffic goes through the gateway, which owns credentials, quotas, fallbacks, and a uniform OpenAI-compatible API. Swapping a managed API for a self-hosted model on private infrastructure becomes a config change.
- Prompts, model IDs, and sampling parameters are configuration, versioned and deployed like code, and evaluated before rollout.
- Telemetry is OpenTelemetry end to end. The LLM tracing tool is one OTLP backend among others, so it can be replaced.
Implementation steps
1. Harden the PoC codebase
- Move notebook logic into a service (FastAPI, Spring, .NET — match the client's stack) with a typed request/response contract.
- Externalise every prompt and model parameter into a registry entry (
prompt_id,version,model,temperature,max_tokens,response_schema). - Containerise, add health/readiness probes, deploy via Helm or Kustomize through GitOps (Argo CD or Flux).
- Add timeouts on every outbound call: connect timeout, first-token timeout, total timeout.
2. Put an LLM gateway in front of all models
Pick one gateway per platform:
| Option | Fits when | Configuration model |
|---|---|---|
| LiteLLM proxy | Many providers, per-team keys and budgets, quick start | config.yaml + admin UI |
| Agent Router (formerly Envoy AI Gateway; renamed Sept 2026, v1.2 as of 2026-10) | Kubernetes-native platform already on Envoy Gateway, Gateway API CRDs, MCP routing | AIGatewayRoute, AIServiceBackend CRDs (API group aigateway.envoyproxy.io unchanged) |
| Cloudflare AI Gateway, Bedrock, STACKIT model serving | Managed edge or cloud gateway is acceptable | Provider console / IaC; see Cloud AI Platforms |
LiteLLM proxy with fallbacks, retries, cooldowns, and per-deployment rate limits:
model_list:
- model_name: chat-default
litellm_params:
model: openai/qwen-chat # served by vLLM, OpenAI-compatible
api_base: http://vllm-chat.inference.svc:8000/v1
api_key: os.environ/VLLM_API_KEY
rpm: 600
- model_name: chat-fallback
litellm_params:
model: openai/fallback-model
api_base: https://managed-endpoint.example.eu/v1
api_key: os.environ/FALLBACK_API_KEY
rpm: 300
router_settings:
enable_pre_call_checks: true # needed for context-window fallbacks
litellm_settings:
num_retries: 2
request_timeout: 30
fallbacks: [{"chat-default": ["chat-fallback"]}]
context_window_fallbacks: [{"chat-default": ["chat-fallback"]}]
allowed_fails: 3
cooldown_time: 30
The gateway holds every provider credential, so treat it as a tier-0 supply-chain asset. LiteLLM versions 1.82.7 and 1.82.8 on PyPI were backdoored on 2026-03-24 (credential harvester and Kubernetes lateral-movement payload); 1.82.6 and earlier are clean, so pin a later verified release (1.106.0 as of 2026-10). Pin image digests or hashed requirements, mirror packages into an internal registry, and rotate secrets on any suspected exposure.
3. Version prompts and evaluate before every change
- Store prompts as files in the service repo (
prompts/answer/v7.yaml) or in Langfuse prompt management with labels (production,staging). The app resolvesprompt_id@labelat startup and caches it. - Build a golden dataset from the PoC: 100–300 real, anonymised inputs with expected properties (reference answer, required citations, forbidden content, expected JSON fields). Grow it from production traces flagged by users or reviewers.
- Score with a mix of deterministic checks (schema valid, citation present, regex, exact match) and LLM-as-judge rubrics calibrated against human labels.
- Gate merges in CI on the eval result. Example with promptfoo:
prompts:
- file://../prompts/answer/v7.txt
providers:
- id: openai:chat:chat-default
config:
apiBaseUrl: http://litellm.ai-platform.svc:4000/v1
tests: file://golden/answer_cases.yaml
defaultTest:
assert:
- type: is-json
- type: latency
threshold: 4000
- type: llm-rubric
value: "Answer is grounded in the provided context and cites at least one source."
eval-gate:
stage: test
image: node:22
script:
- npx promptfoo@0.124.1 eval -c evals/promptfooconfig.yaml -o eval-results.json
- python scripts/check_thresholds.py eval-results.json --min-pass-rate 0.92 --max-regression 0.02
artifacts:
paths: [eval-results.json]
rules:
- changes: [prompts/**, evals/**, src/**, litellm/**]
check_thresholds.py compares against the last main baseline and fails on an absolute floor and on a relative regression; both are needed because judge noise moves scores by a point or two.
4. Ship progressively: flags, shadow, canary
| Stage | Mechanism | Exit criterion |
|---|---|---|
| Offline | CI eval gate | Pass rate ≥ threshold, no safety failures |
| Shadow | Gateway or app duplicates a sample of live requests to the candidate; responses are logged, never returned | Online scores and latency within budget over 24–72 h |
| Canary | Feature flag routes 5% → 25% → 100% by user cohort | SLOs green, user feedback stable, cost per request within budget |
| Rollback | Flip the flag back; prompt and model version revert without deploy | Under 5 minutes |
Use OpenFeature with the client's flag backend (flagd, Unleash, LaunchDarkly) so the flag evaluation code is vendor-neutral:
from openfeature import api
from openfeature.evaluation_context import EvaluationContext
flags = api.get_client()
def select_variant(user_id: str) -> dict:
variant = flags.get_string_value("answer-pipeline", "v7", EvaluationContext(targeting_key=user_id))
return PIPELINES[variant] # {"prompt": "answer@v8", "model": "chat-default", ...}
5. Instrument with OpenTelemetry GenAI conventions
The GenAI semantic conventions are the common schema for LLM spans. As of 2026-10 they are still at Development stability and moved to the dedicated open-telemetry/semantic-conventions-genai repository in June 2026, so pin instrumentation versions and expect attribute renames.
Core attributes to emit on each model call span:
| Attribute | Example |
|---|---|
gen_ai.operation.name | chat, embeddings, execute_tool, invoke_agent |
gen_ai.provider.name | openai (replaces older gen_ai.system) |
gen_ai.request.model / gen_ai.response.model | qwen-chat |
gen_ai.usage.input_tokens / gen_ai.usage.output_tokens | 812 / 164 |
gen_ai.usage.cache_read.input_tokens | 640 |
gen_ai.response.finish_reasons | ["stop"] |
gen_ai.conversation.id | session ID |
Add your own app.prompt.id, app.prompt.version, app.flag.variant, and app.cost.eur attributes so every dashboard can slice by release.
Tooling choices (all accept OTLP):
| Tool | Role | Notes |
|---|---|---|
| Langfuse (self-hosted) | Traces, prompt management, datasets, online evals | OTLP endpoint at /api/public/otel; v4 is GA, self-hosted v3 gets security patches until 2027-01-31 (as of 2026-10) |
| Arize Phoenix | Traces and evals, OpenInference instrumentation | Self-hosted, open source |
| OpenLLMetry (Traceloop) | Auto-instrumentation for many providers and frameworks | Emits OTel spans to any backend |
| Prometheus + Grafana | SLO metrics and alerting | Derive from spans via the Collector spanmetrics connector |
OpenTelemetry Collector fan-out to Langfuse and Prometheus:
receivers:
otlp:
protocols: { grpc: {}, http: {} }
processors:
batch: {}
attributes/redact:
actions:
- key: gen_ai.input.messages
action: delete # keep content out of long-term metrics stores
connectors:
spanmetrics:
dimensions:
- name: gen_ai.request.model
- name: app.prompt.version
exporters:
otlphttp/langfuse:
endpoint: https://langfuse.internal/api/public/otel
headers:
Authorization: "Basic ${env:LANGFUSE_BASIC_AUTH}"
prometheus:
endpoint: 0.0.0.0:8889
service:
pipelines:
traces/llm:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp/langfuse, spanmetrics]
traces/metrics:
receivers: [otlp]
processors: [attributes/redact, batch]
exporters: [spanmetrics]
metrics:
receivers: [spanmetrics]
exporters: [prometheus]
Decide with the client's DPO which pipelines may contain prompt and completion content, for how long, and who can read it.
6. Guardrails: validate in, validate out
Layer cheap deterministic checks first, then classifiers, then LLM-based rails only where needed.
| Layer | Tooling (as of 2026-10) | Purpose |
|---|---|---|
| Input limits | App code, gateway | Max tokens, max attachments, allowed languages |
| PII | Microsoft Presidio | Detect and mask personal data before logging or sending to external models |
| Injection / safety | Llama Prompt Guard 2, Llama Guard 4 (12B, text + image) | Classify user input and retrieved content |
| Policy rails | NVIDIA NeMo Guardrails (0.24.x) | Topic rails, dialogue flows, calling classifiers as actions |
| Structured outputs | JSON Schema via vLLM guided decoding or provider structured outputs; Pydantic validation | Make downstream parsing safe |
| Output checks | Groundedness scorer, toxicity classifier, regex deny-lists | Block or route to fallback answer |
Structured output contract with validation and one repair attempt:
from pydantic import BaseModel, Field, ValidationError
from openai import OpenAI
class Answer(BaseModel):
answer: str = Field(max_length=4000)
citations: list[str] = Field(min_length=1)
confidence: float = Field(ge=0, le=1)
client = OpenAI(base_url="http://litellm.ai-platform.svc:4000/v1", api_key=GATEWAY_KEY)
def generate(messages: list[dict]) -> Answer:
for _ in range(2):
resp = client.chat.completions.create(
model="chat-default",
messages=messages,
response_format={"type": "json_schema",
"json_schema": {"name": "answer", "schema": Answer.model_json_schema()}},
timeout=30,
)
try:
return Answer.model_validate_json(resp.choices[0].message.content)
except ValidationError as e:
messages = messages + [{"role": "user", "content": f"Fix to match the schema: {e.errors()}"}]
raise AnswerUnavailable()
7. Define SLOs for the LLM path
| SLI | Definition | Typical target (interactive chat) |
|---|---|---|
| TTFT | Time from request accepted to first streamed token | p95 under 1.5 s |
| End-to-end latency | Request to last token, or to full JSON for non-streaming | p95 under 8 s |
| Inter-token latency | Mean gap between output tokens | p95 under 60 ms |
| Availability | Non-5xx, non-timeout responses / all requests | 99.5% monthly |
| Quality | Online eval pass rate on sampled traffic | ≥ offline baseline − 2 pts |
| Guardrail block rate | Blocked / all | Alert on 3× deviation from 7-day mean |
| Cost per request | (input × price_in + output × price_out + GPU-hour share) / requests | Budget agreed with product owner |
Prometheus burn-rate alert on availability (multi-window):
groups:
- name: llm-slo
rules:
- alert: LLMErrorBudgetBurnFast
expr: |
(sum(rate(llm_requests_total{outcome!="ok"}[5m])) / sum(rate(llm_requests_total[5m]))) > (14.4 * 0.005)
and
(sum(rate(llm_requests_total{outcome!="ok"}[1h])) / sum(rate(llm_requests_total[1h]))) > (14.4 * 0.005)
for: 2m
labels: { severity: page }
For self-hosted serving, also scrape vLLM's own metrics (queue time, KV-cache usage, preemptions); see Serving at Scale.
8. Caching
| Cache | How | Caveat |
|---|---|---|
| Prefix / prompt caching | Keep the static part of the prompt (system prompt, tool schemas, few-shot) byte-identical and first; vLLM automatic prefix caching or provider prompt caching reuses the KV cache | Any change early in the prompt (timestamps, user names) invalidates the rest; track gen_ai.usage.cache_read.input_tokens |
| Exact-match response cache | Hash of (prompt version, model, params, normalised input) in Redis with TTL | Only for deterministic, non-personalised requests |
| Semantic cache | Embedding similarity over past questions returns a stored answer | Near-duplicate questions can need different answers (dates, negations, user permissions); scope by tenant and ACL, use a high threshold, evaluate false-hit rate on the golden set, and never cache answers built from user-specific documents |
9. Rate limits, retries, fallbacks
- Enforce per-key and per-tenant RPM and TPM at the gateway, plus a monthly budget per virtual key (OWASP LLM10 Unbounded Consumption).
- Retry only idempotent failures (429, 503, connect errors) with exponential backoff and jitter; honour
Retry-After. Never retry after streaming has started to the client. - Fallback order: same model on another replica → smaller self-hosted model → managed model allowed by the data classification → static "service busy" answer. Fallback models must pass the same eval suite.
- Use a circuit breaker (gateway cooldowns, or Envoy outlier detection) so a degraded backend stops receiving traffic.
- Cap
max_tokens, tool-call depth, and agent loop iterations; see Agentic AI Platforms.
10. Security review against OWASP Top 10 for LLM Applications 2025
| ID | Risk | Control we implement | Test |
|---|---|---|---|
| LLM01 | Prompt Injection | Treat retrieved and tool content as untrusted data; classifier on inputs and retrieved chunks; least-privilege tools | promptfoo / garak red-team suite in CI |
| LLM02 | Sensitive Information Disclosure | PII redaction, per-user ACL filtering in retrieval, no secrets in prompts | Canary secrets in corpus must never appear in outputs |
| LLM03 | Supply Chain | Pinned images and hashes, SBOM, model provenance and licence check, internal mirrors | Dependency and image scanning gate |
| LLM04 | Data and Model Poisoning | Signed ingestion sources, review of fine-tuning data, anomaly checks on new documents | Ingest a poisoned doc in staging, verify detection |
| LLM05 | Improper Output Handling | Schema validation, output encoding, never eval or execute model output unsandboxed | XSS / SQL payload tests on outputs |
| LLM06 | Excessive Agency | Tool allow-lists, scoped credentials, human approval for writes | Attempt out-of-scope tool calls |
| LLM07 | System Prompt Leakage | No secrets or authorisation logic in the system prompt | Extraction prompts in red-team suite |
| LLM08 | Vector and Embedding Weaknesses | Tenant isolation in the vector DB, metadata ACL filters | Cross-tenant retrieval test |
| LLM09 | Misinformation | Grounding with citations, abstain when retrieval is empty, UI disclosure | Groundedness eval threshold |
| LLM10 | Unbounded Consumption | Rate limits, budgets, max_tokens, request size limits | Load test with abusive patterns |
For agent-based products, also map to the MITRE ATLAS agent techniques; the AI Assessment page covers the threat-modelling workshop.
11. Data pipelines and model monitoring
- Ingestion and refresh: scheduled jobs (Argo Workflows, Airflow, Dagster) with idempotent upserts, document-level lineage (
source_uri,hash,ingested_at), and a deletion path for GDPR erasure. - Feedback loop: thumbs up/down and free-text feedback attached to the trace ID; a weekly triage moves failures into the golden dataset.
- Drift signals to chart weekly:
- input drift: embedding centroid distance of user queries vs the golden set, language mix, topic clusters;
- retrieval drift: share of queries with empty or low-score retrieval;
- output drift: answer length, refusal rate, guardrail block rate, online eval pass rate;
- provider drift: response model string changes, latency and token-count shifts after provider updates.
- Re-baseline on every model or prompt change: run the full offline suite and store the result as the new baseline artefact.
12. EU AI Act obligations for deployers (as of 2026-10)
The Digital Omnibus on AI, Regulation (EU) 2026/1744, entered into force on 2026-07-27 and moved the high-risk dates. Current timeline:
| Date | What applies |
|---|---|
| 2025-02-02 | Prohibited practices (Art. 5) and AI literacy (Art. 4) |
| 2025-08-02 | General-purpose AI model obligations for providers |
| 2026-08-02 | Article 50 transparency obligations |
| 2026-12-02 | Art. 50(2) machine-readable marking for generative systems placed on the market before 2026-08-02; new Art. 5 prohibitions added by the Omnibus |
| 2027-12-02 | High-risk systems listed in Annex III (employment, education, credit scoring, and others) |
| 2028-08-02 | High-risk systems that are safety components of Annex I products |
What this means in a typical productification:
- Classify the role and risk in the compliance pack: the client is usually the deployer; if Lieber or the client substantially modifies a system or puts its name on it, provider duties can apply (Art. 25).
- Article 50, already applicable: users must know they are interacting with an AI system unless obvious (provider duty, build it into the UI); deployers must disclose deepfakes and AI-generated text published to inform the public on matters of public interest, unless it has human editorial review and responsibility.
- AI literacy (Art. 4): the Omnibus reframed this as a duty to take measures supporting staff AI literacy; deliver operator training and keep the record.
- If the use case is Annex III high-risk (from 2027-12-02), design for deployer duties under Art. 26 now: use per instructions, competent human oversight with authority to override, relevant and representative input data, monitoring with incident reporting, logs kept at least six months, informing workers and affected persons. Art. 27 FRIA can cross-reference an existing DPIA.
- Hungarian deployments: Act LXXV of 2025 and Government Decree 344/2025 designate the national authorities (the minister responsible for business development as market surveillance authority; the central bank for financial services).
13. Runbooks
Each runbook lives in the repo (runbooks/*.md), links to dashboards, and is exercised once before go-live.
| Trigger | First 15 minutes | Follow-up |
|---|---|---|
| Provider or GPU backend down | Confirm gateway fallback is serving; if not, flip flag to fallback pipeline | Post-incident review; check fallback eval scores |
| Quality regression (online eval drop) | Compare by app.prompt.version and gen_ai.response.model; roll back flag | Add failing traces to golden set |
| Cost spike | Identify top virtual keys / tenants; lower budgets or TPM | Review caching hit rate and prompt size |
| Prompt injection or data leak report | Disable affected tool or data source via flag; preserve traces | Security incident process, DPO notification assessment |
| Latency / TTFT breach | Check queue depth, KV-cache usage, preemptions; scale replicas | Capacity review, see Inference Engineering |
Best practices
- Keep one OpenAI-compatible interface at the gateway; application code stays provider-agnostic.
- Make every release reproducible:
(code SHA, prompt version, model ID + revision, gateway config SHA, eval baseline)recorded on the trace and the deployment. - Evaluate on production-shaped data; refresh the golden set monthly from real traffic.
- Put static prompt content first to maximise prefix-cache hits.
- Treat all model output as untrusted input to the rest of the system.
- Version and pin everything that moves fast: instrumentation packages, gateway images, guard models, model weights by revision hash.
Pitfalls to design around
- Judge drift: pin the judge model and its prompt; re-calibrate against human labels when either changes.
- Silent provider updates: alert on changes in
gen_ai.response.modeland in token-count distributions. - Telemetry as a data leak: apply the same data classification to traces as to the source data; redact before export to shared stores.
- Fallbacks that change data residency: encode allowed fallback targets per data class in gateway config, reviewed with the client.
- Semconv churn: the GenAI conventions are not stable; build dashboards on your own
app.*attributes plus the core token and model attributes.
Definition of done
- All model traffic goes through the gateway; no provider keys in application code or images
- Prompts and model parameters versioned; rollback via flag tested and timed under 5 minutes
- Golden dataset (≥ 100 cases) and CI eval gate with absolute and regression thresholds
- Shadow and canary phases completed with recorded results
- OTel traces with GenAI attributes plus
app.prompt.version; dashboards for TTFT, latency, errors, tokens, cost - SLOs agreed and burn-rate alerts routed to on-call
- Input and output guardrails in place; structured outputs validated
- OWASP LLM Top 10 2025 mapping with automated red-team tests in CI
- Rate limits, budgets, timeouts, retries, and fallbacks configured and load-tested
- Data pipelines idempotent with lineage and an erasure path
- Drift and online-eval dashboards with weekly review owner
- EU AI Act role/risk classification, Article 50 transparency implemented, log retention set
- Runbooks written and one game-day executed
- Supply chain: pinned digests, SBOM, model licence and provenance recorded
Sources
- OWASP Top 10 for LLM Applications 2025: https://genai.owasp.org/llm-top-10/
- OpenTelemetry GenAI semantic conventions (Development status; spec now in the semantic-conventions-genai repo): https://opentelemetry.io/docs/specs/semconv/gen-ai/ · https://github.com/open-telemetry/semantic-conventions-genai/tree/main/docs
- Langfuse OpenTelemetry endpoint: https://langfuse.com/integrations/native/opentelemetry · versions and v3 end of life: https://langfuse.com/docs/compatibility
- Arize Phoenix: https://github.com/arize-ai/phoenix · OpenInference: https://github.com/Arize-ai/openinference/blob/main/js/README.md · OpenLLMetry bridge: https://arize.com/docs/ax/integrations/opentelemetry/openllmetry-traceloop
- LiteLLM reliability (fallbacks, retries, cooldowns): https://docs.litellm.ai/docs/proxy/reliability
- LiteLLM March 2026 supply-chain incident: https://docs.litellm.ai/blog/security-update-march-2026 · https://securitylabs.datadoghq.com/articles/litellm-compromised-pypi-teampcp-supply-chain-campaign/
- Agent Router (formerly Envoy AI Gateway) release notes: https://theagentrouter.ai/release-notes/ · v1.0 GA: https://aigateway.envoyproxy.io/release-notes/v1.0/
- OpenFeature: https://openfeature.dev/
- NeMo Guardrails: https://github.com/NVIDIA-NeMo/Guardrails
- Llama Guard 4 model card: https://huggingface.co/meta-llama/Llama-Guard-4-12B · Llama Prompt Guard 2 model card: https://huggingface.co/meta-llama/Llama-Prompt-Guard-2-86M
- MITRE ATLAS: https://atlas.mitre.org/
- EU AI Act Digital Omnibus (Regulation (EU) 2026/1744, EUR-Lex): https://eur-lex.europa.eu/eli/reg/2026/1744/oj · https://artificialintelligenceact.eu/ai-act-explorer/digital-omnibus/ · https://www.orrick.com/en/Insights/2026/07/EU-AI-Act-Update-Digital-Omnibus-Finalizes-8-Compliance-Changes · https://www.joneswalker.com/en/insights/blogs/ai-law-blog/yes-august-2-still-matters-the-eu-approved-a-high-risk-ai-delay-but-most-trans.html
- EU AI Act Article 26 (deployer obligations): https://artificialintelligenceact.eu/article/26/ · Article 50 (transparency): https://artificialintelligenceact.eu/article/50/
- Hungarian implementation (Act LXXV of 2025, Decree 344/2025): https://digitalpolicyalert.org/event/34952-act-lxxv-of-2025-on-the-implementation-of-the-european-union-regulation-on-artificial-intelligence-in-hungary-enters-into-force-with-grace-period · https://www.regulatoryai.eu/ai-act-hungary/