Ugrás a fő tartalomhoz

AI Productification — From PoC to Production

Basic Advanced

Service: AI Productification · Typical shape: 6–12 weeks after a successful GenAI PoC; 2–4 engineers (platform, backend, ML/evals), then a run phase with an agreed on-call model

What we deliver​

A PoC proves that a use case works on a representative slice of data. Productification makes it operable: every request is routed, traced, evaluated, rate-limited, costed, and rolled back without a redeploy. The deliverables:

DeliverableConcrete artefact
LLM gatewayLiteLLM proxy or Agent Router (formerly Envoy AI Gateway) config in Git, with keys, budgets, fallbacks
Prompt and config registryVersioned prompts in Git (or Langfuse prompt management) referenced by ID + version
Eval suite and CI gateGolden dataset, scorers, thresholds; pipeline fails on regression
Progressive deliveryFeature flags (OpenFeature), shadow traffic, canary by percentage
ObservabilityOpenTelemetry traces with GenAI attributes, dashboards for TTFT, latency, error rate, tokens, cost
GuardrailsInput and output validation, structured outputs, safety classifier, PII redaction
Security reviewOWASP Top 10 for LLM Applications 2025 mapping with tests
Data pipelinesIngestion and refresh jobs (for RAG: see RAG Systems), feedback capture
Compliance packEU AI Act role and risk classification, Article 50 transparency, logging and oversight design
RunbooksIncident playbooks for provider outage, quality regression, cost spike, prompt injection

Everything assumes the shared standards in How We Build: IaC, GitOps, Kubernetes, and CI/CD from day one.

Reference architecture​

Design rules behind the picture:

  • The application never calls a model provider directly. All traffic goes through the gateway, which owns credentials, quotas, fallbacks, and a uniform OpenAI-compatible API. Swapping a managed API for a self-hosted model on private infrastructure becomes a config change.
  • Prompts, model IDs, and sampling parameters are configuration, versioned and deployed like code, and evaluated before rollout.
  • Telemetry is OpenTelemetry end to end. The LLM tracing tool is one OTLP backend among others, so it can be replaced.

Implementation steps​

1. Harden the PoC codebase​

  1. Move notebook logic into a service (FastAPI, Spring, .NET — match the client's stack) with a typed request/response contract.
  2. Externalise every prompt and model parameter into a registry entry (prompt_id, version, model, temperature, max_tokens, response_schema).
  3. Containerise, add health/readiness probes, deploy via Helm or Kustomize through GitOps (Argo CD or Flux).
  4. Add timeouts on every outbound call: connect timeout, first-token timeout, total timeout.

2. Put an LLM gateway in front of all models​

Pick one gateway per platform:

OptionFits whenConfiguration model
LiteLLM proxyMany providers, per-team keys and budgets, quick startconfig.yaml + admin UI
Agent Router (formerly Envoy AI Gateway; renamed Sept 2026, v1.2 as of 2026-10)Kubernetes-native platform already on Envoy Gateway, Gateway API CRDs, MCP routingAIGatewayRoute, AIServiceBackend CRDs (API group aigateway.envoyproxy.io unchanged)
Cloudflare AI Gateway, Bedrock, STACKIT model servingManaged edge or cloud gateway is acceptableProvider console / IaC; see Cloud AI Platforms

LiteLLM proxy with fallbacks, retries, cooldowns, and per-deployment rate limits:

litellm/config.yaml
model_list:
- model_name: chat-default
litellm_params:
model: openai/qwen-chat # served by vLLM, OpenAI-compatible
api_base: http://vllm-chat.inference.svc:8000/v1
api_key: os.environ/VLLM_API_KEY
rpm: 600
- model_name: chat-fallback
litellm_params:
model: openai/fallback-model
api_base: https://managed-endpoint.example.eu/v1
api_key: os.environ/FALLBACK_API_KEY
rpm: 300

router_settings:
enable_pre_call_checks: true # needed for context-window fallbacks

litellm_settings:
num_retries: 2
request_timeout: 30
fallbacks: [{"chat-default": ["chat-fallback"]}]
context_window_fallbacks: [{"chat-default": ["chat-fallback"]}]
allowed_fails: 3
cooldown_time: 30
Pin and verify the gateway itself

The gateway holds every provider credential, so treat it as a tier-0 supply-chain asset. LiteLLM versions 1.82.7 and 1.82.8 on PyPI were backdoored on 2026-03-24 (credential harvester and Kubernetes lateral-movement payload); 1.82.6 and earlier are clean, so pin a later verified release (1.106.0 as of 2026-10). Pin image digests or hashed requirements, mirror packages into an internal registry, and rotate secrets on any suspected exposure.

3. Version prompts and evaluate before every change​

  1. Store prompts as files in the service repo (prompts/answer/v7.yaml) or in Langfuse prompt management with labels (production, staging). The app resolves prompt_id@label at startup and caches it.
  2. Build a golden dataset from the PoC: 100–300 real, anonymised inputs with expected properties (reference answer, required citations, forbidden content, expected JSON fields). Grow it from production traces flagged by users or reviewers.
  3. Score with a mix of deterministic checks (schema valid, citation present, regex, exact match) and LLM-as-judge rubrics calibrated against human labels.
  4. Gate merges in CI on the eval result. Example with promptfoo:
evals/promptfooconfig.yaml
prompts:
- file://../prompts/answer/v7.txt
providers:
- id: openai:chat:chat-default
config:
apiBaseUrl: http://litellm.ai-platform.svc:4000/v1
tests: file://golden/answer_cases.yaml
defaultTest:
assert:
- type: is-json
- type: latency
threshold: 4000
- type: llm-rubric
value: "Answer is grounded in the provided context and cites at least one source."
.gitlab-ci.yml (excerpt)
eval-gate:
stage: test
image: node:22
script:
- npx promptfoo@0.124.1 eval -c evals/promptfooconfig.yaml -o eval-results.json
- python scripts/check_thresholds.py eval-results.json --min-pass-rate 0.92 --max-regression 0.02
artifacts:
paths: [eval-results.json]
rules:
- changes: [prompts/**, evals/**, src/**, litellm/**]

check_thresholds.py compares against the last main baseline and fails on an absolute floor and on a relative regression; both are needed because judge noise moves scores by a point or two.

4. Ship progressively: flags, shadow, canary​

StageMechanismExit criterion
OfflineCI eval gatePass rate ≥ threshold, no safety failures
ShadowGateway or app duplicates a sample of live requests to the candidate; responses are logged, never returnedOnline scores and latency within budget over 24–72 h
CanaryFeature flag routes 5% → 25% → 100% by user cohortSLOs green, user feedback stable, cost per request within budget
RollbackFlip the flag back; prompt and model version revert without deployUnder 5 minutes

Use OpenFeature with the client's flag backend (flagd, Unleash, LaunchDarkly) so the flag evaluation code is vendor-neutral:

src/routing.py
from openfeature import api
from openfeature.evaluation_context import EvaluationContext

flags = api.get_client()

def select_variant(user_id: str) -> dict:
variant = flags.get_string_value("answer-pipeline", "v7", EvaluationContext(targeting_key=user_id))
return PIPELINES[variant] # {"prompt": "answer@v8", "model": "chat-default", ...}

5. Instrument with OpenTelemetry GenAI conventions​

The GenAI semantic conventions are the common schema for LLM spans. As of 2026-10 they are still at Development stability and moved to the dedicated open-telemetry/semantic-conventions-genai repository in June 2026, so pin instrumentation versions and expect attribute renames.

Core attributes to emit on each model call span:

AttributeExample
gen_ai.operation.namechat, embeddings, execute_tool, invoke_agent
gen_ai.provider.nameopenai (replaces older gen_ai.system)
gen_ai.request.model / gen_ai.response.modelqwen-chat
gen_ai.usage.input_tokens / gen_ai.usage.output_tokens812 / 164
gen_ai.usage.cache_read.input_tokens640
gen_ai.response.finish_reasons["stop"]
gen_ai.conversation.idsession ID

Add your own app.prompt.id, app.prompt.version, app.flag.variant, and app.cost.eur attributes so every dashboard can slice by release.

Tooling choices (all accept OTLP):

ToolRoleNotes
Langfuse (self-hosted)Traces, prompt management, datasets, online evalsOTLP endpoint at /api/public/otel; v4 is GA, self-hosted v3 gets security patches until 2027-01-31 (as of 2026-10)
Arize PhoenixTraces and evals, OpenInference instrumentationSelf-hosted, open source
OpenLLMetry (Traceloop)Auto-instrumentation for many providers and frameworksEmits OTel spans to any backend
Prometheus + GrafanaSLO metrics and alertingDerive from spans via the Collector spanmetrics connector

OpenTelemetry Collector fan-out to Langfuse and Prometheus:

otel-collector.yaml
receivers:
otlp:
protocols: { grpc: {}, http: {} }
processors:
batch: {}
attributes/redact:
actions:
- key: gen_ai.input.messages
action: delete # keep content out of long-term metrics stores
connectors:
spanmetrics:
dimensions:
- name: gen_ai.request.model
- name: app.prompt.version
exporters:
otlphttp/langfuse:
endpoint: https://langfuse.internal/api/public/otel
headers:
Authorization: "Basic ${env:LANGFUSE_BASIC_AUTH}"
prometheus:
endpoint: 0.0.0.0:8889
service:
pipelines:
traces/llm:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp/langfuse, spanmetrics]
traces/metrics:
receivers: [otlp]
processors: [attributes/redact, batch]
exporters: [spanmetrics]
metrics:
receivers: [spanmetrics]
exporters: [prometheus]

Decide with the client's DPO which pipelines may contain prompt and completion content, for how long, and who can read it.

6. Guardrails: validate in, validate out​

Layer cheap deterministic checks first, then classifiers, then LLM-based rails only where needed.

LayerTooling (as of 2026-10)Purpose
Input limitsApp code, gatewayMax tokens, max attachments, allowed languages
PIIMicrosoft PresidioDetect and mask personal data before logging or sending to external models
Injection / safetyLlama Prompt Guard 2, Llama Guard 4 (12B, text + image)Classify user input and retrieved content
Policy railsNVIDIA NeMo Guardrails (0.24.x)Topic rails, dialogue flows, calling classifiers as actions
Structured outputsJSON Schema via vLLM guided decoding or provider structured outputs; Pydantic validationMake downstream parsing safe
Output checksGroundedness scorer, toxicity classifier, regex deny-listsBlock or route to fallback answer

Structured output contract with validation and one repair attempt:

src/answer.py
from pydantic import BaseModel, Field, ValidationError
from openai import OpenAI

class Answer(BaseModel):
answer: str = Field(max_length=4000)
citations: list[str] = Field(min_length=1)
confidence: float = Field(ge=0, le=1)

client = OpenAI(base_url="http://litellm.ai-platform.svc:4000/v1", api_key=GATEWAY_KEY)

def generate(messages: list[dict]) -> Answer:
for _ in range(2):
resp = client.chat.completions.create(
model="chat-default",
messages=messages,
response_format={"type": "json_schema",
"json_schema": {"name": "answer", "schema": Answer.model_json_schema()}},
timeout=30,
)
try:
return Answer.model_validate_json(resp.choices[0].message.content)
except ValidationError as e:
messages = messages + [{"role": "user", "content": f"Fix to match the schema: {e.errors()}"}]
raise AnswerUnavailable()

7. Define SLOs for the LLM path​

SLIDefinitionTypical target (interactive chat)
TTFTTime from request accepted to first streamed tokenp95 under 1.5 s
End-to-end latencyRequest to last token, or to full JSON for non-streamingp95 under 8 s
Inter-token latencyMean gap between output tokensp95 under 60 ms
AvailabilityNon-5xx, non-timeout responses / all requests99.5% monthly
QualityOnline eval pass rate on sampled traffic≥ offline baseline − 2 pts
Guardrail block rateBlocked / allAlert on 3× deviation from 7-day mean
Cost per request(input × price_in + output × price_out + GPU-hour share) / requestsBudget agreed with product owner

Prometheus burn-rate alert on availability (multi-window):

alerts/llm-slo.yaml
groups:
- name: llm-slo
rules:
- alert: LLMErrorBudgetBurnFast
expr: |
(sum(rate(llm_requests_total{outcome!="ok"}[5m])) / sum(rate(llm_requests_total[5m]))) > (14.4 * 0.005)
and
(sum(rate(llm_requests_total{outcome!="ok"}[1h])) / sum(rate(llm_requests_total[1h]))) > (14.4 * 0.005)
for: 2m
labels: { severity: page }

For self-hosted serving, also scrape vLLM's own metrics (queue time, KV-cache usage, preemptions); see Serving at Scale.

8. Caching​

CacheHowCaveat
Prefix / prompt cachingKeep the static part of the prompt (system prompt, tool schemas, few-shot) byte-identical and first; vLLM automatic prefix caching or provider prompt caching reuses the KV cacheAny change early in the prompt (timestamps, user names) invalidates the rest; track gen_ai.usage.cache_read.input_tokens
Exact-match response cacheHash of (prompt version, model, params, normalised input) in Redis with TTLOnly for deterministic, non-personalised requests
Semantic cacheEmbedding similarity over past questions returns a stored answerNear-duplicate questions can need different answers (dates, negations, user permissions); scope by tenant and ACL, use a high threshold, evaluate false-hit rate on the golden set, and never cache answers built from user-specific documents

9. Rate limits, retries, fallbacks​

  • Enforce per-key and per-tenant RPM and TPM at the gateway, plus a monthly budget per virtual key (OWASP LLM10 Unbounded Consumption).
  • Retry only idempotent failures (429, 503, connect errors) with exponential backoff and jitter; honour Retry-After. Never retry after streaming has started to the client.
  • Fallback order: same model on another replica → smaller self-hosted model → managed model allowed by the data classification → static "service busy" answer. Fallback models must pass the same eval suite.
  • Use a circuit breaker (gateway cooldowns, or Envoy outlier detection) so a degraded backend stops receiving traffic.
  • Cap max_tokens, tool-call depth, and agent loop iterations; see Agentic AI Platforms.

10. Security review against OWASP Top 10 for LLM Applications 2025​

IDRiskControl we implementTest
LLM01Prompt InjectionTreat retrieved and tool content as untrusted data; classifier on inputs and retrieved chunks; least-privilege toolspromptfoo / garak red-team suite in CI
LLM02Sensitive Information DisclosurePII redaction, per-user ACL filtering in retrieval, no secrets in promptsCanary secrets in corpus must never appear in outputs
LLM03Supply ChainPinned images and hashes, SBOM, model provenance and licence check, internal mirrorsDependency and image scanning gate
LLM04Data and Model PoisoningSigned ingestion sources, review of fine-tuning data, anomaly checks on new documentsIngest a poisoned doc in staging, verify detection
LLM05Improper Output HandlingSchema validation, output encoding, never eval or execute model output unsandboxedXSS / SQL payload tests on outputs
LLM06Excessive AgencyTool allow-lists, scoped credentials, human approval for writesAttempt out-of-scope tool calls
LLM07System Prompt LeakageNo secrets or authorisation logic in the system promptExtraction prompts in red-team suite
LLM08Vector and Embedding WeaknessesTenant isolation in the vector DB, metadata ACL filtersCross-tenant retrieval test
LLM09MisinformationGrounding with citations, abstain when retrieval is empty, UI disclosureGroundedness eval threshold
LLM10Unbounded ConsumptionRate limits, budgets, max_tokens, request size limitsLoad test with abusive patterns

For agent-based products, also map to the MITRE ATLAS agent techniques; the AI Assessment page covers the threat-modelling workshop.

11. Data pipelines and model monitoring​

  • Ingestion and refresh: scheduled jobs (Argo Workflows, Airflow, Dagster) with idempotent upserts, document-level lineage (source_uri, hash, ingested_at), and a deletion path for GDPR erasure.
  • Feedback loop: thumbs up/down and free-text feedback attached to the trace ID; a weekly triage moves failures into the golden dataset.
  • Drift signals to chart weekly:
    • input drift: embedding centroid distance of user queries vs the golden set, language mix, topic clusters;
    • retrieval drift: share of queries with empty or low-score retrieval;
    • output drift: answer length, refusal rate, guardrail block rate, online eval pass rate;
    • provider drift: response model string changes, latency and token-count shifts after provider updates.
  • Re-baseline on every model or prompt change: run the full offline suite and store the result as the new baseline artefact.

12. EU AI Act obligations for deployers (as of 2026-10)​

The Digital Omnibus on AI, Regulation (EU) 2026/1744, entered into force on 2026-07-27 and moved the high-risk dates. Current timeline:

DateWhat applies
2025-02-02Prohibited practices (Art. 5) and AI literacy (Art. 4)
2025-08-02General-purpose AI model obligations for providers
2026-08-02Article 50 transparency obligations
2026-12-02Art. 50(2) machine-readable marking for generative systems placed on the market before 2026-08-02; new Art. 5 prohibitions added by the Omnibus
2027-12-02High-risk systems listed in Annex III (employment, education, credit scoring, and others)
2028-08-02High-risk systems that are safety components of Annex I products

What this means in a typical productification:

  1. Classify the role and risk in the compliance pack: the client is usually the deployer; if Lieber or the client substantially modifies a system or puts its name on it, provider duties can apply (Art. 25).
  2. Article 50, already applicable: users must know they are interacting with an AI system unless obvious (provider duty, build it into the UI); deployers must disclose deepfakes and AI-generated text published to inform the public on matters of public interest, unless it has human editorial review and responsibility.
  3. AI literacy (Art. 4): the Omnibus reframed this as a duty to take measures supporting staff AI literacy; deliver operator training and keep the record.
  4. If the use case is Annex III high-risk (from 2027-12-02), design for deployer duties under Art. 26 now: use per instructions, competent human oversight with authority to override, relevant and representative input data, monitoring with incident reporting, logs kept at least six months, informing workers and affected persons. Art. 27 FRIA can cross-reference an existing DPIA.
  5. Hungarian deployments: Act LXXV of 2025 and Government Decree 344/2025 designate the national authorities (the minister responsible for business development as market surveillance authority; the central bank for financial services).

13. Runbooks​

Each runbook lives in the repo (runbooks/*.md), links to dashboards, and is exercised once before go-live.

TriggerFirst 15 minutesFollow-up
Provider or GPU backend downConfirm gateway fallback is serving; if not, flip flag to fallback pipelinePost-incident review; check fallback eval scores
Quality regression (online eval drop)Compare by app.prompt.version and gen_ai.response.model; roll back flagAdd failing traces to golden set
Cost spikeIdentify top virtual keys / tenants; lower budgets or TPMReview caching hit rate and prompt size
Prompt injection or data leak reportDisable affected tool or data source via flag; preserve tracesSecurity incident process, DPO notification assessment
Latency / TTFT breachCheck queue depth, KV-cache usage, preemptions; scale replicasCapacity review, see Inference Engineering

Best practices​

  • Keep one OpenAI-compatible interface at the gateway; application code stays provider-agnostic.
  • Make every release reproducible: (code SHA, prompt version, model ID + revision, gateway config SHA, eval baseline) recorded on the trace and the deployment.
  • Evaluate on production-shaped data; refresh the golden set monthly from real traffic.
  • Put static prompt content first to maximise prefix-cache hits.
  • Treat all model output as untrusted input to the rest of the system.
  • Version and pin everything that moves fast: instrumentation packages, gateway images, guard models, model weights by revision hash.

Pitfalls to design around​

  • Judge drift: pin the judge model and its prompt; re-calibrate against human labels when either changes.
  • Silent provider updates: alert on changes in gen_ai.response.model and in token-count distributions.
  • Telemetry as a data leak: apply the same data classification to traces as to the source data; redact before export to shared stores.
  • Fallbacks that change data residency: encode allowed fallback targets per data class in gateway config, reviewed with the client.
  • Semconv churn: the GenAI conventions are not stable; build dashboards on your own app.* attributes plus the core token and model attributes.

Definition of done​

  • All model traffic goes through the gateway; no provider keys in application code or images
  • Prompts and model parameters versioned; rollback via flag tested and timed under 5 minutes
  • Golden dataset (≥ 100 cases) and CI eval gate with absolute and regression thresholds
  • Shadow and canary phases completed with recorded results
  • OTel traces with GenAI attributes plus app.prompt.version; dashboards for TTFT, latency, errors, tokens, cost
  • SLOs agreed and burn-rate alerts routed to on-call
  • Input and output guardrails in place; structured outputs validated
  • OWASP LLM Top 10 2025 mapping with automated red-team tests in CI
  • Rate limits, budgets, timeouts, retries, and fallbacks configured and load-tested
  • Data pipelines idempotent with lineage and an erasure path
  • Drift and online-eval dashboards with weekly review owner
  • EU AI Act role/risk classification, Article 50 transparency implemented, log retention set
  • Runbooks written and one game-day executed
  • Supply chain: pinned digests, SBOM, model licence and provenance recorded

Sources​