Skip to main content

Private and Sovereign AI Platforms

Advanced

Service: Sovereign & Private AI · On-Premise LLM Deployment · Sovereign AI on STACKIT · Typical shape: 2–4 week PoC on one GPU node, then 6–12 weeks to a production platform (IaC, CI/CD, SSO, observability) with a run/operate option

This page is the implementation playbook for all three private-AI offers. They share one reference architecture; what changes is where the GPUs live:

OfferGPUsKubernetesModel API
On-Premise LLM DeploymentCustomer data centre (bare metal)Upstream K8s, Canonical Kubernetes, RKE2 or OpenShiftSelf-hosted vLLM/SGLang
Sovereign AI on STACKITSTACKIT GPU VMs (EU01, Germany)SKE (STACKIT Kubernetes Engine)Self-hosted, plus STACKIT AI Model Serving for shared models
Sovereign & Private AIEither of the above, or hybridSameGateway in front of both

Read Serving at Scale for engine tuning and autoscaling, and Local AI Best Practices for model selection, quantization and evaluation habits. This page focuses on the platform around the engine.

What we deliver​

  • A private LLM platform exposing an OpenAI-compatible API behind an authenticated gateway, with per-team keys, quotas and usage reporting.
  • A chat front-end (Open WebUI or a custom app) wired to corporate SSO.
  • RAG building blocks: embedding model, reranker, vector database, ingestion pipeline (see RAG Systems).
  • Model supply chain: internal model registry, mirroring, checksum and signature verification, promotion from staging to production.
  • Infrastructure as code (Terraform/OpenTofu + Helm/Argo CD) for every component, plus runbooks for upgrades and model rollouts.
  • Compliance evidence pack: data-flow diagram, records of processing input, NIS2-relevant technical controls mapped to the implementation.
  • Cost model: GPU-hour vs per-token comparison for the customer's actual traffic.

Reference architecture​

LayerDefault choice (as of 2026-10)AlternativesWhy the default
GPU nodesNVIDIA data-centre GPUs, Ubuntu 24.04AMD MI300X/MI325X with ROCmWidest engine and quantization support
GPU enablementNVIDIA GPU Operator v26.7.xPre-baked driver imagesManages driver, container toolkit, device plugin, DCGM exporter, MIG
Inference enginevLLM ≥ 0.31.0SGLang 0.5.x for agentic/prefix-heavy loadsBroadest model coverage, OpenAI-compatible server
Multi-replica routingvLLM production-stack or llm-dPlain K8s ServiceKV-cache-aware routing, prefill/decode disaggregation
LLM gatewayLiteLLM proxy (pinned, see pitfalls)Envoy AI Gateway, Kong AI GatewayVirtual keys, budgets, routing to self-hosted and hosted backends
Front-endOpen WebUI 0.12.xLibreChat, custom React appSSO, RAG, tools, admin out of the box
Vector DBPostgreSQL + pgvectorQdrant, OpenSearchOne less stateful system to run; good to tens of millions of vectors
Observabilitykube-prometheus-stack, DCGM exporter, Loki, OpenTelemetrySTACKIT ObservabilityvLLM and DCGM export Prometheus metrics natively
IdentityCorporate IdP via OIDC; Keycloak when none fitsDexCentral joiner/mover/leaver control
Experiment trackingMLflow 3.x (self-hosted or STACKIT AI Model Experiments)LangfusePrompt registry, LLM tracing, eval runs

Hardware sizing​

GPU choice​

GPUMemoryBandwidthInterconnectSweet spot
L40S48 GB GDDR6864 GB/sPCIe7–32B models (FP8), embeddings, rerankers; cheap per node
A100 80GB80 GB HBM2e~2 TB/sNVLink (SXM) / PCIeExisting fleets; no FP8 tensor cores, prefer BF16/INT8/AWQ
H100 SXM80 GB HBM33.35 TB/sNVLinkgpt-oss-120b on one GPU; 70B FP8 on 2; TP across 8
H100 NVL94 GB HBM33.9 TB/sNVLink bridge pairsPCIe servers needing H100-class bandwidth
H200141 GB HBM3e4.8 TB/sNVLink70B FP8 on one GPU with large KV cache; long context
RTX PRO 6000 Blackwell Server96 GB GDDR71.6 TB/sPCIe only, MIG up to 4Standard air-cooled servers, FP4, many mid-size models per node
B200~180 GB HBM3e (1,440 GB per DGX B200)~8 TB/sNVLink 5Frontier-size MoE (DeepSeek/Kimi/Qwen 235B+ class), highest throughput
B300 (Blackwell Ultra)~262 GB (2.1 TB per DGX B300)HBM3eNVLink 5Largest MoE with long context; liquid-cooled facilities

Sources: NVIDIA product pages for L40S, H100, H200, RTX PRO 6000 Server, DGX B200, DGX B300. Per-GPU B200/B300 memory is derived from the 8-GPU system totals.

Rules of thumb for decode-bound LLM serving:

  • Memory first, then bandwidth. Decode throughput per replica scales roughly with memory bandwidth; capacity decides whether the model plus KV cache fits.
  • Weights: params × bytes_per_param → 70B at FP8 ≈ 70 GB; at FP4/MXFP4 ≈ 35–40 GB. gpt-oss-120b (117B total, 5.1B active, MXFP4 MoE weights) fits a single 80 GB GPU.
  • KV cache per token: 2 × layers × kv_heads × head_dim × bytes. For a Llama-3.3-70B-class model (80 layers, 8 KV heads, head_dim 128) at FP8 KV: 2 × 80 × 8 × 128 × 1 B ≈ 160 KB/token. 32 concurrent users × 8k context ≈ 256k tokens ≈ 41 GB of KV cache.
  • Headroom: budget 10–15 % of VRAM for activations, CUDA graphs and fragmentation (vLLM --gpu-memory-utilization 0.90).
  • Interconnect: tensor parallelism across PCIe-only cards (L40S, RTX PRO 6000) costs throughput; prefer models that fit in one card, or use pipeline/data parallelism instead.

Worked sizing example​

Customer: 400 knowledge workers, peak 40 concurrent streams, RAG with 8k-token prompts, 600-token answers, target p95 time-to-first-token under 2 s.

DecisionResult
Modelgpt-oss-120b (reasoning, tool calling, Apache 2.0) or Qwen3.x 27B-class dense for lower latency
Weights~65 GB on one H100 80GB → too little KV headroom for 40 × 8.6k tokens
Choice2 × H100 (two replicas, data parallel) or 1 × H200 per replica
Embeddings + reranker1 × L40S (bge-m3 / Qwen3 embedding + reranker)
ValidationRun vllm bench serve with the customer's prompt-length distribution before ordering hardware

Implementation steps​

1. Provision infrastructure as code​

STACKIT (provider stackitcloud/stackit, v0.120.0 released 2026-10-15). GPU node pools on SKE must use the Ubuntu image for the GPU Operator (STACKIT: Use NVIDIA GPUs).

infra/stackit/main.tf
terraform {
required_providers {
stackit = { source = "stackitcloud/stackit", version = "~> 0.120" }
time = { source = "hashicorp/time", version = "~> 0.12" }
}
}

provider "stackit" {
default_region = "eu01"
service_account_key_path = var.sa_key_path # service account key file
}

resource "stackit_ske_cluster" "ai" {
project_id = var.project_id
name = "ai-prod"
kubernetes_version_min = var.k8s_version

node_pools = [
{
name = "system"
machine_type = var.system_flavor # current general-purpose flavor
minimum = 3
maximum = 5
availability_zones = ["eu01-1", "eu01-2", "eu01-3"]
max_surge = 3
},
{
name = "gpu-h100"
machine_type = "n3.104d.g8" # 8x H100 HGX 80 GB
os_name = "ubuntu" # required for GPU Operator
minimum = 1
maximum = 2
availability_zones = ["eu01-1"]
max_surge = 1
volume_type = "storage_premium_perf6"
volume_size = 500
labels = { "workload" = "llm" }
taints = [{ key = "nvidia.com/gpu", value = "present", effect = "NoSchedule" }]
},
]

extensions = { acl = { enabled = true, allowed_cidrs = var.admin_cidrs } }

# STACKIT Network Area (SNA) control plane: availability depends on the account; check provider docs
# network = { control_plane = { access_scope = "SNA" } }

maintenance = {
enable_kubernetes_version_updates = true
enable_machine_image_version_updates = true
start = "01:00:00Z"
end = "03:00:00Z"
}
}

resource "time_rotating" "ms" { rotation_days = 80 }

resource "stackit_modelserving_token" "gateway" {
project_id = var.project_id
name = "litellm-gateway"
rotate_when_changed = { rotation = time_rotating.ms.id }
}

output "model_serving_token" {
value = stackit_modelserving_token.gateway.token
sensitive = true
}

STACKIT GPU flavors in EU01 (as of 2026-10, machine types):

FlavorGPUsvCPU / RAM
n1.14d.g1 / n1.28d.g2 / n1.56d.g41/2/4 × A100 80 GB14/28/56 vCPU, 108/216/460 GB
n2.14d.g1 / n2.28d.g2 / n2.56d.g41/2/4 × L40S 48 GB14/28/56 vCPU, 50/100/200 GB
n3.104d.g88 × H100 HGX 80 GB104 vCPU, 1,900 GB, 1,536 GB local disk

GPU VMs are not live-migrated (per STACKIT guidance; confirm in current docs); plan maintenance windows and run at least two replicas of anything user-facing. n1.56d.g4 is requested via Help Center ticket.

On-premise: same Terraform layout with the provider for the customer's virtualisation (vSphere, OpenStack, MAAS for bare metal) and Canonical Kubernetes, RKE2 or kubeadm. Keep GPU nodes in a dedicated node pool with the same taint so manifests are identical across sites.

2. Enable GPUs​

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update
helm install gpu-operator nvidia/gpu-operator \
-n gpu-operator --create-namespace --wait \
--version v26.7.2 \
--set dcgmExporter.serviceMonitor.enabled=true
kubectl get nodes -o json | jq '.items[].status.allocatable["nvidia.com/gpu"]'

Check the GPU Operator component matrix against the node OS, kernel and containerd version before bumping; STACKIT's guide uses v26.3.3 as its example, so confirm v26.7.x on SKE against the matrix before rollout. Use MIG profiles on H100/H200/RTX PRO 6000 for small embedding/reranker models.

3. Deploy the inference engine​

k8s/vllm-gpt-oss.yaml
apiVersion: apps/v1
kind: Deployment
metadata: { name: vllm-gpt-oss-120b, namespace: llm }
spec:
replicas: 2
selector: { matchLabels: { app: vllm-gpt-oss-120b } }
template:
metadata: { labels: { app: vllm-gpt-oss-120b } }
spec:
tolerations: [{ key: nvidia.com/gpu, operator: Exists, effect: NoSchedule }]
containers:
- name: vllm
image: registry.internal/mirror/vllm/vllm-openai:v0.31.0 # mirrored, digest-pinned in prod
args:
- --model=/models/openai/gpt-oss-120b/2026-09-01
- --served-model-name=gpt-oss-120b
- --gpu-memory-utilization=0.90
- --max-model-len=65536
- --enable-prefix-caching
- --tool-call-parser=openai
- --enable-auto-tool-choice
env:
- { name: HF_HUB_OFFLINE, value: "1" }
- { name: VLLM_NO_USAGE_STATS, value: "1" }
ports: [{ containerPort: 8000, name: http }]
readinessProbe: { httpGet: { path: /health, port: http }, periodSeconds: 10 }
resources: { limits: { nvidia.com/gpu: 1 } }
volumeMounts: [{ name: models, mountPath: /models, readOnly: true }]
volumes:
- name: models
persistentVolumeClaim: { claimName: model-store }
  • vLLM ships releases frequently (latest 0.32.0 on 2026-10-18); this page uses ≥ 0.31.0 as the floor. Check each release's notes for security fixes before pinning (PyPI, releases).
  • HF_HUB_OFFLINE=1 and a local path guarantee the pod never reaches Hugging Face.
  • For more than two replicas, put the vLLM production-stack router or llm-d in front for prefix-aware routing; details in Serving at Scale.

4. Put the gateway in front​

litellm/config.yaml
model_list:
- model_name: gpt-oss-120b
litellm_params:
model: hosted_vllm/gpt-oss-120b
api_base: http://vllm-gpt-oss-120b.llm.svc:8000/v1
- model_name: embed
litellm_params:
model: hosted_vllm/Qwen/Qwen3-Embedding-8B
api_base: http://vllm-embed.llm.svc:8000/v1
# Overflow / extra models on STACKIT AI Model Serving (EU01)
- model_name: qwen3-vl-235b
litellm_params:
model: openai/Qwen/Qwen3-VL-235B-A22B-Instruct-FP8
api_base: https://api.openai-compat.model-serving.eu01.onstackit.cloud/v1
api_key: os.environ/STACKIT_MODEL_SERVING_TOKEN

router_settings:
routing_strategy: least-busy
num_retries: 2

general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
database_url: os.environ/DATABASE_URL # Postgres for keys, budgets, spend logs

litellm_settings:
turn_off_message_logging: true # do not persist prompt/response bodies

Issue one virtual key per application/team with max_budget, tpm_limit, rpm_limit, and an allowed model list. Users of the chat UI authenticate through SSO; machine clients use virtual keys stored in the secrets manager.

5. Front-end with SSO​

open-webui/values.yaml (excerpt)
extraEnvVars:
- { name: ENABLE_OAUTH_SIGNUP, value: "true" }
- { name: OAUTH_PROVIDER_NAME, value: "Corporate SSO" }
- { name: OPENID_PROVIDER_URL, value: "https://login.example.com/.well-known/openid-configuration" }
- { name: OAUTH_CLIENT_ID, valueFrom: { secretKeyRef: { name: owui-oidc, key: client_id } } }
- { name: OAUTH_CLIENT_SECRET, valueFrom: { secretKeyRef: { name: owui-oidc, key: client_secret } } }
- { name: ENABLE_OAUTH_ROLE_MANAGEMENT, value: "true" }
- { name: OPENAI_API_BASE_URL, value: "http://litellm.llm.svc:4000/v1" }
- { name: ENABLE_LOGIN_FORM, value: "false" }
Open WebUI licence

Open WebUI ships under its own BSD-3-based licence with a branding clause: deployments above 50 end users in a rolling 30-day window must keep the Open WebUI branding unless an enterprise licence is obtained. Settle this with the customer before white-labelling; use LibreChat or a custom front-end if rebranding is required.

6. Air-gapped model distribution​

  1. Pin by revision. Never download main; pin the commit hash.

    hf download openai/gpt-oss-120b --revision <commit-sha> \
    --local-dir /stage/openai/gpt-oss-120b/2026-09-01
  2. Accept only safetensors (no pickle .bin/.pt), and keep trust_remote_code off unless the code has been reviewed and vendored.

  3. Checksum and manifest. Produce SHA256SUMS over every file, plus a manifest with source repo, revision, licence, and intended served name.

    cd /stage/openai/gpt-oss-120b/2026-09-01
    find . -type f ! -name SHA256SUMS -print0 | sort -z | xargs -0 sha256sum > SHA256SUMS
    cosign sign-blob --key kms://... --output-signature SHA256SUMS.sig SHA256SUMS
  4. Ship as an OCI artifact (oras push) or to an S3 bucket with object lock. Container images go through the same registry mirror (Harbor), digest-pinned.

  5. Verify at load time. An init container runs cosign verify-blob and sha256sum -c SHA256SUMS before vLLM starts; a mismatch fails the pod.

  6. Promote by tag after the eval suite passes in staging; production manifests reference the dated path, never latest.

7. Network isolation​

  • GPU and model namespaces get a default-deny NetworkPolicy; allow ingress only from the gateway namespace and egress only to the model store, DNS, and the OTel collector.
  • No internet egress from inference pods. On STACKIT, restrict cluster egress with security groups and check egress_address_ranges of the SKE cluster for allow-listing at downstream systems.
  • Restrict the Kubernetes API: on STACKIT enable the SKE ACL extension (extensions.acl.allowed_cidrs), or control plane access scope SNA (STACKIT Network Area, private preview) where enabled; on-premise expose the API server only to the admin network/VPN.
  • TLS everywhere: ingress terminates TLS with cert-manager; service mesh (Istio ambient or Linkerd) for mTLS between gateway, RAG and engines when the customer's threat model requires it.
  • If STACKIT AI Model Serving is used, it is a public HTTPS endpoint in EU01: route it via the gateway only, with the token in the secrets manager, so applications never hold the token.

8. Observability and evaluation​

  • Scrape vLLM /metrics (time-to-first-token, inter-token latency, queue length, KV-cache usage) and DCGM (DCGM_FI_DEV_GPU_UTIL, memory, temperature, XID errors).
  • Gateway spend logs → per-team usage dashboards and chargeback.
  • Alert on: queue time p95, KV-cache usage > 95 % sustained, XID errors, 5xx rate at the gateway.
  • Keep an offline eval suite (golden questions, RAG faithfulness, tool-call accuracy) and run it on every model or engine change; track runs in MLflow.

STACKIT specifics​

ProductWhat it is (as of 2026-10)How we use it
AI Model ServingShared open-weight models behind an OpenAI-compatible API in EU01, pay per token; STACKIT states prompts are not stored or used for training (confirm in the STACKIT FAQ before quoting)Fast start, overflow capacity, large models not worth self-hosting
Compute Engine GPUA100, L40S, H100 flavors (table above)Dedicated inference for fixed models, fine-tuning
SKEManaged Kubernetes, Ubuntu node pools for GPUs (required by the GPU Operator)Platform runtime
AI Model ExperimentsManaged MLflow tracking server, artifacts in a STACKIT Object Storage bucket (release stage unconfirmed)Experiment tracking, prompt registry, LLM tracing
Object Storage, Secrets Manager, KMS, ObservabilityS3-compatible storage, secrets management, key management, observability serviceModel store, secrets, encryption keys, monitoring

Shared models on AI Model Serving (docs, as of 2026-10):

Model IDTierContextLimitsPrice € / 1M in · out (third-party, see note)
Qwen/Qwen3-VL-235B-A22B-Instruct-FP8Premium200K350k TPM, 30 RPM1.50 · 1.75
Qwen/Qwen3.8-27BPlus262K350k TPM, 45 RPMunconfirmed
openai/gpt-oss-120bPlus131K200k TPM, 30 RPM0.45 · 0.65
google/gemma-4-31B-itPlus256K200k TPM, 80 RPM0.45 · 0.65
cortecs/Llama-3.3-70B-Instruct-FP8-DynamicPlus128K200k TPM, 80 RPM0.45 · 0.65
openai/gpt-oss-20bStandard131K200k TPM, 80 RPM0.15 · 0.25
intfloat/e5-mistral-7b-instructEmbedding4k input600 RPM0.02
Qwen/Qwen3-VL-Embedding-8BEmbedding32k input600 RPM0.08

Prices come from the European Cloud price table (updated 2026-08-29). STACKIT's shared-models page lists limits but no prices, and the Qwen3.8 27B price was not found in any source; confirm all prices in the STACKIT portal before quoting.

Two details that matter for capacity planning: TPM counts output tokens with a weight of 5, and Qwen/Qwen3.6-27B is marked deprecated on the shared-models page — keep model names in gateway config, never in application code, so swaps are a one-line change.

stackit_client.py
import os
from openai import OpenAI

client = OpenAI(
base_url="https://api.openai-compat.model-serving.eu01.onstackit.cloud/v1",
api_key=os.environ["STACKIT_MODEL_SERVING_TOKEN"],
)
resp = client.chat.completions.create(
model="openai/gpt-oss-120b",
messages=[{"role": "user", "content": "Summarise NIS2 incident reporting deadlines."}],
max_completion_tokens=400,
)
print(resp.choices[0].message.content)

GDPR and NIS2 technical controls​

RequirementTechnical control in this platform
GDPR Art. 5(1)(c) data minimisationGateway does not persist prompt bodies (turn_off_message_logging); RAG indexes only approved sources; PII redaction (Presidio) in ingestion where needed
GDPR Art. 25 privacy by designPrivate endpoints, default-deny network, no third-party processors in the inference path
GDPR Art. 28 processorsSelf-hosted: no AI processor. STACKIT: DPA with STACKIT, EU01 region only
GDPR Art. 30 / 35Data-flow diagram and model inventory feed the RoPA and DPIA
GDPR Art. 32 securityTLS + encryption at rest (KMS-backed volumes and buckets), RBAC via IdP groups, audit logs
GDPR Art. 17 erasureChat history retention policy in the front-end; document deletion propagates to vector DB by source ID
NIS2 Art. 21(2)(d) supply chainSigned model manifests, mirrored and digest-pinned images, SBOMs for gateway and front-end
NIS2 Art. 21(2)(i) access controlSSO with MFA, per-team virtual keys, short-lived tokens (ttl_duration, time_rotating)
NIS2 Art. 21(2)(b) incident handling, Art. 23 reportingCentralised logs with retention, alerting, runbook mapping to 24 h early warning / 72 h notification
NIS2 Art. 21(2)(c) business continuityMulti-replica engines, IaC rebuild of the whole stack, model store backup
EU AI Act transparency (Art. 50)UI labels AI-generated content; audit trail of model versions per answer

Operations​

Upgrades. Pin engine, gateway, front-end and GPU Operator versions in Git. Upgrade in staging first, run vllm bench serve and the eval suite, compare against the stored baseline, then roll production with maxUnavailable: 0. GPU Operator and node OS upgrades go node pool by node pool inside the maintenance window.

Model rollouts.

  • Clients call stable aliases (chat-default, chat-reasoning, embed); the gateway maps aliases to concrete models.
  • Changing an embedding model requires a full re-index; run old and new indexes side by side and switch the alias when the new index is complete.
  • Keep the previous model deployed (scaled to zero or one) for fast rollback during the first week.

Capacity. Watch queue time and KV-cache usage, not GPU utilisation. Scale out replicas on queue depth (KEDA on vllm:num_requests_waiting).

Cost model: GPU-hour vs per-token​

Break-even monthly volume:

T_breakeven = (N_gpu × price_gpu_hour × 730 + C_ops) / price_per_token_blended
price_per_token_blended = (r_in × p_in + r_out × p_out) / (r_in + r_out)

Worked example (illustrative GPU price — replace with the current STACKIT or hardware quote):

InputValue
Modelgpt-oss-120b on 1 × H100 80GB
GPU cost (assumed placeholder, not a published STACKIT price)€4.00 / GPU-hour → €2,920 / month
Ops overhead attributed to this model€1,000 / month
Per-token alternativeSTACKIT AI Model Serving gpt-oss-120b: €0.45 in, €0.65 out per 1M
Traffic mix (RAG)4 input : 1 output → blended €0.49 per 1M tokens

T_breakeven = (2,920 + 1,000) / 0.49 ≈ 8,000 M tokens/month ≈ 8 B tokens/month, i.e. about 3,050 tokens/s sustained 24×7.

  • The 400-user example above produces roughly 400 users × 30 requests/day × 9k tokens × 22 days ≈ 2.4 B tokens/month → per-token is cheaper for that model alone.
  • Self-hosting wins when volume exceeds the break-even, when one GPU pool serves several workloads (batch summarisation at night), when the model or a fine-tune is not offered as shared model, or when the data may not leave the customer's premises at all.
  • Measure the actual sustainable throughput of the chosen GPU at the target latency SLO with vllm bench serve; a GPU that cannot reach the break-even token rate at the SLO can never pay off on cost alone.

Best practices​

  • Start the PoC on STACKIT AI Model Serving or a single GPU VM, with the gateway in place from day one; switching to self-hosted is then a config change.
  • One OpenAI-compatible contract for every consumer; model names live in the gateway.
  • Treat models like container images: pinned, signed, scanned, promoted.
  • Separate pools: latency-sensitive chat, batch jobs, embeddings. Use MIG or small GPUs for embeddings.
  • Prefer FP8 on Hopper and FP4/MXFP4 on Blackwell; validate quantized quality with the eval suite (Local AI Best Practices).
  • Disable engine usage-stats telemetry and set offline mode in every inference container.

Pitfalls to design around​

  • Pin the gateway supply chain. LiteLLM releases 1.82.7 and 1.82.8 on PyPI were compromised on 2026-03-24 (BerriAI/litellm#24518). Install from a mirrored, hash-pinned lockfile or a digest-pinned image, and scan before promotion.
  • Size KV cache explicitly. Fit the model and the expected concurrent context, not just the weights.
  • Keep GPU nodes on Ubuntu node pools on SKE and plan for non-live-migratable GPU VMs with replica counts ≥ 2.
  • Budget re-indexing time whenever the embedding model changes.
  • Track shared-model deprecations on STACKIT AI Model Serving and keep aliases in the gateway.
  • Agree on front-end licensing before branding work starts.

Definition of done​

  • All infrastructure (cluster, node pools, buckets, tokens, DNS) created from Terraform/OpenTofu in Git; plan is clean
  • GPU Operator, engine, gateway and front-end versions pinned; images mirrored and digest-pinned
  • Model artifacts pinned by revision, checksummed, signed, and verified at pod start
  • Inference namespaces default-deny; no internet egress from engine pods
  • SSO with MFA for the UI; per-team virtual keys with budgets and rate limits for APIs
  • Prompt bodies not persisted by the gateway; retention policy configured in the front-end
  • Dashboards for TTFT, ITL, queue time, KV-cache usage, GPU health, spend per team; alerts routed
  • vllm bench serve results at target concurrency stored as baseline; SLO met
  • Eval suite runs in CI on model/engine changes; results tracked in MLflow
  • Model rollout and rollback runbook rehearsed once (canary + revert alias)
  • GDPR/NIS2 control table filled with evidence links for this customer
  • Cost model filled with the customer's real traffic and current GPU/token prices

Sources​