Private and Sovereign AI Platforms
Service: Sovereign & Private AI · On-Premise LLM Deployment · Sovereign AI on STACKIT · Typical shape: 2–4 week PoC on one GPU node, then 6–12 weeks to a production platform (IaC, CI/CD, SSO, observability) with a run/operate option
This page is the implementation playbook for all three private-AI offers. They share one reference architecture; what changes is where the GPUs live:
| Offer | GPUs | Kubernetes | Model API |
|---|---|---|---|
| On-Premise LLM Deployment | Customer data centre (bare metal) | Upstream K8s, Canonical Kubernetes, RKE2 or OpenShift | Self-hosted vLLM/SGLang |
| Sovereign AI on STACKIT | STACKIT GPU VMs (EU01, Germany) | SKE (STACKIT Kubernetes Engine) | Self-hosted, plus STACKIT AI Model Serving for shared models |
| Sovereign & Private AI | Either of the above, or hybrid | Same | Gateway in front of both |
Read Serving at Scale for engine tuning and autoscaling, and Local AI Best Practices for model selection, quantization and evaluation habits. This page focuses on the platform around the engine.
What we deliver
- A private LLM platform exposing an OpenAI-compatible API behind an authenticated gateway, with per-team keys, quotas and usage reporting.
- A chat front-end (Open WebUI or a custom app) wired to corporate SSO.
- RAG building blocks: embedding model, reranker, vector database, ingestion pipeline (see RAG Systems).
- Model supply chain: internal model registry, mirroring, checksum and signature verification, promotion from staging to production.
- Infrastructure as code (Terraform/OpenTofu + Helm/Argo CD) for every component, plus runbooks for upgrades and model rollouts.
- Compliance evidence pack: data-flow diagram, records of processing input, NIS2-relevant technical controls mapped to the implementation.
- Cost model: GPU-hour vs per-token comparison for the customer's actual traffic.
Reference architecture
| Layer | Default choice (as of 2026-10) | Alternatives | Why the default |
|---|---|---|---|
| GPU nodes | NVIDIA data-centre GPUs, Ubuntu 24.04 | AMD MI300X/MI325X with ROCm | Widest engine and quantization support |
| GPU enablement | NVIDIA GPU Operator v26.7.x | Pre-baked driver images | Manages driver, container toolkit, device plugin, DCGM exporter, MIG |
| Inference engine | vLLM ≥ 0.31.0 | SGLang 0.5.x for agentic/prefix-heavy loads | Broadest model coverage, OpenAI-compatible server |
| Multi-replica routing | vLLM production-stack or llm-d | Plain K8s Service | KV-cache-aware routing, prefill/decode disaggregation |
| LLM gateway | LiteLLM proxy (pinned, see pitfalls) | Envoy AI Gateway, Kong AI Gateway | Virtual keys, budgets, routing to self-hosted and hosted backends |
| Front-end | Open WebUI 0.12.x | LibreChat, custom React app | SSO, RAG, tools, admin out of the box |
| Vector DB | PostgreSQL + pgvector | Qdrant, OpenSearch | One less stateful system to run; good to tens of millions of vectors |
| Observability | kube-prometheus-stack, DCGM exporter, Loki, OpenTelemetry | STACKIT Observability | vLLM and DCGM export Prometheus metrics natively |
| Identity | Corporate IdP via OIDC; Keycloak when none fits | Dex | Central joiner/mover/leaver control |
| Experiment tracking | MLflow 3.x (self-hosted or STACKIT AI Model Experiments) | Langfuse | Prompt registry, LLM tracing, eval runs |
Hardware sizing
GPU choice
| GPU | Memory | Bandwidth | Interconnect | Sweet spot |
|---|---|---|---|---|
| L40S | 48 GB GDDR6 | 864 GB/s | PCIe | 7–32B models (FP8), embeddings, rerankers; cheap per node |
| A100 80GB | 80 GB HBM2e | ~2 TB/s | NVLink (SXM) / PCIe | Existing fleets; no FP8 tensor cores, prefer BF16/INT8/AWQ |
| H100 SXM | 80 GB HBM3 | 3.35 TB/s | NVLink | gpt-oss-120b on one GPU; 70B FP8 on 2; TP across 8 |
| H100 NVL | 94 GB HBM3 | 3.9 TB/s | NVLink bridge pairs | PCIe servers needing H100-class bandwidth |
| H200 | 141 GB HBM3e | 4.8 TB/s | NVLink | 70B FP8 on one GPU with large KV cache; long context |
| RTX PRO 6000 Blackwell Server | 96 GB GDDR7 | 1.6 TB/s | PCIe only, MIG up to 4 | Standard air-cooled servers, FP4, many mid-size models per node |
| B200 | ~180 GB HBM3e (1,440 GB per DGX B200) | ~8 TB/s | NVLink 5 | Frontier-size MoE (DeepSeek/Kimi/Qwen 235B+ class), highest throughput |
| B300 (Blackwell Ultra) | ~262 GB (2.1 TB per DGX B300) | HBM3e | NVLink 5 | Largest MoE with long context; liquid-cooled facilities |
Sources: NVIDIA product pages for L40S, H100, H200, RTX PRO 6000 Server, DGX B200, DGX B300. Per-GPU B200/B300 memory is derived from the 8-GPU system totals.
Rules of thumb for decode-bound LLM serving:
- Memory first, then bandwidth. Decode throughput per replica scales roughly with memory bandwidth; capacity decides whether the model plus KV cache fits.
- Weights:
params × bytes_per_param→ 70B at FP8 ≈ 70 GB; at FP4/MXFP4 ≈ 35–40 GB. gpt-oss-120b (117B total, 5.1B active, MXFP4 MoE weights) fits a single 80 GB GPU. - KV cache per token:
2 × layers × kv_heads × head_dim × bytes. For a Llama-3.3-70B-class model (80 layers, 8 KV heads, head_dim 128) at FP8 KV: 2 × 80 × 8 × 128 × 1 B ≈ 160 KB/token. 32 concurrent users × 8k context ≈ 256k tokens ≈ 41 GB of KV cache. - Headroom: budget 10–15 % of VRAM for activations, CUDA graphs and fragmentation (vLLM
--gpu-memory-utilization 0.90). - Interconnect: tensor parallelism across PCIe-only cards (L40S, RTX PRO 6000) costs throughput; prefer models that fit in one card, or use pipeline/data parallelism instead.
Worked sizing example
Customer: 400 knowledge workers, peak 40 concurrent streams, RAG with 8k-token prompts, 600-token answers, target p95 time-to-first-token under 2 s.
| Decision | Result |
|---|---|
| Model | gpt-oss-120b (reasoning, tool calling, Apache 2.0) or Qwen3.x 27B-class dense for lower latency |
| Weights | ~65 GB on one H100 80GB → too little KV headroom for 40 × 8.6k tokens |
| Choice | 2 × H100 (two replicas, data parallel) or 1 × H200 per replica |
| Embeddings + reranker | 1 × L40S (bge-m3 / Qwen3 embedding + reranker) |
| Validation | Run vllm bench serve with the customer's prompt-length distribution before ordering hardware |
Implementation steps
1. Provision infrastructure as code
STACKIT (provider stackitcloud/stackit, v0.120.0 released 2026-10-15). GPU node pools on SKE must use the Ubuntu image for the GPU Operator (STACKIT: Use NVIDIA GPUs).
terraform {
required_providers {
stackit = { source = "stackitcloud/stackit", version = "~> 0.120" }
time = { source = "hashicorp/time", version = "~> 0.12" }
}
}
provider "stackit" {
default_region = "eu01"
service_account_key_path = var.sa_key_path # service account key file
}
resource "stackit_ske_cluster" "ai" {
project_id = var.project_id
name = "ai-prod"
kubernetes_version_min = var.k8s_version
node_pools = [
{
name = "system"
machine_type = var.system_flavor # current general-purpose flavor
minimum = 3
maximum = 5
availability_zones = ["eu01-1", "eu01-2", "eu01-3"]
max_surge = 3
},
{
name = "gpu-h100"
machine_type = "n3.104d.g8" # 8x H100 HGX 80 GB
os_name = "ubuntu" # required for GPU Operator
minimum = 1
maximum = 2
availability_zones = ["eu01-1"]
max_surge = 1
volume_type = "storage_premium_perf6"
volume_size = 500
labels = { "workload" = "llm" }
taints = [{ key = "nvidia.com/gpu", value = "present", effect = "NoSchedule" }]
},
]
extensions = { acl = { enabled = true, allowed_cidrs = var.admin_cidrs } }
# STACKIT Network Area (SNA) control plane: availability depends on the account; check provider docs
# network = { control_plane = { access_scope = "SNA" } }
maintenance = {
enable_kubernetes_version_updates = true
enable_machine_image_version_updates = true
start = "01:00:00Z"
end = "03:00:00Z"
}
}
resource "time_rotating" "ms" { rotation_days = 80 }
resource "stackit_modelserving_token" "gateway" {
project_id = var.project_id
name = "litellm-gateway"
rotate_when_changed = { rotation = time_rotating.ms.id }
}
output "model_serving_token" {
value = stackit_modelserving_token.gateway.token
sensitive = true
}
STACKIT GPU flavors in EU01 (as of 2026-10, machine types):
| Flavor | GPUs | vCPU / RAM |
|---|---|---|
n1.14d.g1 / n1.28d.g2 / n1.56d.g4 | 1/2/4 × A100 80 GB | 14/28/56 vCPU, 108/216/460 GB |
n2.14d.g1 / n2.28d.g2 / n2.56d.g4 | 1/2/4 × L40S 48 GB | 14/28/56 vCPU, 50/100/200 GB |
n3.104d.g8 | 8 × H100 HGX 80 GB | 104 vCPU, 1,900 GB, 1,536 GB local disk |
GPU VMs are not live-migrated (per STACKIT guidance; confirm in current docs); plan maintenance windows and run at least two replicas of anything user-facing. n1.56d.g4 is requested via Help Center ticket.
On-premise: same Terraform layout with the provider for the customer's virtualisation (vSphere, OpenStack, MAAS for bare metal) and Canonical Kubernetes, RKE2 or kubeadm. Keep GPU nodes in a dedicated node pool with the same taint so manifests are identical across sites.
2. Enable GPUs
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update
helm install gpu-operator nvidia/gpu-operator \
-n gpu-operator --create-namespace --wait \
--version v26.7.2 \
--set dcgmExporter.serviceMonitor.enabled=true
kubectl get nodes -o json | jq '.items[].status.allocatable["nvidia.com/gpu"]'
Check the GPU Operator component matrix against the node OS, kernel and containerd version before bumping; STACKIT's guide uses v26.3.3 as its example, so confirm v26.7.x on SKE against the matrix before rollout. Use MIG profiles on H100/H200/RTX PRO 6000 for small embedding/reranker models.
3. Deploy the inference engine
apiVersion: apps/v1
kind: Deployment
metadata: { name: vllm-gpt-oss-120b, namespace: llm }
spec:
replicas: 2
selector: { matchLabels: { app: vllm-gpt-oss-120b } }
template:
metadata: { labels: { app: vllm-gpt-oss-120b } }
spec:
tolerations: [{ key: nvidia.com/gpu, operator: Exists, effect: NoSchedule }]
containers:
- name: vllm
image: registry.internal/mirror/vllm/vllm-openai:v0.31.0 # mirrored, digest-pinned in prod
args:
- --model=/models/openai/gpt-oss-120b/2026-09-01
- --served-model-name=gpt-oss-120b
- --gpu-memory-utilization=0.90
- --max-model-len=65536
- --enable-prefix-caching
- --tool-call-parser=openai
- --enable-auto-tool-choice
env:
- { name: HF_HUB_OFFLINE, value: "1" }
- { name: VLLM_NO_USAGE_STATS, value: "1" }
ports: [{ containerPort: 8000, name: http }]
readinessProbe: { httpGet: { path: /health, port: http }, periodSeconds: 10 }
resources: { limits: { nvidia.com/gpu: 1 } }
volumeMounts: [{ name: models, mountPath: /models, readOnly: true }]
volumes:
- name: models
persistentVolumeClaim: { claimName: model-store }
- vLLM ships releases frequently (latest 0.32.0 on 2026-10-18); this page uses ≥ 0.31.0 as the floor. Check each release's notes for security fixes before pinning (PyPI, releases).
HF_HUB_OFFLINE=1and a local path guarantee the pod never reaches Hugging Face.- For more than two replicas, put the vLLM production-stack router or llm-d in front for prefix-aware routing; details in Serving at Scale.
4. Put the gateway in front
model_list:
- model_name: gpt-oss-120b
litellm_params:
model: hosted_vllm/gpt-oss-120b
api_base: http://vllm-gpt-oss-120b.llm.svc:8000/v1
- model_name: embed
litellm_params:
model: hosted_vllm/Qwen/Qwen3-Embedding-8B
api_base: http://vllm-embed.llm.svc:8000/v1
# Overflow / extra models on STACKIT AI Model Serving (EU01)
- model_name: qwen3-vl-235b
litellm_params:
model: openai/Qwen/Qwen3-VL-235B-A22B-Instruct-FP8
api_base: https://api.openai-compat.model-serving.eu01.onstackit.cloud/v1
api_key: os.environ/STACKIT_MODEL_SERVING_TOKEN
router_settings:
routing_strategy: least-busy
num_retries: 2
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
database_url: os.environ/DATABASE_URL # Postgres for keys, budgets, spend logs
litellm_settings:
turn_off_message_logging: true # do not persist prompt/response bodies
Issue one virtual key per application/team with max_budget, tpm_limit, rpm_limit, and an allowed model list. Users of the chat UI authenticate through SSO; machine clients use virtual keys stored in the secrets manager.
5. Front-end with SSO
extraEnvVars:
- { name: ENABLE_OAUTH_SIGNUP, value: "true" }
- { name: OAUTH_PROVIDER_NAME, value: "Corporate SSO" }
- { name: OPENID_PROVIDER_URL, value: "https://login.example.com/.well-known/openid-configuration" }
- { name: OAUTH_CLIENT_ID, valueFrom: { secretKeyRef: { name: owui-oidc, key: client_id } } }
- { name: OAUTH_CLIENT_SECRET, valueFrom: { secretKeyRef: { name: owui-oidc, key: client_secret } } }
- { name: ENABLE_OAUTH_ROLE_MANAGEMENT, value: "true" }
- { name: OPENAI_API_BASE_URL, value: "http://litellm.llm.svc:4000/v1" }
- { name: ENABLE_LOGIN_FORM, value: "false" }
Open WebUI ships under its own BSD-3-based licence with a branding clause: deployments above 50 end users in a rolling 30-day window must keep the Open WebUI branding unless an enterprise licence is obtained. Settle this with the customer before white-labelling; use LibreChat or a custom front-end if rebranding is required.
6. Air-gapped model distribution
-
Pin by revision. Never download
main; pin the commit hash.hf download openai/gpt-oss-120b --revision <commit-sha> \--local-dir /stage/openai/gpt-oss-120b/2026-09-01 -
Accept only safetensors (no pickle
.bin/.pt), and keeptrust_remote_codeoff unless the code has been reviewed and vendored. -
Checksum and manifest. Produce
SHA256SUMSover every file, plus a manifest with source repo, revision, licence, and intended served name.cd /stage/openai/gpt-oss-120b/2026-09-01find . -type f ! -name SHA256SUMS -print0 | sort -z | xargs -0 sha256sum > SHA256SUMScosign sign-blob --key kms://... --output-signature SHA256SUMS.sig SHA256SUMS -
Ship as an OCI artifact (
oras push) or to an S3 bucket with object lock. Container images go through the same registry mirror (Harbor), digest-pinned. -
Verify at load time. An init container runs
cosign verify-blobandsha256sum -c SHA256SUMSbefore vLLM starts; a mismatch fails the pod. -
Promote by tag after the eval suite passes in staging; production manifests reference the dated path, never
latest.
7. Network isolation
- GPU and model namespaces get a default-deny
NetworkPolicy; allow ingress only from the gateway namespace and egress only to the model store, DNS, and the OTel collector. - No internet egress from inference pods. On STACKIT, restrict cluster egress with security groups and check
egress_address_rangesof the SKE cluster for allow-listing at downstream systems. - Restrict the Kubernetes API: on STACKIT enable the SKE ACL extension (
extensions.acl.allowed_cidrs), or control plane access scopeSNA(STACKIT Network Area, private preview) where enabled; on-premise expose the API server only to the admin network/VPN. - TLS everywhere: ingress terminates TLS with cert-manager; service mesh (Istio ambient or Linkerd) for mTLS between gateway, RAG and engines when the customer's threat model requires it.
- If STACKIT AI Model Serving is used, it is a public HTTPS endpoint in EU01: route it via the gateway only, with the token in the secrets manager, so applications never hold the token.
8. Observability and evaluation
- Scrape vLLM
/metrics(time-to-first-token, inter-token latency, queue length, KV-cache usage) and DCGM (DCGM_FI_DEV_GPU_UTIL, memory, temperature, XID errors). - Gateway spend logs → per-team usage dashboards and chargeback.
- Alert on: queue time p95, KV-cache usage > 95 % sustained, XID errors, 5xx rate at the gateway.
- Keep an offline eval suite (golden questions, RAG faithfulness, tool-call accuracy) and run it on every model or engine change; track runs in MLflow.
STACKIT specifics
| Product | What it is (as of 2026-10) | How we use it |
|---|---|---|
| AI Model Serving | Shared open-weight models behind an OpenAI-compatible API in EU01, pay per token; STACKIT states prompts are not stored or used for training (confirm in the STACKIT FAQ before quoting) | Fast start, overflow capacity, large models not worth self-hosting |
| Compute Engine GPU | A100, L40S, H100 flavors (table above) | Dedicated inference for fixed models, fine-tuning |
| SKE | Managed Kubernetes, Ubuntu node pools for GPUs (required by the GPU Operator) | Platform runtime |
| AI Model Experiments | Managed MLflow tracking server, artifacts in a STACKIT Object Storage bucket (release stage unconfirmed) | Experiment tracking, prompt registry, LLM tracing |
| Object Storage, Secrets Manager, KMS, Observability | S3-compatible storage, secrets management, key management, observability service | Model store, secrets, encryption keys, monitoring |
Shared models on AI Model Serving (docs, as of 2026-10):
| Model ID | Tier | Context | Limits | Price € / 1M in · out (third-party, see note) |
|---|---|---|---|---|
Qwen/Qwen3-VL-235B-A22B-Instruct-FP8 | Premium | 200K | 350k TPM, 30 RPM | 1.50 · 1.75 |
Qwen/Qwen3.8-27B | Plus | 262K | 350k TPM, 45 RPM | unconfirmed |
openai/gpt-oss-120b | Plus | 131K | 200k TPM, 30 RPM | 0.45 · 0.65 |
google/gemma-4-31B-it | Plus | 256K | 200k TPM, 80 RPM | 0.45 · 0.65 |
cortecs/Llama-3.3-70B-Instruct-FP8-Dynamic | Plus | 128K | 200k TPM, 80 RPM | 0.45 · 0.65 |
openai/gpt-oss-20b | Standard | 131K | 200k TPM, 80 RPM | 0.15 · 0.25 |
intfloat/e5-mistral-7b-instruct | Embedding | 4k input | 600 RPM | 0.02 |
Qwen/Qwen3-VL-Embedding-8B | Embedding | 32k input | 600 RPM | 0.08 |
Prices come from the European Cloud price table (updated 2026-08-29). STACKIT's shared-models page lists limits but no prices, and the Qwen3.8 27B price was not found in any source; confirm all prices in the STACKIT portal before quoting.
Two details that matter for capacity planning: TPM counts output tokens with a weight of 5, and Qwen/Qwen3.6-27B is marked deprecated on the shared-models page — keep model names in gateway config, never in application code, so swaps are a one-line change.
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.openai-compat.model-serving.eu01.onstackit.cloud/v1",
api_key=os.environ["STACKIT_MODEL_SERVING_TOKEN"],
)
resp = client.chat.completions.create(
model="openai/gpt-oss-120b",
messages=[{"role": "user", "content": "Summarise NIS2 incident reporting deadlines."}],
max_completion_tokens=400,
)
print(resp.choices[0].message.content)
GDPR and NIS2 technical controls
| Requirement | Technical control in this platform |
|---|---|
| GDPR Art. 5(1)(c) data minimisation | Gateway does not persist prompt bodies (turn_off_message_logging); RAG indexes only approved sources; PII redaction (Presidio) in ingestion where needed |
| GDPR Art. 25 privacy by design | Private endpoints, default-deny network, no third-party processors in the inference path |
| GDPR Art. 28 processors | Self-hosted: no AI processor. STACKIT: DPA with STACKIT, EU01 region only |
| GDPR Art. 30 / 35 | Data-flow diagram and model inventory feed the RoPA and DPIA |
| GDPR Art. 32 security | TLS + encryption at rest (KMS-backed volumes and buckets), RBAC via IdP groups, audit logs |
| GDPR Art. 17 erasure | Chat history retention policy in the front-end; document deletion propagates to vector DB by source ID |
| NIS2 Art. 21(2)(d) supply chain | Signed model manifests, mirrored and digest-pinned images, SBOMs for gateway and front-end |
| NIS2 Art. 21(2)(i) access control | SSO with MFA, per-team virtual keys, short-lived tokens (ttl_duration, time_rotating) |
| NIS2 Art. 21(2)(b) incident handling, Art. 23 reporting | Centralised logs with retention, alerting, runbook mapping to 24 h early warning / 72 h notification |
| NIS2 Art. 21(2)(c) business continuity | Multi-replica engines, IaC rebuild of the whole stack, model store backup |
| EU AI Act transparency (Art. 50) | UI labels AI-generated content; audit trail of model versions per answer |
Operations
Upgrades. Pin engine, gateway, front-end and GPU Operator versions in Git. Upgrade in staging first, run vllm bench serve and the eval suite, compare against the stored baseline, then roll production with maxUnavailable: 0. GPU Operator and node OS upgrades go node pool by node pool inside the maintenance window.
Model rollouts.
- Clients call stable aliases (
chat-default,chat-reasoning,embed); the gateway maps aliases to concrete models. - Changing an embedding model requires a full re-index; run old and new indexes side by side and switch the alias when the new index is complete.
- Keep the previous model deployed (scaled to zero or one) for fast rollback during the first week.
Capacity. Watch queue time and KV-cache usage, not GPU utilisation. Scale out replicas on queue depth (KEDA on vllm:num_requests_waiting).
Cost model: GPU-hour vs per-token
Break-even monthly volume:
T_breakeven = (N_gpu × price_gpu_hour × 730 + C_ops) / price_per_token_blended
price_per_token_blended = (r_in × p_in + r_out × p_out) / (r_in + r_out)
Worked example (illustrative GPU price — replace with the current STACKIT or hardware quote):
| Input | Value |
|---|---|
| Model | gpt-oss-120b on 1 × H100 80GB |
| GPU cost (assumed placeholder, not a published STACKIT price) | €4.00 / GPU-hour → €2,920 / month |
| Ops overhead attributed to this model | €1,000 / month |
| Per-token alternative | STACKIT AI Model Serving gpt-oss-120b: €0.45 in, €0.65 out per 1M |
| Traffic mix (RAG) | 4 input : 1 output → blended €0.49 per 1M tokens |
T_breakeven = (2,920 + 1,000) / 0.49 ≈ 8,000 M tokens/month ≈ 8 B tokens/month, i.e. about 3,050 tokens/s sustained 24×7.
- The 400-user example above produces roughly 400 users × 30 requests/day × 9k tokens × 22 days ≈ 2.4 B tokens/month → per-token is cheaper for that model alone.
- Self-hosting wins when volume exceeds the break-even, when one GPU pool serves several workloads (batch summarisation at night), when the model or a fine-tune is not offered as shared model, or when the data may not leave the customer's premises at all.
- Measure the actual sustainable throughput of the chosen GPU at the target latency SLO with
vllm bench serve; a GPU that cannot reach the break-even token rate at the SLO can never pay off on cost alone.
Best practices
- Start the PoC on STACKIT AI Model Serving or a single GPU VM, with the gateway in place from day one; switching to self-hosted is then a config change.
- One OpenAI-compatible contract for every consumer; model names live in the gateway.
- Treat models like container images: pinned, signed, scanned, promoted.
- Separate pools: latency-sensitive chat, batch jobs, embeddings. Use MIG or small GPUs for embeddings.
- Prefer FP8 on Hopper and FP4/MXFP4 on Blackwell; validate quantized quality with the eval suite (Local AI Best Practices).
- Disable engine usage-stats telemetry and set offline mode in every inference container.
Pitfalls to design around
- Pin the gateway supply chain. LiteLLM releases 1.82.7 and 1.82.8 on PyPI were compromised on 2026-03-24 (BerriAI/litellm#24518). Install from a mirrored, hash-pinned lockfile or a digest-pinned image, and scan before promotion.
- Size KV cache explicitly. Fit the model and the expected concurrent context, not just the weights.
- Keep GPU nodes on Ubuntu node pools on SKE and plan for non-live-migratable GPU VMs with replica counts ≥ 2.
- Budget re-indexing time whenever the embedding model changes.
- Track shared-model deprecations on STACKIT AI Model Serving and keep aliases in the gateway.
- Agree on front-end licensing before branding work starts.
Definition of done
- All infrastructure (cluster, node pools, buckets, tokens, DNS) created from Terraform/OpenTofu in Git;
planis clean - GPU Operator, engine, gateway and front-end versions pinned; images mirrored and digest-pinned
- Model artifacts pinned by revision, checksummed, signed, and verified at pod start
- Inference namespaces default-deny; no internet egress from engine pods
- SSO with MFA for the UI; per-team virtual keys with budgets and rate limits for APIs
- Prompt bodies not persisted by the gateway; retention policy configured in the front-end
- Dashboards for TTFT, ITL, queue time, KV-cache usage, GPU health, spend per team; alerts routed
-
vllm bench serveresults at target concurrency stored as baseline; SLO met - Eval suite runs in CI on model/engine changes; results tracked in MLflow
- Model rollout and rollback runbook rehearsed once (canary + revert alias)
- GDPR/NIS2 control table filled with evidence links for this customer
- Cost model filled with the customer's real traffic and current GPU/token prices
Sources
- STACKIT AI Model Serving — Available shared models, Use the models, FAQ, product page and pricing
- European Cloud: A closer look at STACKIT AI Model Serving (price table)
- STACKIT machine types EU01, STACKIT GPU comparison
- STACKIT SKE: Use NVIDIA GPUs
- STACKIT AI Model Experiments
- Terraform provider: stackitcloud/terraform-provider-stackit, stackit_ske_cluster, stackit_modelserving_token
- NVIDIA GPU Operator, installation guide
- NVIDIA specs: L40S, H100, H200, RTX PRO 6000 Blackwell Server Edition, DGX B200, DGX B300
- vLLM on PyPI, vLLM releases, vLLM production-stack, llm-d
- openai/gpt-oss-120b model card
- LiteLLM docs, LiteLLM PyPI compromise timeline (issue #24518), Datadog Security Labs analysis
- Open WebUI repository and licence