Serving LLMs at Scale on Kubernetes
Service: On-Premise LLM Deployment, Sovereign AI on STACKIT, AI Productification · Typical shape: production build after a successful PoC; platform team of 2–4 engineers, IaC and GitOps from day one
A single vllm serve on one GPU is a PoC. Production means: several models and LoRA adapters behind one API, routing that understands KV caches, autoscaling on the signals that actually predict latency, models that load in seconds rather than minutes, GPUs shared where it is safe, and rollouts that never drop a streaming response. This page is the playbook for that platform on Kubernetes — on the customer's own GPU nodes, on OpenStack, or on STACKIT SKE with GPU node pools.
Prerequisites: Local AI Basics, Inference Engineering. Platform standards: How We Build.
What we deliver
| Deliverable | Content |
|---|---|
| Serving platform | Kubernetes stack (vLLM + llm-d or KServe) deployed by GitOps, with GPU operator, gateway, monitoring |
| Model catalogue | Each model/adapter as a versioned artifact (OCI or PVC cache) with pinned engine flags and benchmark results |
| Routing layer | OpenAI-compatible gateway with auth, quotas, and inference-aware (KV/prefix/LoRA-aware) routing |
| Autoscaling | Queue- and KV-based scaling with tested scale-up time and scale-to-zero policy where appropriate |
| Runbooks | Rollout, drain, GPU failure, model rollback, capacity planning |
Reference architecture
Two tiers is the pattern we recommend: hot models (high traffic, SLO-bound) run as dedicated autoscaled vLLM pools behind an inference-aware gateway; long-tail models (occasional use, experiments) share GPUs through dynamic loading (llama-swap, Ollama, llama.cpp router, vLLM sleep mode).
1. Choose the engine
| vLLM | SGLang | TensorRT-LLM | llama.cpp (llama-server) | |
|---|---|---|---|---|
| Version (as of 2026-10) | v0.32.0 (2026-10-18) | v0.5.21 (GitHub); 0.5.22 on PyPI | 1.2.x stable (1.2.1 on PyPI) | rolling builds |
| Sweet spot | Default for GPU serving; broadest model + hardware coverage (NVIDIA, AMD, TPU, Intel) | Prefix-heavy agentic/multi-turn, large MoE, PD disaggregation | Max performance on NVIDIA datacenter GPUs | Edge, CPU, Apple, Vulkan, small deployments, many GGUF models |
| Prefix cache | Hash-based APC | RadixAttention (Rust core) | Yes | Per-slot cache reuse |
| Disaggregated PD | Yes (KV connectors, NIXL) | Yes (role-switching PD) | Yes (via Dynamo) | No |
| Multi-LoRA | Yes, dynamic load | Yes | Yes | Yes (static) |
| Ecosystem | llm-d, KServe, production-stack, Dynamo, Ray Serve | llm-d, Dynamo, Ray Serve | Dynamo, Triton | llama-swap, Ollama |
| Operational cost | Low | Low–medium | Higher (NVIDIA-only, faster-moving APIs) | Very low |
Recommendation: vLLM by default. Benchmark SGLang when the workload is dominated by shared prefixes or large MoE. Consider TensorRT-LLM only on large NVIDIA fleets where the extra throughput pays for the extra operational surface. Use llama.cpp for the long-tail tier and for non-NVIDIA edge hardware. TGI is archived — migrate existing TGI deployments to vLLM.
2. Choose the Kubernetes serving stack
| Stack | What it is (as of 2026-10) | Choose when |
|---|---|---|
| Plain Deployment + Gateway API Inference Extension | vLLM Deployments grouped into InferencePools (API GA at inference.networking.k8s.io/v1, latest v1.6.x); an Endpoint Picker (EPP) scores pods on KV usage, queue length, active LoRAs | Small–medium platforms; you want standard Kubernetes primitives and minimum moving parts |
| llm-d | CNCF Sandbox project (Red Hat, Google, IBM, NVIDIA, CoreWeave…); vLLM + Inference Gateway + LeaderWorkerSet with "well-lit paths": intelligent scheduling, precise prefix-cache routing, PD disaggregation, wide EP, tiered KV cache. v0.10.0 (2026-09-29) | Production platforms that need KV-aware routing, PD disaggregation, or large MoE across nodes. Our default for multi-replica production |
| KServe | CNCF model-serving platform; LLMInferenceService integrates llm-d; InferenceService + Hugging Face runtime; modelcars (OCI); LocalModelCache. v0.21 (2026-09) | Customer already runs KServe / OpenShift AI, or needs one API for LLMs and classic ML models |
| vLLM production-stack | Helm chart (vllm-stack, 0.1.13) with a router (round-robin, session, prefix-aware, KV-aware via LMCache) and Grafana dashboards | Fast start for a vLLM-only cluster; smaller teams |
| NVIDIA Dynamo | Distributed inference framework (v1.5.1 stable, 2026-10-07) with KV router, PD disaggregation, SLA-driven planner, multi-engine (vLLM, SGLang, TRT-LLM) | Large NVIDIA fleets, especially with TensorRT-LLM or GB200 NVL racks |
| Ray Serve LLM | ray[llm]: build_openai_app over vLLM; Python-native autoscaling, multi-model, multi-LoRA | Teams already on Ray (Anyscale, KubeRay); mixed Python pipelines |
Decision rule: start with vLLM + Inference Gateway for one or two models; move to llm-d (directly or through KServe LLMInferenceService) when you need prefix-aware routing at scale, PD disaggregation, or multi-node MoE. Whichever you choose, keep the client contract at the OpenAI API so the stack is replaceable.
3. Inference-aware routing (Gateway API Inference Extension)
Round-robin load balancing is wrong for LLMs: requests differ by 100× in cost, and a replica that already holds a prompt's prefix in KV cache serves it far faster. The Gateway API Inference Extension adds:
InferencePool— a set of model-server pods (like a Service, but for inference) plus a reference to its EPP.- Endpoint Picker (EPP) — an ext-proc service the gateway (Envoy-based: kgateway, Istio, GKE, agentgateway…) calls per request. It scores pods by queue depth, KV-cache utilisation, prefix-cache affinity, and loaded LoRA adapters.
- Flow control / priorities — critical vs sheddable traffic.
apiVersion: inference.networking.k8s.io/v1
kind: InferencePool
metadata:
name: qwen38-27b
namespace: llm
spec:
selector:
matchLabels:
app: vllm-qwen38-27b
targetPorts:
- number: 8000
endpointPickerRef:
name: qwen38-27b-epp
port:
number: 9002
failureMode: FailOpen
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: qwen38-27b
namespace: llm
spec:
parentRefs:
- name: inference-gateway
rules:
- backendRefs:
- group: inference.networking.k8s.io
kind: InferencePool
name: qwen38-27b
llm-d's scheduler builds on the EPP with precise prefix-cache scoring: it subscribes to KV-cache events from each vLLM pod and knows which blocks live where, instead of estimating from traffic. llm-d reported large TTFT reductions and roughly doubled throughput against round-robin on shared-prefix workloads. Combine it with prompts designed static-first (see Inference Engineering).
4. Autoscaling on the right signals
GPU utilisation is a poor scaling signal: a bandwidth-bound decode shows "100 %" at both 2 and 60 concurrent sequences. Scale on queue depth and KV-cache usage, which move before latency does.
| Signal (vLLM metric) | Meaning | Use |
|---|---|---|
vllm:num_requests_waiting | Requests queued, not yet scheduled | Primary scale-up trigger |
vllm:num_requests_running | Requests in the batch | Combine with waiting to avoid scale-down mid-stream |
vllm:kv_cache_usage_perc | KV cache fill (0–1) | Secondary trigger; > 0.9 sustained = preemptions coming |
vllm:time_to_first_token_seconds (p95) | SLO itself | Alerting; lagging as a scaling signal |
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: vllm-qwen38-27b
namespace: llm
spec:
scaleTargetRef:
name: vllm-qwen38-27b
minReplicaCount: 1
maxReplicaCount: 6
pollingInterval: 10
cooldownPeriod: 600
advanced:
horizontalPodAutoscalerConfig:
behavior:
scaleDown:
stabilizationWindowSeconds: 600
triggers:
- type: prometheus
name: queue
metadata:
serverAddress: http://prometheus.monitoring.svc:9090
query: sum(vllm:num_requests_waiting{namespace="llm",pod=~"vllm-qwen38-27b.*"}) or vector(0)
metricType: AverageValue
threshold: "4"
- type: prometheus
name: kv
metadata:
serverAddress: http://prometheus.monitoring.svc:9090
query: avg(vllm:kv_cache_usage_perc{namespace="llm",pod=~"vllm-qwen38-27b.*"}) or vector(0)
threshold: "0.85"
- KEDA takes the maximum replica count across triggers.
- Set the thresholds from the benchmark knee (max concurrency per replica at SLO), not from intuition.
- Scale-up time = node provisioning + image pull + weight load + CUDA graph capture. Measure it; with a cold node and a 60 GB model it can be several minutes. Model caching (section 7) and vLLM's weight-preloading features shrink the last two.
- Scale-to-zero needs a signal from in front of the pods (gateway queue or KServe/Knative activator), because a scaled-to-zero vLLM exports no metrics. Use it for long-tail models only.
- llm-d ships a workload-variant autoscaler (
llm-d-autoscaling) and KServeLLMInferenceServicesupports direct KEDA scaling; evaluate them when you adopt those stacks.
5. Dynamic model loading (long-tail tier)
Not every model deserves a dedicated GPU. These tools load models on demand and evict idle ones.
| Tool | Mechanism | Key settings |
|---|---|---|
| Ollama | Loads on request, unloads after idle | keep_alive per request (default 5 min; 0 = unload now, -1 = forever), OLLAMA_KEEP_ALIVE, OLLAMA_MAX_LOADED_MODELS (default 3 × GPUs), OLLAMA_NUM_PARALLEL (default 1), OLLAMA_MAX_QUEUE (default 512) |
| llama.cpp router mode | llama-server with no model: one process per model, LRU eviction, OpenAI model field selects | --models-dir, --models-preset config.ini, --models-max (default 4), --no-models-autoload; POST /models/load, POST /models/unload, GET /models |
| llama-swap | Go proxy that starts/stops any OpenAI-compatible server (llama-server, vLLM, SGLang, containers) per model | cmd, ttl, groups (swap/exclusive/persistent), aliases, hooks.on_startup.preload |
| vLLM sleep mode | Keep the engine process alive, release GPU memory | --enable-sleep-mode + VLLM_SERVER_DEV_MODE=1; POST /sleep?level=1 (weights to CPU RAM, KV dropped), level=2 (weights discarded), POST /wake_up, GET /is_sleeping. Switch models in seconds instead of a full cold start |
| vLLM dynamic LoRA | Load adapters into a running base model | --enable-lora, VLLM_ALLOW_RUNTIME_LORA_UPDATING=True; POST /v1/load_lora_adapter, POST /v1/unload_lora_adapter; or LoRA resolver plugins (filesystem, HF Hub) |
| KServe ModelMesh | Multi-model placement for many small models (classic ML) | Mostly superseded for LLMs by LLMInferenceService + LoRA; use only for small-model fleets |
| LiteLLM | Gateway routing one OpenAI API across many backends (vLLM pools, llama-swap, Ollama, cloud) | model_list, router_settings.routing_strategy, retries, cooldowns, fallbacks, virtual keys, budgets |
vLLM's /sleep, /wake_up, /collective_rpc and runtime LoRA endpoints require dev mode or runtime-update flags. Bind them to the pod network, block them at the gateway, and drive them only from a trusted controller.
llama-swap example
healthCheckTimeout: 300
logLevel: info
startPort: 10001
macros:
"llama": >
/app/llama-server --port ${PORT} --host 127.0.0.1
--n-gpu-layers 999 --flash-attn on --jinja --metrics
models:
"qwen3-30b-a3b":
cmd: |
${llama}
--model /models/Qwen3-30B-A3B-Q4_K_M.gguf
--ctx-size 65536 --parallel 4
aliases: ["default-chat"]
ttl: 900
"gpt-oss-20b":
cmd: |
${llama}
--model /models/gpt-oss-20b-mxfp4.gguf
--ctx-size 32768 --parallel 2
ttl: 600
"bge-m3-embed":
cmd: |
${llama}
--model /models/bge-m3-Q8_0.gguf --embeddings --ctx-size 8192
ttl: 0
"qwen38-27b-vllm":
cmd: |
docker run --rm --name ${MODEL_ID} --gpus all -p ${PORT}:8000
-v /models/hf:/root/.cache/huggingface
vllm/vllm-openai:v0.32.0 --model Qwen/Qwen3.8-27B-FP8 --max-model-len 16384
cmdStop: docker stop ${MODEL_ID}
useModelName: "Qwen/Qwen3.8-27B-FP8"
ttl: 1800
routing:
router:
use: group
settings:
groups:
"always-on":
swap: false
exclusive: false
persistent: true
members: ["bge-m3-embed"]
"chat":
swap: true
exclusive: false
members: ["qwen3-30b-a3b", "gpt-oss-20b", "qwen38-27b-vllm"]
hooks:
on_startup:
preload: ["bge-m3-embed", "qwen3-30b-a3b"]
- The embedding model stays resident (
persistent,ttl: 0); chat models swap among themselves on one GPU. - Requests to
default-chathitqwen3-30b-a3bthrough the alias. - The first request to a swapped-out model pays the load time — expose it in client timeouts and pre-warm with
preloadfor the morning peak. - Groups sit under
routing.router.settings.groupsin the current llama-swap schema (docs/config.example.yaml); check the example for the version you pin.
llama.cpp router presets
[*]
n-gpu-layers = 999
flash-attn = on
[qwen3-30b-a3b]
model = /models/Qwen3-30B-A3B-Q4_K_M.gguf
ctx-size = 65536
[gpt-oss-20b]
model = /models/gpt-oss-20b-mxfp4.gguf
ctx-size = 32768
llama-server --models-preset models.ini --models-max 2 --host 0.0.0.0 --port 8080
curl -s localhost:8080/models | jq
LiteLLM in front of everything
model_list:
- model_name: chat-large
litellm_params:
model: hosted_vllm/Qwen/Qwen3.8-27B-FP8
api_base: http://inference-gateway.llm.svc/v1
model_info: { allowed_fails: 2, cooldown_time: 30 }
- model_name: chat-small
litellm_params:
model: openai/qwen3-30b-a3b
api_base: http://llama-swap.llm.svc:8080/v1
api_key: none
- model_name: embed
litellm_params:
model: openai/bge-m3-embed
api_base: http://llama-swap.llm.svc:8080/v1
api_key: none
router_settings:
routing_strategy: simple-shuffle
num_retries: 2
fallbacks: [{ "chat-large": ["chat-small"] }]
The gateway gives one API, virtual keys per team, budgets, and fallbacks; the inference gateway below it does KV-aware pod selection. Keep the two responsibilities separate.
6. Multi-LoRA serving
Fine-tuned adapters (see Fine-Tuning) are tens to hundreds of MB; serving them on one base model replaces N full deployments.
VLLM_ALLOW_RUNTIME_LORA_UPDATING=True \
vllm serve Qwen/Qwen3.8-27B-FP8 \
--enable-lora --max-loras 8 --max-lora-rank 64 --max-cpu-loras 32 \
--lora-modules legal=/adapters/legal-v3 support=/adapters/support-v7
curl -X POST http://vllm:8000/v1/load_lora_adapter -H 'Content-Type: application/json' \
-d '{"lora_name": "hr-v1", "lora_path": "/adapters/hr-v1"}'
# Clients select an adapter by model name: {"model": "legal", ...}
--max-loras= adapters active in one batch (GPU memory);--max-cpu-loras= warm cache in host RAM.- Runtime load endpoints affect one pod. In a replicated deployment, declare adapters statically (
--lora-modules), use a resolver plugin that every pod shares, or let the platform manage them (KServe/llm-d LoRA-aware routing, AIBrix ModelAdapter). The EPP routes requests to pods that already hold the adapter. - Version adapters like models: immutable paths or OCI tags, evaluated before release.
7. Model caching and pre-pulling
Downloading 60 GB from Hugging Face on every pod start makes autoscaling and recovery slow and ties production to an external service. Options, best first for sovereign deployments:
| Approach | How | Notes |
|---|---|---|
| OCI image volume | Package weights as an OCI image/artifact in the customer's registry (Harbor); mount with volumes[].image | Kubernetes OCI volume source (ImageVolume, KEP-4639) is GA in 1.36. Runtime support depends on your containerd/CRI-O version; verify it against your distro before relying on it |
| KServe modelcar | storageUri: oci://registry/model:tag; sidecar exposes /models from the image via /proc | Default-enabled in KServe; oci+native:// mode uses image volumes |
| ModelPack (CNCF) | Standard OCI artifact layout for models; modctl build/push; KitOps can pack ModelPack | Registry-native versioning and signing; Dragonfly P2P for large clusters |
| Docker Model Runner OCI | docker model package --gguf ... --push registry/model:tag | Fine for developer and edge distribution |
| Shared RWX PVC | NFS/CephFS/Lustre volume with a pre-populated HF cache, mounted read-only | Simple; throughput of the filesystem bounds load time |
| Node-local cache | Pre-pull DaemonSet or KServe LocalModelCache onto NVMe | Fastest loads; costs disk per node |
Always: mirror models into the customer's registry or storage, pin by digest, set HF_HUB_OFFLINE=1 in production pods so nothing reaches the internet, and pre-pull images and models onto GPU nodes before scaling events. Sign artifacts (cosign) and verify on admission — it closes the "who changed the weights" question auditors ask.
volumes:
- name: model
image:
reference: harbor.example.internal/models/qwen38-27b-fp8@sha256:<digest>
pullPolicy: IfNotPresent
# container: volumeMounts: [{name: model, mountPath: /models/qwen38-27b, readOnly: true}]
8. GPU sharing
| Mechanism | Isolation | Granularity | Use for |
|---|---|---|---|
| Whole GPU | Full | 1 GPU | Production LLM pools — default |
| MIG (A100/H100/H200/B200) | Hardware: memory, cache, SMs, fault isolation | Up to 7 slices | Small models, embeddings, rerankers, multi-tenant dev; predictable latency |
| Time-slicing (GPU operator) | None — shared memory, no fault isolation | N replicas per GPU | Dev/test, notebooks, low-traffic long-tail; never mix tenants |
| MPS | Partial (compute partitioning, shared address-space risks) | Percent of SMs | Many small concurrent inference processes from one tenant |
| Dynamic Resource Allocation (DRA) | Depends on driver | Claims with selectors | Newer clusters: richer GPU requests (MIG profiles, sharing) instead of extended resources |
apiVersion: v1
kind: ConfigMap
metadata:
name: time-slicing-config
namespace: gpu-operator
data:
any: |-
version: v1
flags:
migStrategy: none
sharing:
timeSlicing:
renameByDefault: false
failRequestsGreaterThanOne: true
resources:
- name: nvidia.com/gpu
replicas: 4
vLLM pre-allocates --gpu-memory-utilization of the whole device; on a shared GPU set it explicitly (e.g. 0.45 for two engines) or use MIG.
9. Fault tolerance and graceful drain
- Drain:
--shutdown-timeout(default 0) keeps serving running requests after SIGTERM for up to N seconds; at 0, vLLM aborts in-flight requests immediately. Grace period must exceed preStop + drain + teardown (llm-d uses 15 + 45 + 15 = 75 s). - Probes:
startupProbeon/healthwith a long budget (weights + compile can take minutes);readinessProbeon/health; a conservativelivenessProbeso a long prefill never gets the pod killed. - Rollouts:
maxUnavailable: 0,maxSurge: 1when you have spare GPUs; otherwisemaxSurge: 0, maxUnavailable: 1with a PodDisruptionBudget. Canary new model versions by weight on theHTTPRouteor LiteLLM. - GPU failures: DCGM health checks + GPU operator; Xid errors should cordon the node. Keep ≥ 2 replicas for every SLO-bound model, spread across nodes with topology spread constraints.
- Clients: retry idempotent non-streaming requests at the gateway; for streams, surface the error and let the client resubmit. Set request timeouts above worst-case model load time on the long-tail tier.
- EPP failure:
failureMode: FailOpenkeeps traffic flowing with plain load balancing.
10. Complete example: vLLM on Kubernetes
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: model-cache
namespace: llm
spec:
accessModes: ["ReadWriteMany"]
storageClassName: cephfs
resources:
requests:
storage: 500Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-qwen38-27b
namespace: llm
labels: { app: vllm-qwen38-27b }
spec:
replicas: 2
strategy:
type: RollingUpdate
rollingUpdate: { maxSurge: 0, maxUnavailable: 1 }
selector:
matchLabels: { app: vllm-qwen38-27b }
template:
metadata:
labels: { app: vllm-qwen38-27b }
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8000"
prometheus.io/path: /metrics
spec:
terminationGracePeriodSeconds: 75
nodeSelector:
nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3
tolerations:
- { key: nvidia.com/gpu, operator: Exists, effect: NoSchedule }
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels: { app: vllm-qwen38-27b }
containers:
- name: vllm
image: vllm/vllm-openai:v0.32.0
args:
- --model=Qwen/Qwen3.8-27B-FP8
- --revision=<pinned-commit-sha>
- --served-model-name=qwen38-27b
- --max-model-len=16384
- --gpu-memory-utilization=0.92
- --kv-cache-dtype=fp8
- --max-num-seqs=64
- --enable-auto-tool-choice
- --tool-call-parser=hermes
- --shutdown-timeout=45
- --port=8000
env:
- { name: HF_HOME, value: /models/hf }
- { name: HF_HUB_OFFLINE, value: "1" }
- name: VLLM_API_KEY
valueFrom: { secretKeyRef: { name: vllm-api-key, key: key } }
ports:
- { name: http, containerPort: 8000 }
resources:
requests: { cpu: "8", memory: 64Gi, nvidia.com/gpu: "1" }
limits: { memory: 96Gi, nvidia.com/gpu: "1" }
volumeMounts:
- { name: model-cache, mountPath: /models, readOnly: true }
- { name: dshm, mountPath: /dev/shm }
startupProbe:
httpGet: { path: /health, port: http }
periodSeconds: 10
failureThreshold: 90
readinessProbe:
httpGet: { path: /health, port: http }
periodSeconds: 5
failureThreshold: 3
livenessProbe:
httpGet: { path: /health, port: http }
periodSeconds: 30
timeoutSeconds: 10
failureThreshold: 5
lifecycle:
preStop:
exec: { command: ["/bin/sleep", "15"] }
volumes:
- name: model-cache
persistentVolumeClaim: { claimName: model-cache, readOnly: true }
- name: dshm
emptyDir: { medium: Memory, sizeLimit: 16Gi }
---
apiVersion: v1
kind: Service
metadata:
name: vllm-qwen38-27b
namespace: llm
spec:
selector: { app: vllm-qwen38-27b }
ports:
- { name: http, port: 8000, targetPort: http }
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: vllm-qwen38-27b
namespace: llm
spec:
minAvailable: 1
selector:
matchLabels: { app: vllm-qwen38-27b }
kubectl -n llm run hf-download --rm -it --restart=Never \
--image=python:3.12-slim \
--overrides='{"spec":{"containers":[{"name":"hf-download","image":"python:3.12-slim","stdin":true,"tty":true,
"command":["bash","-c","pip install -q huggingface_hub && hf download Qwen/Qwen3.8-27B-FP8 --revision <pinned-commit-sha> --cache-dir /models/hf/hub"],
"volumeMounts":[{"name":"m","mountPath":"/models"}]}],
"volumes":[{"name":"m","persistentVolumeClaim":{"claimName":"model-cache"}}]}}'
/dev/shmas memory-backedemptyDiris required for tensor parallelism (NCCL) and recommended in general.- With TP > 1, request that many GPUs and add
--tensor-parallel-size; for multi-node use LeaderWorkerSet (llm-d, KServe). - Ray Serve LLM's HTTP surface is reported not to enforce
VLLM_API_KEY(Ray issue #66487, unverified here); with any stack, enforce auth at the gateway and use NetworkPolicies so only the gateway reaches pods.
Best practices
- Two tiers: dedicated autoscaled pools for hot models; dynamic loading for the long tail.
- Inference-aware routing (EPP / llm-d) for every multi-replica pool; prompts designed for prefix reuse.
- Scale on queue depth and KV usage, thresholds from benchmarks; measure cold-start time end to end.
- Models are artifacts: mirrored, pinned by digest, signed, loaded offline.
- Graceful drain everywhere: preStop +
--shutdown-timeout+ adequate grace period + PDB. - MIG or whole GPUs in production; time-slicing only on dev pools.
- Everything in Git: Helm/Kustomize, engine flags, KEDA objects, dashboards, benchmark results.
Pitfalls to design around
| Situation | What to do |
|---|---|
| Autoscaler adds pods that take 8 minutes to become ready | Pre-pull images and weights onto GPU nodes; keep a warm minimum; size maxReplicaCount with node-pool limits |
| Runtime LoRA loaded on one pod, requests fail on others | Static --lora-modules, shared resolver, or platform-managed adapters with LoRA-aware routing |
| Rolling update interrupts long streams | --shutdown-timeout with matching grace period; maxUnavailable with PDB |
| Two vLLM engines on one time-sliced GPU OOM | Explicit --gpu-memory-utilization per engine, or MIG |
| Sleep/LoRA admin endpoints reachable from clients | Block at gateway; NetworkPolicy; drive from a controller |
| Pods download from the internet at start | Mirror models; HF_HUB_OFFLINE=1; PVC or OCI image volumes |
Definition of done
- Engine and serving stack chosen and documented against the decision tables
- Every model and adapter mirrored, pinned by digest/revision, loaded without internet access
- Inference gateway with EPP (or llm-d scheduler) routing to each pool; auth enforced at the gateway
- KEDA scaling on queue depth and KV usage with thresholds from the benchmark; cold-start time measured
- Long-tail tier (llama-swap / router / Ollama) with TTLs and preloads defined
- Probes, preStop,
--shutdown-timeout, grace period, PDB, topology spread in every model Deployment - GPU sharing policy: whole GPU or MIG in production; time-slicing limited to dev
- Dashboards and alerts for TTFT/TPOT p95, queue depth, KV usage, GPU health
- Runbooks for rollout, rollback, drain, and GPU failure tested once
Exercises
- Gateway. On a kind or k3s cluster with one GPU node, deploy the manifest above with a small model, add an
InferencePoolwith EPP, and compare TTFT p95 under a shared-prefix workload against a plain Service. - Autoscale. Apply the
ScaledObject, drive load withvllm bench serve --max-concurrency 64, and record time from first queued request to new pod Ready. Identify which phase dominates. - Swap. Run llama-swap with the config above on one GPU. Measure first-request latency for a swapped-out model and the effect of
preload. - Sleep mode. Start two vLLM engines on one GPU with
--enable-sleep-modeand--gpu-memory-utilization 0.45; put one to sleep at level 1, wake it, and time the switch against a cold start. - Drain. Start a 2,000-token streaming generation and
kubectl delete podmid-stream with and without--shutdown-timeout. Observe the client.
Sources
- vLLM — Releases, Sleep mode, Sleep mode blog, LoRA adapters, LoRA resolver plugins, Metrics, Engine arguments
- vLLM production-stack — Repository, Helm chart, 2026 roadmap
- SGLang — Releases; TensorRT-LLM — Releases, PyPI
- Hugging Face — TGI maintenance mode
- Gateway API Inference Extension — InferencePool, Releases, CNCF deep dive
- llm-d — Releases, KV-cache wins you can see, v0.5 release post, Graceful drain PR #2601, Red Hat: intelligent inference scheduling
- KServe — Releases, OCI storage / modelcar, Configurations, KServe + KEDA for vLLM (Red Hat)
- NVIDIA Dynamo — Releases, Release notes
- Ray — Serving LLMs, Issue #66487 (API key enforcement)
- KEDA autoscaling examples — ScaleOps: vLLM on Kubernetes
- Ollama — FAQ; llama.cpp — Model management / router mode; llama-swap — Repository, config.example.yaml
- LiteLLM — Routing
- Kubernetes — KEP-4639 OCI VolumeSource, Kubernetes v1.36 release announcement (OCI volume source GA)
- ModelPack — modelpack.org, modctl, CNCF: the weight of AI models
- Docker — Docker Model Runner
- NVIDIA GPU Operator — GPU sharing / time-slicing
- Zhong et al. — DistServe