Ugrás a fő tartalomhoz

Serving LLMs at Scale on Kubernetes

Advanced

Service: On-Premise LLM Deployment, Sovereign AI on STACKIT, AI Productification · Typical shape: production build after a successful PoC; platform team of 2–4 engineers, IaC and GitOps from day one

A single vllm serve on one GPU is a PoC. Production means: several models and LoRA adapters behind one API, routing that understands KV caches, autoscaling on the signals that actually predict latency, models that load in seconds rather than minutes, GPUs shared where it is safe, and rollouts that never drop a streaming response. This page is the playbook for that platform on Kubernetes — on the customer's own GPU nodes, on OpenStack, or on STACKIT SKE with GPU node pools.

Prerequisites: Local AI Basics, Inference Engineering. Platform standards: How We Build.


What we deliver​

DeliverableContent
Serving platformKubernetes stack (vLLM + llm-d or KServe) deployed by GitOps, with GPU operator, gateway, monitoring
Model catalogueEach model/adapter as a versioned artifact (OCI or PVC cache) with pinned engine flags and benchmark results
Routing layerOpenAI-compatible gateway with auth, quotas, and inference-aware (KV/prefix/LoRA-aware) routing
AutoscalingQueue- and KV-based scaling with tested scale-up time and scale-to-zero policy where appropriate
RunbooksRollout, drain, GPU failure, model rollback, capacity planning

Reference architecture​

Two tiers is the pattern we recommend: hot models (high traffic, SLO-bound) run as dedicated autoscaled vLLM pools behind an inference-aware gateway; long-tail models (occasional use, experiments) share GPUs through dynamic loading (llama-swap, Ollama, llama.cpp router, vLLM sleep mode).


1. Choose the engine​

vLLMSGLangTensorRT-LLMllama.cpp (llama-server)
Version (as of 2026-10)v0.32.0 (2026-10-18)v0.5.21 (GitHub); 0.5.22 on PyPI1.2.x stable (1.2.1 on PyPI)rolling builds
Sweet spotDefault for GPU serving; broadest model + hardware coverage (NVIDIA, AMD, TPU, Intel)Prefix-heavy agentic/multi-turn, large MoE, PD disaggregationMax performance on NVIDIA datacenter GPUsEdge, CPU, Apple, Vulkan, small deployments, many GGUF models
Prefix cacheHash-based APCRadixAttention (Rust core)YesPer-slot cache reuse
Disaggregated PDYes (KV connectors, NIXL)Yes (role-switching PD)Yes (via Dynamo)No
Multi-LoRAYes, dynamic loadYesYesYes (static)
Ecosystemllm-d, KServe, production-stack, Dynamo, Ray Servellm-d, Dynamo, Ray ServeDynamo, Tritonllama-swap, Ollama
Operational costLowLow–mediumHigher (NVIDIA-only, faster-moving APIs)Very low

Recommendation: vLLM by default. Benchmark SGLang when the workload is dominated by shared prefixes or large MoE. Consider TensorRT-LLM only on large NVIDIA fleets where the extra throughput pays for the extra operational surface. Use llama.cpp for the long-tail tier and for non-NVIDIA edge hardware. TGI is archived — migrate existing TGI deployments to vLLM.


2. Choose the Kubernetes serving stack​

StackWhat it is (as of 2026-10)Choose when
Plain Deployment + Gateway API Inference ExtensionvLLM Deployments grouped into InferencePools (API GA at inference.networking.k8s.io/v1, latest v1.6.x); an Endpoint Picker (EPP) scores pods on KV usage, queue length, active LoRAsSmall–medium platforms; you want standard Kubernetes primitives and minimum moving parts
llm-dCNCF Sandbox project (Red Hat, Google, IBM, NVIDIA, CoreWeave…); vLLM + Inference Gateway + LeaderWorkerSet with "well-lit paths": intelligent scheduling, precise prefix-cache routing, PD disaggregation, wide EP, tiered KV cache. v0.10.0 (2026-09-29)Production platforms that need KV-aware routing, PD disaggregation, or large MoE across nodes. Our default for multi-replica production
KServeCNCF model-serving platform; LLMInferenceService integrates llm-d; InferenceService + Hugging Face runtime; modelcars (OCI); LocalModelCache. v0.21 (2026-09)Customer already runs KServe / OpenShift AI, or needs one API for LLMs and classic ML models
vLLM production-stackHelm chart (vllm-stack, 0.1.13) with a router (round-robin, session, prefix-aware, KV-aware via LMCache) and Grafana dashboardsFast start for a vLLM-only cluster; smaller teams
NVIDIA DynamoDistributed inference framework (v1.5.1 stable, 2026-10-07) with KV router, PD disaggregation, SLA-driven planner, multi-engine (vLLM, SGLang, TRT-LLM)Large NVIDIA fleets, especially with TensorRT-LLM or GB200 NVL racks
Ray Serve LLMray[llm]: build_openai_app over vLLM; Python-native autoscaling, multi-model, multi-LoRATeams already on Ray (Anyscale, KubeRay); mixed Python pipelines

Decision rule: start with vLLM + Inference Gateway for one or two models; move to llm-d (directly or through KServe LLMInferenceService) when you need prefix-aware routing at scale, PD disaggregation, or multi-node MoE. Whichever you choose, keep the client contract at the OpenAI API so the stack is replaceable.


3. Inference-aware routing (Gateway API Inference Extension)​

Round-robin load balancing is wrong for LLMs: requests differ by 100× in cost, and a replica that already holds a prompt's prefix in KV cache serves it far faster. The Gateway API Inference Extension adds:

  • InferencePool — a set of model-server pods (like a Service, but for inference) plus a reference to its EPP.
  • Endpoint Picker (EPP) — an ext-proc service the gateway (Envoy-based: kgateway, Istio, GKE, agentgateway…) calls per request. It scores pods by queue depth, KV-cache utilisation, prefix-cache affinity, and loaded LoRA adapters.
  • Flow control / priorities — critical vs sheddable traffic.
inferencepool.yaml
apiVersion: inference.networking.k8s.io/v1
kind: InferencePool
metadata:
name: qwen38-27b
namespace: llm
spec:
selector:
matchLabels:
app: vllm-qwen38-27b
targetPorts:
- number: 8000
endpointPickerRef:
name: qwen38-27b-epp
port:
number: 9002
failureMode: FailOpen
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: qwen38-27b
namespace: llm
spec:
parentRefs:
- name: inference-gateway
rules:
- backendRefs:
- group: inference.networking.k8s.io
kind: InferencePool
name: qwen38-27b

llm-d's scheduler builds on the EPP with precise prefix-cache scoring: it subscribes to KV-cache events from each vLLM pod and knows which blocks live where, instead of estimating from traffic. llm-d reported large TTFT reductions and roughly doubled throughput against round-robin on shared-prefix workloads. Combine it with prompts designed static-first (see Inference Engineering).


4. Autoscaling on the right signals​

GPU utilisation is a poor scaling signal: a bandwidth-bound decode shows "100 %" at both 2 and 60 concurrent sequences. Scale on queue depth and KV-cache usage, which move before latency does.

Signal (vLLM metric)MeaningUse
vllm:num_requests_waitingRequests queued, not yet scheduledPrimary scale-up trigger
vllm:num_requests_runningRequests in the batchCombine with waiting to avoid scale-down mid-stream
vllm:kv_cache_usage_percKV cache fill (0–1)Secondary trigger; > 0.9 sustained = preemptions coming
vllm:time_to_first_token_seconds (p95)SLO itselfAlerting; lagging as a scaling signal
keda-scaledobject.yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: vllm-qwen38-27b
namespace: llm
spec:
scaleTargetRef:
name: vllm-qwen38-27b
minReplicaCount: 1
maxReplicaCount: 6
pollingInterval: 10
cooldownPeriod: 600
advanced:
horizontalPodAutoscalerConfig:
behavior:
scaleDown:
stabilizationWindowSeconds: 600
triggers:
- type: prometheus
name: queue
metadata:
serverAddress: http://prometheus.monitoring.svc:9090
query: sum(vllm:num_requests_waiting{namespace="llm",pod=~"vllm-qwen38-27b.*"}) or vector(0)
metricType: AverageValue
threshold: "4"
- type: prometheus
name: kv
metadata:
serverAddress: http://prometheus.monitoring.svc:9090
query: avg(vllm:kv_cache_usage_perc{namespace="llm",pod=~"vllm-qwen38-27b.*"}) or vector(0)
threshold: "0.85"
  • KEDA takes the maximum replica count across triggers.
  • Set the thresholds from the benchmark knee (max concurrency per replica at SLO), not from intuition.
  • Scale-up time = node provisioning + image pull + weight load + CUDA graph capture. Measure it; with a cold node and a 60 GB model it can be several minutes. Model caching (section 7) and vLLM's weight-preloading features shrink the last two.
  • Scale-to-zero needs a signal from in front of the pods (gateway queue or KServe/Knative activator), because a scaled-to-zero vLLM exports no metrics. Use it for long-tail models only.
  • llm-d ships a workload-variant autoscaler (llm-d-autoscaling) and KServe LLMInferenceService supports direct KEDA scaling; evaluate them when you adopt those stacks.

5. Dynamic model loading (long-tail tier)​

Not every model deserves a dedicated GPU. These tools load models on demand and evict idle ones.

ToolMechanismKey settings
OllamaLoads on request, unloads after idlekeep_alive per request (default 5 min; 0 = unload now, -1 = forever), OLLAMA_KEEP_ALIVE, OLLAMA_MAX_LOADED_MODELS (default 3 × GPUs), OLLAMA_NUM_PARALLEL (default 1), OLLAMA_MAX_QUEUE (default 512)
llama.cpp router modellama-server with no model: one process per model, LRU eviction, OpenAI model field selects--models-dir, --models-preset config.ini, --models-max (default 4), --no-models-autoload; POST /models/load, POST /models/unload, GET /models
llama-swapGo proxy that starts/stops any OpenAI-compatible server (llama-server, vLLM, SGLang, containers) per modelcmd, ttl, groups (swap/exclusive/persistent), aliases, hooks.on_startup.preload
vLLM sleep modeKeep the engine process alive, release GPU memory--enable-sleep-mode + VLLM_SERVER_DEV_MODE=1; POST /sleep?level=1 (weights to CPU RAM, KV dropped), level=2 (weights discarded), POST /wake_up, GET /is_sleeping. Switch models in seconds instead of a full cold start
vLLM dynamic LoRALoad adapters into a running base model--enable-lora, VLLM_ALLOW_RUNTIME_LORA_UPDATING=True; POST /v1/load_lora_adapter, POST /v1/unload_lora_adapter; or LoRA resolver plugins (filesystem, HF Hub)
KServe ModelMeshMulti-model placement for many small models (classic ML)Mostly superseded for LLMs by LLMInferenceService + LoRA; use only for small-model fleets
LiteLLMGateway routing one OpenAI API across many backends (vLLM pools, llama-swap, Ollama, cloud)model_list, router_settings.routing_strategy, retries, cooldowns, fallbacks, virtual keys, budgets
Dev endpoints are admin endpoints

vLLM's /sleep, /wake_up, /collective_rpc and runtime LoRA endpoints require dev mode or runtime-update flags. Bind them to the pod network, block them at the gateway, and drive them only from a trusted controller.

llama-swap example​

llama-swap/config.yaml
healthCheckTimeout: 300
logLevel: info
startPort: 10001

macros:
"llama": >
/app/llama-server --port ${PORT} --host 127.0.0.1
--n-gpu-layers 999 --flash-attn on --jinja --metrics

models:
"qwen3-30b-a3b":
cmd: |
${llama}
--model /models/Qwen3-30B-A3B-Q4_K_M.gguf
--ctx-size 65536 --parallel 4
aliases: ["default-chat"]
ttl: 900

"gpt-oss-20b":
cmd: |
${llama}
--model /models/gpt-oss-20b-mxfp4.gguf
--ctx-size 32768 --parallel 2
ttl: 600

"bge-m3-embed":
cmd: |
${llama}
--model /models/bge-m3-Q8_0.gguf --embeddings --ctx-size 8192
ttl: 0

"qwen38-27b-vllm":
cmd: |
docker run --rm --name ${MODEL_ID} --gpus all -p ${PORT}:8000
-v /models/hf:/root/.cache/huggingface
vllm/vllm-openai:v0.32.0 --model Qwen/Qwen3.8-27B-FP8 --max-model-len 16384
cmdStop: docker stop ${MODEL_ID}
useModelName: "Qwen/Qwen3.8-27B-FP8"
ttl: 1800

routing:
router:
use: group
settings:
groups:
"always-on":
swap: false
exclusive: false
persistent: true
members: ["bge-m3-embed"]
"chat":
swap: true
exclusive: false
members: ["qwen3-30b-a3b", "gpt-oss-20b", "qwen38-27b-vllm"]

hooks:
on_startup:
preload: ["bge-m3-embed", "qwen3-30b-a3b"]
  • The embedding model stays resident (persistent, ttl: 0); chat models swap among themselves on one GPU.
  • Requests to default-chat hit qwen3-30b-a3b through the alias.
  • The first request to a swapped-out model pays the load time — expose it in client timeouts and pre-warm with preload for the morning peak.
  • Groups sit under routing.router.settings.groups in the current llama-swap schema (docs/config.example.yaml); check the example for the version you pin.

llama.cpp router presets​

models.ini
[*]
n-gpu-layers = 999
flash-attn = on

[qwen3-30b-a3b]
model = /models/Qwen3-30B-A3B-Q4_K_M.gguf
ctx-size = 65536

[gpt-oss-20b]
model = /models/gpt-oss-20b-mxfp4.gguf
ctx-size = 32768
llama-server --models-preset models.ini --models-max 2 --host 0.0.0.0 --port 8080
curl -s localhost:8080/models | jq

LiteLLM in front of everything​

litellm/config.yaml
model_list:
- model_name: chat-large
litellm_params:
model: hosted_vllm/Qwen/Qwen3.8-27B-FP8
api_base: http://inference-gateway.llm.svc/v1
model_info: { allowed_fails: 2, cooldown_time: 30 }
- model_name: chat-small
litellm_params:
model: openai/qwen3-30b-a3b
api_base: http://llama-swap.llm.svc:8080/v1
api_key: none
- model_name: embed
litellm_params:
model: openai/bge-m3-embed
api_base: http://llama-swap.llm.svc:8080/v1
api_key: none

router_settings:
routing_strategy: simple-shuffle
num_retries: 2
fallbacks: [{ "chat-large": ["chat-small"] }]

The gateway gives one API, virtual keys per team, budgets, and fallbacks; the inference gateway below it does KV-aware pod selection. Keep the two responsibilities separate.


6. Multi-LoRA serving​

Fine-tuned adapters (see Fine-Tuning) are tens to hundreds of MB; serving them on one base model replaces N full deployments.

VLLM_ALLOW_RUNTIME_LORA_UPDATING=True \
vllm serve Qwen/Qwen3.8-27B-FP8 \
--enable-lora --max-loras 8 --max-lora-rank 64 --max-cpu-loras 32 \
--lora-modules legal=/adapters/legal-v3 support=/adapters/support-v7

curl -X POST http://vllm:8000/v1/load_lora_adapter -H 'Content-Type: application/json' \
-d '{"lora_name": "hr-v1", "lora_path": "/adapters/hr-v1"}'
# Clients select an adapter by model name: {"model": "legal", ...}
  • --max-loras = adapters active in one batch (GPU memory); --max-cpu-loras = warm cache in host RAM.
  • Runtime load endpoints affect one pod. In a replicated deployment, declare adapters statically (--lora-modules), use a resolver plugin that every pod shares, or let the platform manage them (KServe/llm-d LoRA-aware routing, AIBrix ModelAdapter). The EPP routes requests to pods that already hold the adapter.
  • Version adapters like models: immutable paths or OCI tags, evaluated before release.

7. Model caching and pre-pulling​

Downloading 60 GB from Hugging Face on every pod start makes autoscaling and recovery slow and ties production to an external service. Options, best first for sovereign deployments:

ApproachHowNotes
OCI image volumePackage weights as an OCI image/artifact in the customer's registry (Harbor); mount with volumes[].imageKubernetes OCI volume source (ImageVolume, KEP-4639) is GA in 1.36. Runtime support depends on your containerd/CRI-O version; verify it against your distro before relying on it
KServe modelcarstorageUri: oci://registry/model:tag; sidecar exposes /models from the image via /procDefault-enabled in KServe; oci+native:// mode uses image volumes
ModelPack (CNCF)Standard OCI artifact layout for models; modctl build/push; KitOps can pack ModelPackRegistry-native versioning and signing; Dragonfly P2P for large clusters
Docker Model Runner OCIdocker model package --gguf ... --push registry/model:tagFine for developer and edge distribution
Shared RWX PVCNFS/CephFS/Lustre volume with a pre-populated HF cache, mounted read-onlySimple; throughput of the filesystem bounds load time
Node-local cachePre-pull DaemonSet or KServe LocalModelCache onto NVMeFastest loads; costs disk per node

Always: mirror models into the customer's registry or storage, pin by digest, set HF_HUB_OFFLINE=1 in production pods so nothing reaches the internet, and pre-pull images and models onto GPU nodes before scaling events. Sign artifacts (cosign) and verify on admission — it closes the "who changed the weights" question auditors ask.

Image volume (Kubernetes ≥ 1.36)
volumes:
- name: model
image:
reference: harbor.example.internal/models/qwen38-27b-fp8@sha256:<digest>
pullPolicy: IfNotPresent
# container: volumeMounts: [{name: model, mountPath: /models/qwen38-27b, readOnly: true}]

8. GPU sharing​

MechanismIsolationGranularityUse for
Whole GPUFull1 GPUProduction LLM pools — default
MIG (A100/H100/H200/B200)Hardware: memory, cache, SMs, fault isolationUp to 7 slicesSmall models, embeddings, rerankers, multi-tenant dev; predictable latency
Time-slicing (GPU operator)None — shared memory, no fault isolationN replicas per GPUDev/test, notebooks, low-traffic long-tail; never mix tenants
MPSPartial (compute partitioning, shared address-space risks)Percent of SMsMany small concurrent inference processes from one tenant
Dynamic Resource Allocation (DRA)Depends on driverClaims with selectorsNewer clusters: richer GPU requests (MIG profiles, sharing) instead of extended resources
Time-slicing config for the NVIDIA GPU Operator (dev node pool only)
apiVersion: v1
kind: ConfigMap
metadata:
name: time-slicing-config
namespace: gpu-operator
data:
any: |-
version: v1
flags:
migStrategy: none
sharing:
timeSlicing:
renameByDefault: false
failRequestsGreaterThanOne: true
resources:
- name: nvidia.com/gpu
replicas: 4

vLLM pre-allocates --gpu-memory-utilization of the whole device; on a shared GPU set it explicitly (e.g. 0.45 for two engines) or use MIG.


9. Fault tolerance and graceful drain​

  • Drain: --shutdown-timeout (default 0) keeps serving running requests after SIGTERM for up to N seconds; at 0, vLLM aborts in-flight requests immediately. Grace period must exceed preStop + drain + teardown (llm-d uses 15 + 45 + 15 = 75 s).
  • Probes: startupProbe on /health with a long budget (weights + compile can take minutes); readinessProbe on /health; a conservative livenessProbe so a long prefill never gets the pod killed.
  • Rollouts: maxUnavailable: 0, maxSurge: 1 when you have spare GPUs; otherwise maxSurge: 0, maxUnavailable: 1 with a PodDisruptionBudget. Canary new model versions by weight on the HTTPRoute or LiteLLM.
  • GPU failures: DCGM health checks + GPU operator; Xid errors should cordon the node. Keep ≥ 2 replicas for every SLO-bound model, spread across nodes with topology spread constraints.
  • Clients: retry idempotent non-streaming requests at the gateway; for streams, surface the error and let the client resubmit. Set request timeouts above worst-case model load time on the long-tail tier.
  • EPP failure: failureMode: FailOpen keeps traffic flowing with plain load balancing.

10. Complete example: vLLM on Kubernetes​

vllm-qwen38-27b.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: model-cache
namespace: llm
spec:
accessModes: ["ReadWriteMany"]
storageClassName: cephfs
resources:
requests:
storage: 500Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-qwen38-27b
namespace: llm
labels: { app: vllm-qwen38-27b }
spec:
replicas: 2
strategy:
type: RollingUpdate
rollingUpdate: { maxSurge: 0, maxUnavailable: 1 }
selector:
matchLabels: { app: vllm-qwen38-27b }
template:
metadata:
labels: { app: vllm-qwen38-27b }
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8000"
prometheus.io/path: /metrics
spec:
terminationGracePeriodSeconds: 75
nodeSelector:
nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3
tolerations:
- { key: nvidia.com/gpu, operator: Exists, effect: NoSchedule }
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels: { app: vllm-qwen38-27b }
containers:
- name: vllm
image: vllm/vllm-openai:v0.32.0
args:
- --model=Qwen/Qwen3.8-27B-FP8
- --revision=<pinned-commit-sha>
- --served-model-name=qwen38-27b
- --max-model-len=16384
- --gpu-memory-utilization=0.92
- --kv-cache-dtype=fp8
- --max-num-seqs=64
- --enable-auto-tool-choice
- --tool-call-parser=hermes
- --shutdown-timeout=45
- --port=8000
env:
- { name: HF_HOME, value: /models/hf }
- { name: HF_HUB_OFFLINE, value: "1" }
- name: VLLM_API_KEY
valueFrom: { secretKeyRef: { name: vllm-api-key, key: key } }
ports:
- { name: http, containerPort: 8000 }
resources:
requests: { cpu: "8", memory: 64Gi, nvidia.com/gpu: "1" }
limits: { memory: 96Gi, nvidia.com/gpu: "1" }
volumeMounts:
- { name: model-cache, mountPath: /models, readOnly: true }
- { name: dshm, mountPath: /dev/shm }
startupProbe:
httpGet: { path: /health, port: http }
periodSeconds: 10
failureThreshold: 90
readinessProbe:
httpGet: { path: /health, port: http }
periodSeconds: 5
failureThreshold: 3
livenessProbe:
httpGet: { path: /health, port: http }
periodSeconds: 30
timeoutSeconds: 10
failureThreshold: 5
lifecycle:
preStop:
exec: { command: ["/bin/sleep", "15"] }
volumes:
- name: model-cache
persistentVolumeClaim: { claimName: model-cache, readOnly: true }
- name: dshm
emptyDir: { medium: Memory, sizeLimit: 16Gi }
---
apiVersion: v1
kind: Service
metadata:
name: vllm-qwen38-27b
namespace: llm
spec:
selector: { app: vllm-qwen38-27b }
ports:
- { name: http, port: 8000, targetPort: http }
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: vllm-qwen38-27b
namespace: llm
spec:
minAvailable: 1
selector:
matchLabels: { app: vllm-qwen38-27b }
Populate the cache once (Job or admin pod with internet/mirror access)
kubectl -n llm run hf-download --rm -it --restart=Never \
--image=python:3.12-slim \
--overrides='{"spec":{"containers":[{"name":"hf-download","image":"python:3.12-slim","stdin":true,"tty":true,
"command":["bash","-c","pip install -q huggingface_hub && hf download Qwen/Qwen3.8-27B-FP8 --revision <pinned-commit-sha> --cache-dir /models/hf/hub"],
"volumeMounts":[{"name":"m","mountPath":"/models"}]}],
"volumes":[{"name":"m","persistentVolumeClaim":{"claimName":"model-cache"}}]}}'
  • /dev/shm as memory-backed emptyDir is required for tensor parallelism (NCCL) and recommended in general.
  • With TP > 1, request that many GPUs and add --tensor-parallel-size; for multi-node use LeaderWorkerSet (llm-d, KServe).
  • Ray Serve LLM's HTTP surface is reported not to enforce VLLM_API_KEY (Ray issue #66487, unverified here); with any stack, enforce auth at the gateway and use NetworkPolicies so only the gateway reaches pods.

Best practices​

  • Two tiers: dedicated autoscaled pools for hot models; dynamic loading for the long tail.
  • Inference-aware routing (EPP / llm-d) for every multi-replica pool; prompts designed for prefix reuse.
  • Scale on queue depth and KV usage, thresholds from benchmarks; measure cold-start time end to end.
  • Models are artifacts: mirrored, pinned by digest, signed, loaded offline.
  • Graceful drain everywhere: preStop + --shutdown-timeout + adequate grace period + PDB.
  • MIG or whole GPUs in production; time-slicing only on dev pools.
  • Everything in Git: Helm/Kustomize, engine flags, KEDA objects, dashboards, benchmark results.

Pitfalls to design around​

SituationWhat to do
Autoscaler adds pods that take 8 minutes to become readyPre-pull images and weights onto GPU nodes; keep a warm minimum; size maxReplicaCount with node-pool limits
Runtime LoRA loaded on one pod, requests fail on othersStatic --lora-modules, shared resolver, or platform-managed adapters with LoRA-aware routing
Rolling update interrupts long streams--shutdown-timeout with matching grace period; maxUnavailable with PDB
Two vLLM engines on one time-sliced GPU OOMExplicit --gpu-memory-utilization per engine, or MIG
Sleep/LoRA admin endpoints reachable from clientsBlock at gateway; NetworkPolicy; drive from a controller
Pods download from the internet at startMirror models; HF_HUB_OFFLINE=1; PVC or OCI image volumes

Definition of done​

  • Engine and serving stack chosen and documented against the decision tables
  • Every model and adapter mirrored, pinned by digest/revision, loaded without internet access
  • Inference gateway with EPP (or llm-d scheduler) routing to each pool; auth enforced at the gateway
  • KEDA scaling on queue depth and KV usage with thresholds from the benchmark; cold-start time measured
  • Long-tail tier (llama-swap / router / Ollama) with TTLs and preloads defined
  • Probes, preStop, --shutdown-timeout, grace period, PDB, topology spread in every model Deployment
  • GPU sharing policy: whole GPU or MIG in production; time-slicing limited to dev
  • Dashboards and alerts for TTFT/TPOT p95, queue depth, KV usage, GPU health
  • Runbooks for rollout, rollback, drain, and GPU failure tested once

Exercises​

  1. Gateway. On a kind or k3s cluster with one GPU node, deploy the manifest above with a small model, add an InferencePool with EPP, and compare TTFT p95 under a shared-prefix workload against a plain Service.
  2. Autoscale. Apply the ScaledObject, drive load with vllm bench serve --max-concurrency 64, and record time from first queued request to new pod Ready. Identify which phase dominates.
  3. Swap. Run llama-swap with the config above on one GPU. Measure first-request latency for a swapped-out model and the effect of preload.
  4. Sleep mode. Start two vLLM engines on one GPU with --enable-sleep-mode and --gpu-memory-utilization 0.45; put one to sleep at level 1, wake it, and time the switch against a cold start.
  5. Drain. Start a 2,000-token streaming generation and kubectl delete pod mid-stream with and without --shutdown-timeout. Observe the client.

Sources​