Inference Engineering Fundamentals
Service: On-Premise LLM Deployment, Sovereign & Private AI, AI Productification · Typical shape: the performance half of every private-AI build: sizing, benchmarking, and tuning before go-live
Inference engineering is the discipline of turning GPUs into tokens at a target latency and cost. Every decision in a private-AI deployment — which GPU, how many, which quantization, which engine flags — follows from a few physical facts: decoding is memory-bandwidth-bound, prefill is compute-bound, and the KV cache is the resource that limits concurrency. This page builds those facts up from basic to advanced, then turns them into a benchmarking method and a tuning table.
Prerequisite: Local AI Basics (memory sizing, quantization formats).
What we deliver
| Deliverable | Content |
|---|---|
| Performance model | Spreadsheet: expected TTFT, TPOT, and max concurrency per replica, derived from bandwidth, FLOPs, and KV cache math |
| SLO definition | Agreed targets, e.g. p95 TTFT ≤ 1.5 s, p95 TPOT ≤ 50 ms at the customer's input/output length mix |
| Benchmark report | Load sweep (concurrency or request rate) with TTFT/TPOT/throughput percentiles and goodput, reproducible from a script in the repo |
| Tuned engine config | vLLM/SGLang flags with the reason for each, committed as code |
1. The two phases of transformer inference Basic
| Prefill | Decode | |
|---|---|---|
| Work | All prompt tokens at once (big matrix × matrix) | One token per sequence per step (matrix × vector, batched) |
| Bottleneck | Compute (FLOPs) | Memory bandwidth (read all weights + KV each step) |
| User-visible metric | Time to first token (TTFT) | Time per output token (TPOT) / inter-token latency (ITL) |
| Cost ~ | 2 × params × prompt_tokens FLOPs | bytes(weights) + bytes(KV read) per step |
Example: an 8B model prefilling a 4,096-token prompt needs 2 × 8e9 × 4096 ≈ 65 TFLOP. On a GPU with ~1 PFLOPS dense BF16 (H100-class) at 50 % utilisation that is ≈ 0.13 s. Ten times the prompt means ten times the TTFT — long RAG contexts are a prefill problem.
2. Compute-bound vs bandwidth-bound: the roofline Basic
A kernel is compute-bound when its arithmetic intensity (FLOPs per byte moved) exceeds the hardware's ridge point:
ridge_point = peak_FLOPs / memory_bandwidth
H100 SXM: ~1e15 FLOP/s / 3.35e12 B/s ≈ 300 FLOP/byte
During decode with batch size B in BF16, each weight (2 bytes) is used for 2 × B FLOPs → intensity ≈ B FLOP/byte. Until B reaches the hundreds, decode is bandwidth-bound: the GPU waits on memory, and adding more sequences to the batch is almost free. That is why batching is the single biggest throughput lever (section 5).
Decode speed estimate
tokens/s per sequence (batch 1) ≈ effective_bandwidth / bytes_read_per_token
bytes_read_per_token ≈ active_params × bytes_per_param + KV_cache_bytes_for_context
effective_bandwidth ≈ 0.6–0.8 × spec bandwidth
| Setup | Bytes per token | Spec BW | Ceiling | Realistic |
|---|---|---|---|---|
| Llama 3.1 8B Q4_K_M on RTX 5090 | ~4.9 GB | 1,792 GB/s | 365 tok/s | 220–290 tok/s |
| Llama 3.3 70B FP8 on H200 | ~70 GB | 4,800 GB/s | 68 tok/s | 45–55 tok/s |
| Llama 3.3 70B Q8_0 on DGX Spark | ~75 GB | 273 GB/s | 3.6 tok/s | ~3 tok/s |
| gpt-oss-120b (MoE, ~5.1 B active, MXFP4) on DGX Spark | ~3 GB | 273 GB/s | ~90 tok/s | ~60 tok/s (secondary report, unverified) |
| Qwen3-30B-A3B Q4_K_M (~3.3 B active) on Strix Halo | ~2 GB | 256 GB/s | ~125 tok/s | 70–90 tok/s |
Two consequences you will use in every sizing call:
- Quantization speeds up decode almost linearly with bytes saved (FP8 ≈ 2× BF16) — until you become compute- or overhead-bound at large batch.
- MoE decouples memory from speed. Memory follows total parameters; speed follows active parameters.
3. KV cache math Basic
KV_bytes_per_token = 2 × n_layers × n_kv_heads × head_dim × bytes_per_element
KV_bytes_total = KV_bytes_per_token × Σ(context length of every live sequence)
max_concurrency ≈ (GPU_mem × util − weights − activations) / (KV_bytes_per_token × avg_context)
| Model | Attention | KV / token (BF16) | 32k context |
|---|---|---|---|
| Llama 3.1 8B | GQA, 8 KV heads × 128 | 128 KiB | 4 GiB |
| Qwen3-32B | GQA, 8 KV heads × 128, 64 layers | 256 KiB | 8 GiB |
| Llama 3.3 70B | GQA, 8 KV heads × 128, 80 layers | 320 KiB | 10 GiB |
| Same 70B with MHA (64 KV heads) — hypothetical | MHA | 2.5 MiB | 80 GiB |
| gpt-oss-120b | GQA 8 × 64; half the layers sliding-window (128 tokens) | ~36 KiB for full layers + bounded sliding part | ~1.1 GiB |
| DeepSeek-V2/V3 family | MLA: caches a 512-dim latent + 64-dim RoPE key per layer | (512 + 64) × layers × 2 B | far below GQA of similar size |
What shrinks the KV cache:
- GQA (grouped-query attention): many query heads share few KV heads. 8 KV heads instead of 64 → 8× smaller.
- MLA (multi-head latent attention, DeepSeek): K and V are compressed into one low-rank latent per token; DeepSeek-V2 reports a 93.3 % KV reduction vs its 67B dense predecessor.
- Sliding-window / hybrid attention: some layers only cache the last W tokens.
- KV quantization: FP8 KV (
--kv-cache-dtype fp8in vLLM,q8_0in llama.cpp/Ollama) halves it. - Offloading: tiered KV cache to CPU RAM or SSD (vLLM
--kv-offloading-size, LMCache, llm-d tiered cache) trades bandwidth for capacity.
4. Metrics that matter Basic
| Metric | Definition | Drives |
|---|---|---|
| TTFT | Request sent → first token received (includes queueing + prefill) | Perceived responsiveness; RAG with long prompts |
| TPOT | (E2E − TTFT) / (output_tokens − 1), per request | Streaming smoothness; 30–50 ms ≈ faster than reading speed |
| ITL | Gap between consecutive streamed tokens (distribution over all gaps) | Stutter; spikes reveal prefill interference |
| E2E latency | Request → last token | Agents and batch pipelines (non-streaming) |
| Throughput | Output tokens/s (and requests/s) across all users | Cost per million tokens |
| Goodput | Requests/s that meet every SLO (e.g. TTFT ≤ 1 s and TPOT ≤ 50 ms) | The number to optimise — throughput that violates SLOs is waste |
| Percentiles | p50 / p90 / p95 / p99 of each latency | SLOs are written on p95/p99, never on means |
Throughput rises with load while latency degrades slowly — until the KV cache fills, requests queue, and TTFT climbs steeply. The benchmark's job is to find the knee.
5. Serving techniques Advanced
Batching: static vs continuous
- Static batching waits for a batch, runs it until the longest sequence finishes. Short requests wait for long ones; the GPU idles on padding.
- Continuous (in-flight) batching schedules at every decode step: finished sequences leave, new ones join immediately. All modern engines (vLLM, SGLang, TensorRT-LLM, llama-server) do this. It is why one GPU can serve dozens of users at almost the single-user TPOT.
PagedAttention
The KV cache is stored in fixed-size blocks (pages) mapped through a block table, like virtual memory. No up-front reservation of max_len per request, near-zero fragmentation, and blocks can be shared between sequences (parallel sampling, common prefixes). This is vLLM's founding idea and the reason it reached 2–4× throughput over earlier systems.
Prefix caching
Requests that share a prefix (system prompt, tool definitions, RAG preamble, previous chat turns) reuse the already-computed KV blocks; only the new suffix is prefilled. vLLM hashes blocks (automatic prefix caching, on by default in V1); SGLang uses a radix tree (RadixAttention). Design prompts static-first, dynamic-last to maximise hits, and route requests with the same prefix to the same replica (see Serving at Scale).
Chunked prefill
A 30k-token prefill would stall every running decode for hundreds of milliseconds (ITL spike). Chunked prefill splits it into chunks of max_num_batched_tokens and mixes them with decode steps. vLLM V1 enables it by default and schedules pending decodes first. Smaller token budgets (e.g. 2048) favour ITL; larger (> 8192) favour TTFT and throughput.
Speculative decoding
A cheap proposer drafts k tokens; the target model verifies them in one forward pass (which costs about the same as one decode step, because decode is bandwidth-bound). Accepted tokens are free speed. Output distribution is unchanged.
Method (vLLM --speculative-config) | Proposer | Gain profile | When to use |
|---|---|---|---|
draft_model | Small model from the same family | Medium–high at low QPS | A good small sibling exists |
eagle3 | Light head trained on the target's hidden states (EAGLE-3, up to ~6.5× in the paper) | High at low QPS, medium–high at high QPS | Head published for your model (check model hub) |
mtp | Model's own multi-token-prediction layers (DeepSeek, Qwen-Next, GLM, etc.) | High | Model ships MTP weights — free win |
ngram / suffix | Copies continuations from prompt or past outputs | Low–medium, no extra model | RAG, summarisation, code edits (output repeats input) |
vllm serve Qwen/Qwen3.8-27B -tp 2 \
--speculative-config '{"method": "mtp", "num_speculative_tokens": 2}'
vllm serve <model> \
--speculative-config '{"method": "ngram", "num_speculative_tokens": 4, "prompt_lookup_min": 2, "prompt_lookup_max": 5}'
Gains shrink as batch size grows (the GPU becomes compute-bound and verification is no longer free). Always measure acceptance rate and TPOT at your real concurrency.
Quantization effects on speed
| Quantization | Decode (bandwidth-bound) | Prefill (compute-bound) | Notes |
|---|---|---|---|
| Weight-only INT4 (AWQ/GPTQ, W4A16) | ~2–3× faster vs BF16 at low batch | No gain or slower (dequant overhead) | Good for single-user latency; fades at high batch |
| FP8 W8A8 | ~2× | ~2× on FP8 tensor cores (Hopper+) | Best all-round for H100/H200/L40S |
| NVFP4 / MXFP4 W4A4 | ~3–4× | Up to 2× over FP8 on Blackwell FP4 tensor cores | Needs Blackwell (MXFP4 also MI355X) |
| FP8 KV cache | Faster at long context (fewer KV bytes) | Neutral | Doubles concurrency at a given memory |
Parallelism
| Strategy | Splits | Communication | Use when |
|---|---|---|---|
| Tensor (TP) | Each layer's matrices across GPUs | All-reduce every layer → needs NVLink | Model does not fit one GPU; also lowers TPOT. Keep within a node |
| Pipeline (PP) | Layers across GPUs/nodes | Activations between stages | TP exhausted, or multi-node |
| Data (DP) | Full replicas | None (router balances) | Model fits; you need more throughput. Simplest and most robust |
| Expert (EP) | MoE experts across GPUs | All-to-all token dispatch | Large MoE (DeepSeek, Kimi, Qwen MoE); vLLM --enable-expert-parallel with DP attention |
| Context (CP) | Long sequence across GPUs | Ring/all-gather of KV | Very long prompts (≥ 128k) |
Prefer DP over TP whenever the model fits on one GPU: replicas fail independently and scale linearly.
Disaggregated prefill/decode
Prefill and decode interfere (long prefills stall decodes) and want different parallelism. Disaggregation runs them on separate GPU pools and transfers the KV cache between them (NIXL over RDMA). DistServe reports up to 7.4× more requests served within TTFT/TPOT SLOs (or a 12.6× tighter SLO) than colocated systems. Supported by vLLM (KV connectors), SGLang (PD instances that can switch role without restart in v0.5.21), llm-d, and NVIDIA Dynamo. Worth it for large models, long prompts, strict TPOT SLOs, and multi-node deployments; for a single-node 8B–70B deployment, chunked prefill is simpler and usually sufficient.
Attention kernels
- FlashAttention fuses attention into tiled kernels that never materialise the N×N matrix. FA3 targets Hopper; FlashAttention-4 (paper March 2026) targets Blackwell, up to 1.3× over cuDNN 9.13 and 2.7× over Triton on B200 in BF16, written in CuTe-DSL.
- FlashInfer is a kernel library for serving (paged KV, ragged batches, MLA, sampling); vLLM and SGLang use it as a backend.
- FlashMLA for DeepSeek-style MLA decoding.
- Engines pick a backend automatically; override with vLLM's attention-backend setting only when benchmarking proves a gain.
6. Benchmarking methodology Advanced
Tools (as of 2026-10)
| Tool | Use | Notes |
|---|---|---|
vllm bench serve | Fast load tests against any OpenAI-compatible server | vllm bench sweep serve resets caches between runs |
| GuideLLM (vLLM project) | SLO-oriented sweeps: synchronous → throughput → constant rates | guidellm run --profile kind=sweep |
AIPerf (NVIDIA, ai-dynamo/aiperf) | Successor to GenAI-Perf; multi-process load generator, trace replay, goodput, GPU telemetry | pip install aiperf; GenAI-Perf is being phased out |
Method
- Define the workload from real data: input/output token length distribution, share of shared prefix, streaming or not. A RAG assistant (4k in / 300 out) and a chat bot (300 in / 600 out) produce opposite conclusions.
- Fix the environment: pinned engine image and model revision, same GPU clocks, no other tenants, warm-up run discarded.
- Sweep load: concurrency 1, 2, 4, … or request rate until SLO violation. Record TTFT/TPOT/ITL p50/p95/p99, output tok/s, requests/s, error rate.
- Control prefix caching: random prompts can still collide; vary
--seedor restart between runs — or deliberately model your real prefix-sharing ratio. - Compute goodput at the SLO and pick the operating point (max concurrency per replica) for autoscaling thresholds.
- Commit the benchmark script, raw JSON results, and engine flags to the repo.
for c in 1 4 16 32 64; do
vllm bench serve \
--base-url http://vllm:8000 --model Qwen/Qwen3.8-27B-FP8 \
--dataset-name random --random-input-len 4000 --random-output-len 300 \
--num-prompts $((c * 20)) --request-rate inf --max-concurrency "$c" \
--percentile-metrics ttft,tpot,itl,e2el --metric-percentiles 50,95,99 \
--goodput ttft:1500 tpot:50 \
--save-result --result-filename "bench_c${c}.json"
done
pip install "guidellm[recommended]"
guidellm run \
--backend kind=openai_http,target=http://vllm:8000 \
--profile kind=sweep \
--constraint kind=max_duration,seconds=60 \
--data kind=synthetic_text,prompt_tokens=4000,output_tokens=300
Reading results
| Observation | Diagnosis | Action |
|---|---|---|
| TTFT p99 ≫ p50 at moderate load | Queueing or long prefills blocking | Lower max-num-batched-tokens, add replicas, prefix-aware routing |
| TPOT rises steadily with concurrency | Normal: bigger batches; check it stays under SLO | Cap max-num-seqs at the SLO knee |
| ITL spikes (p99) with flat median | Prefill interference | Smaller chunk budget; consider PD disaggregation |
| Throughput plateaus, KV usage ~100 %, preemptions | KV-bound | FP8 KV, raise gpu-memory-utilization, lower max-model-len, TP=2 |
| Throughput plateaus, GPU util ~100 %, KV usage low | Compute-bound | FP8/FP4 weights, more GPUs, speculative decoding won't help |
| Results improve on rerun | Prefix cache hits | Reset caches or vary seed |
Production metrics to watch (vLLM Prometheus): vllm:num_requests_running, vllm:num_requests_waiting, vllm:kv_cache_usage_perc, vllm:time_to_first_token_seconds, vllm:inter_token_latency_seconds, vllm:request_time_per_output_token_seconds, vllm:e2e_request_latency_seconds, vllm:prefix_cache_hits / vllm:prefix_cache_queries.
7. vLLM tuning knobs Advanced
| Flag | Default (as of 2026-10) | Effect | Tune when |
|---|---|---|---|
--max-model-len | Model's max context | Upper bound per request; caps KV per sequence | Always set to what the use case needs — big memory saving |
--gpu-memory-utilization | 0.92 | Fraction of VRAM vLLM may use (weights + KV) | Raise to 0.95 on dedicated GPUs; lower when sharing |
--max-num-seqs | Hardware-dependent | Max concurrent sequences per step | Cap at the SLO knee from your benchmark |
--max-num-batched-tokens | Hardware-dependent | Token budget per step (chunked-prefill size) | ~2048 for ITL; > 8192 for throughput/TTFT |
--kv-cache-dtype | auto (model dtype) | fp8 halves KV memory | Long context or high concurrency |
--enable-prefix-caching | On in V1 | Reuse KV for shared prefixes | Leave on; disable only for benchmarking pure compute |
--tensor-parallel-size | 1 | Shard across GPUs in a node | Model does not fit or TPOT too high |
--pipeline-parallel-size | 1 | Shard layers across GPUs/nodes | Multi-node |
--data-parallel-size | 1 | Replicas inside one server (MoE: with EP) | Large MoE with DP attention |
--enable-expert-parallel | False | EP instead of TP for MoE layers | MoE on ≥ 2 GPUs |
--speculative-config | none | Speculative decoding | Low-to-medium concurrency latency targets |
--quantization | from model config | Force a quant method | Rarely needed; prefer pre-quantized checkpoints |
--kv-offloading-size | disabled | CPU KV tier in GiB | Long multi-turn sessions with re-use |
--enforce-eager | False | Disable CUDA graphs | Debugging only — slower decode |
--max-loras, --max-lora-rank | 1, 16 | Multi-LoRA capacity | Multi-adapter serving |
--shutdown-timeout | 0 (abort immediately) | Drain in-flight requests on SIGTERM | Always in Kubernetes |
vllm serve Qwen/Qwen3.8-27B-FP8 \
--max-model-len 16384 \
--gpu-memory-utilization 0.92 \
--kv-cache-dtype fp8 \
--max-num-seqs 64 \
--max-num-batched-tokens 8192 \
--enable-auto-tool-choice --tool-call-parser hermes \
--shutdown-timeout 45
Best practices
- Model the system before benchmarking — if measured decode is far below
bandwidth / bytes_per_token, look for CPU offload, wrong kernels, or eager mode. - Optimise goodput at the SLO, not peak throughput.
- Prefer FP8 weights + FP8 KV on Hopper/Ada; evaluate NVFP4 on Blackwell.
- Use DP replicas before TP when the model fits.
- Turn on speculative decoding when the model ships MTP or an EAGLE-3 head and the workload is latency-sensitive at moderate concurrency.
- Keep prompts cache-friendly: static system prompt and tool definitions first.
- Benchmark with the customer's length distribution, and commit the script.
Pitfalls to design around
| Situation | What to do |
|---|---|
| Benchmark uses 128-token prompts, production uses 6k RAG prompts | Derive lengths from logs or the eval set |
| Comparing engines with different defaults (context, prefix cache, quant) | Align settings and document them in the report |
| Reporting mean latency | Report p50/p95/p99 and goodput |
| Speculative decoding enabled everywhere | Measure acceptance rate at real batch sizes; disable where TPOT worsens |
| KV usage at 100 % with preemptions | Lower max-num-seqs, FP8 KV, shorter max-model-len, or scale out |
Definition of done
- Workload profile (input/output lengths, prefix share, concurrency) written down from real data
- SLOs agreed: TTFT and TPOT percentiles
- Back-of-envelope performance model matches measured numbers within ~30 %
- Load sweep with p50/p95/p99, throughput, and goodput committed with raw results
- Engine flags committed with rationale;
max-model-lenandmax-num-seqsset explicitly - Max concurrency per replica documented and used as the autoscaling target
- Prometheus dashboards for queue depth, KV usage, TTFT, TPOT live
Exercises
- Bandwidth math. Predict batch-1 decode tok/s for Qwen3.8-27B at FP8 on an H100 SXM and at Q4_K_M on an M5 Max (614 GB/s). Measure one of them and compute your efficiency factor.
- KV budget. For Llama 3.3 70B FP8 on 2× H100 (TP=2, utilisation 0.92), how many concurrent 8k-token sequences fit with BF16 KV? With FP8 KV?
- Find the knee. Run the
vllm bench serveloop above for an 8B model. Plot TTFT p95 and throughput against concurrency; mark the goodput maximum for TTFT ≤ 1 s, TPOT ≤ 40 ms. - Chunk budget. Repeat exercise 3 with
--max-num-batched-tokens 2048and16384while mixing 8k-token and 200-token prompts. Compare ITL p99. - Speculation. Enable
ngramspeculation and benchmark a summarisation workload and a creative-writing workload. Explain the difference in acceptance rate.
Sources
- Kwon et al. — Efficient Memory Management for LLM Serving with PagedAttention
- Zhong et al. — DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving
- Li et al. — EAGLE-3
- DeepSeek-AI — DeepSeek-V2 (MLA)
- Zadouri, Dao et al. — FlashAttention-4; PyTorch — Low-precision FlashAttention-4
- vLLM docs — Optimization and tuning, Engine arguments, Speculative decoding, Benchmark CLI, Metrics, Releases
- vLLM blog — Speculators v0.3.0; Red Hat — EAGLE-3 with vLLM
- SGLang — Releases
- GuideLLM — Repository; NVIDIA — Benchmarking LLM inference at scale with AIPerf
- NVIDIA — Introducing NVFP4, HGX H100/H200/B200 components
- Model configs — Qwen3.8-27B, gpt-oss-120b, Qwen3-30B-A3B
- Hardware bandwidth — DGX Spark specs, Apple Mac Studio M5, Strix Halo guide