Ugrás a fő tartalomhoz

Inference Engineering Fundamentals

Basic Advanced

Service: On-Premise LLM Deployment, Sovereign & Private AI, AI Productification · Typical shape: the performance half of every private-AI build: sizing, benchmarking, and tuning before go-live

Inference engineering is the discipline of turning GPUs into tokens at a target latency and cost. Every decision in a private-AI deployment — which GPU, how many, which quantization, which engine flags — follows from a few physical facts: decoding is memory-bandwidth-bound, prefill is compute-bound, and the KV cache is the resource that limits concurrency. This page builds those facts up from basic to advanced, then turns them into a benchmarking method and a tuning table.

Prerequisite: Local AI Basics (memory sizing, quantization formats).


What we deliver​

DeliverableContent
Performance modelSpreadsheet: expected TTFT, TPOT, and max concurrency per replica, derived from bandwidth, FLOPs, and KV cache math
SLO definitionAgreed targets, e.g. p95 TTFT ≤ 1.5 s, p95 TPOT ≤ 50 ms at the customer's input/output length mix
Benchmark reportLoad sweep (concurrency or request rate) with TTFT/TPOT/throughput percentiles and goodput, reproducible from a script in the repo
Tuned engine configvLLM/SGLang flags with the reason for each, committed as code

1. The two phases of transformer inference Basic​

PrefillDecode
WorkAll prompt tokens at once (big matrix × matrix)One token per sequence per step (matrix × vector, batched)
BottleneckCompute (FLOPs)Memory bandwidth (read all weights + KV each step)
User-visible metricTime to first token (TTFT)Time per output token (TPOT) / inter-token latency (ITL)
Cost ~2 × params × prompt_tokens FLOPsbytes(weights) + bytes(KV read) per step

Example: an 8B model prefilling a 4,096-token prompt needs 2 × 8e9 × 4096 ≈ 65 TFLOP. On a GPU with ~1 PFLOPS dense BF16 (H100-class) at 50 % utilisation that is ≈ 0.13 s. Ten times the prompt means ten times the TTFT — long RAG contexts are a prefill problem.


2. Compute-bound vs bandwidth-bound: the roofline Basic​

A kernel is compute-bound when its arithmetic intensity (FLOPs per byte moved) exceeds the hardware's ridge point:

ridge_point = peak_FLOPs / memory_bandwidth
H100 SXM: ~1e15 FLOP/s / 3.35e12 B/s ≈ 300 FLOP/byte

During decode with batch size B in BF16, each weight (2 bytes) is used for 2 × B FLOPs → intensity ≈ B FLOP/byte. Until B reaches the hundreds, decode is bandwidth-bound: the GPU waits on memory, and adding more sequences to the batch is almost free. That is why batching is the single biggest throughput lever (section 5).

Decode speed estimate​

tokens/s per sequence (batch 1) ≈ effective_bandwidth / bytes_read_per_token
bytes_read_per_token ≈ active_params × bytes_per_param + KV_cache_bytes_for_context
effective_bandwidth ≈ 0.6–0.8 × spec bandwidth
SetupBytes per tokenSpec BWCeilingRealistic
Llama 3.1 8B Q4_K_M on RTX 5090~4.9 GB1,792 GB/s365 tok/s220–290 tok/s
Llama 3.3 70B FP8 on H200~70 GB4,800 GB/s68 tok/s45–55 tok/s
Llama 3.3 70B Q8_0 on DGX Spark~75 GB273 GB/s3.6 tok/s~3 tok/s
gpt-oss-120b (MoE, ~5.1 B active, MXFP4) on DGX Spark~3 GB273 GB/s~90 tok/s~60 tok/s (secondary report, unverified)
Qwen3-30B-A3B Q4_K_M (~3.3 B active) on Strix Halo~2 GB256 GB/s~125 tok/s70–90 tok/s

Two consequences you will use in every sizing call:

  • Quantization speeds up decode almost linearly with bytes saved (FP8 ≈ 2× BF16) — until you become compute- or overhead-bound at large batch.
  • MoE decouples memory from speed. Memory follows total parameters; speed follows active parameters.

3. KV cache math Basic​

KV_bytes_per_token = 2 × n_layers × n_kv_heads × head_dim × bytes_per_element
KV_bytes_total = KV_bytes_per_token × Σ(context length of every live sequence)
max_concurrency ≈ (GPU_mem × util − weights − activations) / (KV_bytes_per_token × avg_context)
ModelAttentionKV / token (BF16)32k context
Llama 3.1 8BGQA, 8 KV heads × 128128 KiB4 GiB
Qwen3-32BGQA, 8 KV heads × 128, 64 layers256 KiB8 GiB
Llama 3.3 70BGQA, 8 KV heads × 128, 80 layers320 KiB10 GiB
Same 70B with MHA (64 KV heads) — hypotheticalMHA2.5 MiB80 GiB
gpt-oss-120bGQA 8 × 64; half the layers sliding-window (128 tokens)~36 KiB for full layers + bounded sliding part~1.1 GiB
DeepSeek-V2/V3 familyMLA: caches a 512-dim latent + 64-dim RoPE key per layer(512 + 64) × layers × 2 Bfar below GQA of similar size

What shrinks the KV cache:

  • GQA (grouped-query attention): many query heads share few KV heads. 8 KV heads instead of 64 → 8× smaller.
  • MLA (multi-head latent attention, DeepSeek): K and V are compressed into one low-rank latent per token; DeepSeek-V2 reports a 93.3 % KV reduction vs its 67B dense predecessor.
  • Sliding-window / hybrid attention: some layers only cache the last W tokens.
  • KV quantization: FP8 KV (--kv-cache-dtype fp8 in vLLM, q8_0 in llama.cpp/Ollama) halves it.
  • Offloading: tiered KV cache to CPU RAM or SSD (vLLM --kv-offloading-size, LMCache, llm-d tiered cache) trades bandwidth for capacity.

4. Metrics that matter Basic​

MetricDefinitionDrives
TTFTRequest sent → first token received (includes queueing + prefill)Perceived responsiveness; RAG with long prompts
TPOT(E2E − TTFT) / (output_tokens − 1), per requestStreaming smoothness; 30–50 ms ≈ faster than reading speed
ITLGap between consecutive streamed tokens (distribution over all gaps)Stutter; spikes reveal prefill interference
E2E latencyRequest → last tokenAgents and batch pipelines (non-streaming)
ThroughputOutput tokens/s (and requests/s) across all usersCost per million tokens
GoodputRequests/s that meet every SLO (e.g. TTFT ≤ 1 s and TPOT ≤ 50 ms)The number to optimise — throughput that violates SLOs is waste
Percentilesp50 / p90 / p95 / p99 of each latencySLOs are written on p95/p99, never on means

Throughput rises with load while latency degrades slowly — until the KV cache fills, requests queue, and TTFT climbs steeply. The benchmark's job is to find the knee.


5. Serving techniques Advanced​

Batching: static vs continuous​

  • Static batching waits for a batch, runs it until the longest sequence finishes. Short requests wait for long ones; the GPU idles on padding.
  • Continuous (in-flight) batching schedules at every decode step: finished sequences leave, new ones join immediately. All modern engines (vLLM, SGLang, TensorRT-LLM, llama-server) do this. It is why one GPU can serve dozens of users at almost the single-user TPOT.

PagedAttention​

The KV cache is stored in fixed-size blocks (pages) mapped through a block table, like virtual memory. No up-front reservation of max_len per request, near-zero fragmentation, and blocks can be shared between sequences (parallel sampling, common prefixes). This is vLLM's founding idea and the reason it reached 2–4× throughput over earlier systems.

Prefix caching​

Requests that share a prefix (system prompt, tool definitions, RAG preamble, previous chat turns) reuse the already-computed KV blocks; only the new suffix is prefilled. vLLM hashes blocks (automatic prefix caching, on by default in V1); SGLang uses a radix tree (RadixAttention). Design prompts static-first, dynamic-last to maximise hits, and route requests with the same prefix to the same replica (see Serving at Scale).

Chunked prefill​

A 30k-token prefill would stall every running decode for hundreds of milliseconds (ITL spike). Chunked prefill splits it into chunks of max_num_batched_tokens and mixes them with decode steps. vLLM V1 enables it by default and schedules pending decodes first. Smaller token budgets (e.g. 2048) favour ITL; larger (> 8192) favour TTFT and throughput.

Speculative decoding​

A cheap proposer drafts k tokens; the target model verifies them in one forward pass (which costs about the same as one decode step, because decode is bandwidth-bound). Accepted tokens are free speed. Output distribution is unchanged.

Method (vLLM --speculative-config)ProposerGain profileWhen to use
draft_modelSmall model from the same familyMedium–high at low QPSA good small sibling exists
eagle3Light head trained on the target's hidden states (EAGLE-3, up to ~6.5× in the paper)High at low QPS, medium–high at high QPSHead published for your model (check model hub)
mtpModel's own multi-token-prediction layers (DeepSeek, Qwen-Next, GLM, etc.)HighModel ships MTP weights — free win
ngram / suffixCopies continuations from prompt or past outputsLow–medium, no extra modelRAG, summarisation, code edits (output repeats input)
vllm serve Qwen/Qwen3.8-27B -tp 2 \
--speculative-config '{"method": "mtp", "num_speculative_tokens": 2}'

vllm serve <model> \
--speculative-config '{"method": "ngram", "num_speculative_tokens": 4, "prompt_lookup_min": 2, "prompt_lookup_max": 5}'

Gains shrink as batch size grows (the GPU becomes compute-bound and verification is no longer free). Always measure acceptance rate and TPOT at your real concurrency.

Quantization effects on speed​

QuantizationDecode (bandwidth-bound)Prefill (compute-bound)Notes
Weight-only INT4 (AWQ/GPTQ, W4A16)~2–3× faster vs BF16 at low batchNo gain or slower (dequant overhead)Good for single-user latency; fades at high batch
FP8 W8A8~2×~2× on FP8 tensor cores (Hopper+)Best all-round for H100/H200/L40S
NVFP4 / MXFP4 W4A4~3–4×Up to 2× over FP8 on Blackwell FP4 tensor coresNeeds Blackwell (MXFP4 also MI355X)
FP8 KV cacheFaster at long context (fewer KV bytes)NeutralDoubles concurrency at a given memory

Parallelism​

StrategySplitsCommunicationUse when
Tensor (TP)Each layer's matrices across GPUsAll-reduce every layer → needs NVLinkModel does not fit one GPU; also lowers TPOT. Keep within a node
Pipeline (PP)Layers across GPUs/nodesActivations between stagesTP exhausted, or multi-node
Data (DP)Full replicasNone (router balances)Model fits; you need more throughput. Simplest and most robust
Expert (EP)MoE experts across GPUsAll-to-all token dispatchLarge MoE (DeepSeek, Kimi, Qwen MoE); vLLM --enable-expert-parallel with DP attention
Context (CP)Long sequence across GPUsRing/all-gather of KVVery long prompts (≥ 128k)

Prefer DP over TP whenever the model fits on one GPU: replicas fail independently and scale linearly.

Disaggregated prefill/decode​

Prefill and decode interfere (long prefills stall decodes) and want different parallelism. Disaggregation runs them on separate GPU pools and transfers the KV cache between them (NIXL over RDMA). DistServe reports up to 7.4× more requests served within TTFT/TPOT SLOs (or a 12.6× tighter SLO) than colocated systems. Supported by vLLM (KV connectors), SGLang (PD instances that can switch role without restart in v0.5.21), llm-d, and NVIDIA Dynamo. Worth it for large models, long prompts, strict TPOT SLOs, and multi-node deployments; for a single-node 8B–70B deployment, chunked prefill is simpler and usually sufficient.

Attention kernels​

  • FlashAttention fuses attention into tiled kernels that never materialise the N×N matrix. FA3 targets Hopper; FlashAttention-4 (paper March 2026) targets Blackwell, up to 1.3× over cuDNN 9.13 and 2.7× over Triton on B200 in BF16, written in CuTe-DSL.
  • FlashInfer is a kernel library for serving (paged KV, ragged batches, MLA, sampling); vLLM and SGLang use it as a backend.
  • FlashMLA for DeepSeek-style MLA decoding.
  • Engines pick a backend automatically; override with vLLM's attention-backend setting only when benchmarking proves a gain.

6. Benchmarking methodology Advanced​

Tools (as of 2026-10)​

ToolUseNotes
vllm bench serveFast load tests against any OpenAI-compatible servervllm bench sweep serve resets caches between runs
GuideLLM (vLLM project)SLO-oriented sweeps: synchronous → throughput → constant ratesguidellm run --profile kind=sweep
AIPerf (NVIDIA, ai-dynamo/aiperf)Successor to GenAI-Perf; multi-process load generator, trace replay, goodput, GPU telemetrypip install aiperf; GenAI-Perf is being phased out

Method​

  1. Define the workload from real data: input/output token length distribution, share of shared prefix, streaming or not. A RAG assistant (4k in / 300 out) and a chat bot (300 in / 600 out) produce opposite conclusions.
  2. Fix the environment: pinned engine image and model revision, same GPU clocks, no other tenants, warm-up run discarded.
  3. Sweep load: concurrency 1, 2, 4, … or request rate until SLO violation. Record TTFT/TPOT/ITL p50/p95/p99, output tok/s, requests/s, error rate.
  4. Control prefix caching: random prompts can still collide; vary --seed or restart between runs — or deliberately model your real prefix-sharing ratio.
  5. Compute goodput at the SLO and pick the operating point (max concurrency per replica) for autoscaling thresholds.
  6. Commit the benchmark script, raw JSON results, and engine flags to the repo.
vLLM benchmark: concurrency-limited sweep
for c in 1 4 16 32 64; do
vllm bench serve \
--base-url http://vllm:8000 --model Qwen/Qwen3.8-27B-FP8 \
--dataset-name random --random-input-len 4000 --random-output-len 300 \
--num-prompts $((c * 20)) --request-rate inf --max-concurrency "$c" \
--percentile-metrics ttft,tpot,itl,e2el --metric-percentiles 50,95,99 \
--goodput ttft:1500 tpot:50 \
--save-result --result-filename "bench_c${c}.json"
done
GuideLLM sweep
pip install "guidellm[recommended]"
guidellm run \
--backend kind=openai_http,target=http://vllm:8000 \
--profile kind=sweep \
--constraint kind=max_duration,seconds=60 \
--data kind=synthetic_text,prompt_tokens=4000,output_tokens=300

Reading results​

ObservationDiagnosisAction
TTFT p99 ≫ p50 at moderate loadQueueing or long prefills blockingLower max-num-batched-tokens, add replicas, prefix-aware routing
TPOT rises steadily with concurrencyNormal: bigger batches; check it stays under SLOCap max-num-seqs at the SLO knee
ITL spikes (p99) with flat medianPrefill interferenceSmaller chunk budget; consider PD disaggregation
Throughput plateaus, KV usage ~100 %, preemptionsKV-boundFP8 KV, raise gpu-memory-utilization, lower max-model-len, TP=2
Throughput plateaus, GPU util ~100 %, KV usage lowCompute-boundFP8/FP4 weights, more GPUs, speculative decoding won't help
Results improve on rerunPrefix cache hitsReset caches or vary seed

Production metrics to watch (vLLM Prometheus): vllm:num_requests_running, vllm:num_requests_waiting, vllm:kv_cache_usage_perc, vllm:time_to_first_token_seconds, vllm:inter_token_latency_seconds, vllm:request_time_per_output_token_seconds, vllm:e2e_request_latency_seconds, vllm:prefix_cache_hits / vllm:prefix_cache_queries.


7. vLLM tuning knobs Advanced​

FlagDefault (as of 2026-10)EffectTune when
--max-model-lenModel's max contextUpper bound per request; caps KV per sequenceAlways set to what the use case needs — big memory saving
--gpu-memory-utilization0.92Fraction of VRAM vLLM may use (weights + KV)Raise to 0.95 on dedicated GPUs; lower when sharing
--max-num-seqsHardware-dependentMax concurrent sequences per stepCap at the SLO knee from your benchmark
--max-num-batched-tokensHardware-dependentToken budget per step (chunked-prefill size)~2048 for ITL; > 8192 for throughput/TTFT
--kv-cache-dtypeauto (model dtype)fp8 halves KV memoryLong context or high concurrency
--enable-prefix-cachingOn in V1Reuse KV for shared prefixesLeave on; disable only for benchmarking pure compute
--tensor-parallel-size1Shard across GPUs in a nodeModel does not fit or TPOT too high
--pipeline-parallel-size1Shard layers across GPUs/nodesMulti-node
--data-parallel-size1Replicas inside one server (MoE: with EP)Large MoE with DP attention
--enable-expert-parallelFalseEP instead of TP for MoE layersMoE on ≥ 2 GPUs
--speculative-confignoneSpeculative decodingLow-to-medium concurrency latency targets
--quantizationfrom model configForce a quant methodRarely needed; prefer pre-quantized checkpoints
--kv-offloading-sizedisabledCPU KV tier in GiBLong multi-turn sessions with re-use
--enforce-eagerFalseDisable CUDA graphsDebugging only — slower decode
--max-loras, --max-lora-rank1, 16Multi-LoRA capacityMulti-adapter serving
--shutdown-timeout0 (abort immediately)Drain in-flight requests on SIGTERMAlways in Kubernetes
A tuned single-GPU starting point (H100 80 GB, 32B FP8, RAG workload)
vllm serve Qwen/Qwen3.8-27B-FP8 \
--max-model-len 16384 \
--gpu-memory-utilization 0.92 \
--kv-cache-dtype fp8 \
--max-num-seqs 64 \
--max-num-batched-tokens 8192 \
--enable-auto-tool-choice --tool-call-parser hermes \
--shutdown-timeout 45

Best practices​

  • Model the system before benchmarking — if measured decode is far below bandwidth / bytes_per_token, look for CPU offload, wrong kernels, or eager mode.
  • Optimise goodput at the SLO, not peak throughput.
  • Prefer FP8 weights + FP8 KV on Hopper/Ada; evaluate NVFP4 on Blackwell.
  • Use DP replicas before TP when the model fits.
  • Turn on speculative decoding when the model ships MTP or an EAGLE-3 head and the workload is latency-sensitive at moderate concurrency.
  • Keep prompts cache-friendly: static system prompt and tool definitions first.
  • Benchmark with the customer's length distribution, and commit the script.

Pitfalls to design around​

SituationWhat to do
Benchmark uses 128-token prompts, production uses 6k RAG promptsDerive lengths from logs or the eval set
Comparing engines with different defaults (context, prefix cache, quant)Align settings and document them in the report
Reporting mean latencyReport p50/p95/p99 and goodput
Speculative decoding enabled everywhereMeasure acceptance rate at real batch sizes; disable where TPOT worsens
KV usage at 100 % with preemptionsLower max-num-seqs, FP8 KV, shorter max-model-len, or scale out

Definition of done​

  • Workload profile (input/output lengths, prefix share, concurrency) written down from real data
  • SLOs agreed: TTFT and TPOT percentiles
  • Back-of-envelope performance model matches measured numbers within ~30 %
  • Load sweep with p50/p95/p99, throughput, and goodput committed with raw results
  • Engine flags committed with rationale; max-model-len and max-num-seqs set explicitly
  • Max concurrency per replica documented and used as the autoscaling target
  • Prometheus dashboards for queue depth, KV usage, TTFT, TPOT live

Exercises​

  1. Bandwidth math. Predict batch-1 decode tok/s for Qwen3.8-27B at FP8 on an H100 SXM and at Q4_K_M on an M5 Max (614 GB/s). Measure one of them and compute your efficiency factor.
  2. KV budget. For Llama 3.3 70B FP8 on 2× H100 (TP=2, utilisation 0.92), how many concurrent 8k-token sequences fit with BF16 KV? With FP8 KV?
  3. Find the knee. Run the vllm bench serve loop above for an 8B model. Plot TTFT p95 and throughput against concurrency; mark the goodput maximum for TTFT ≤ 1 s, TPOT ≤ 40 ms.
  4. Chunk budget. Repeat exercise 3 with --max-num-batched-tokens 2048 and 16384 while mixing 8k-token and 200-token prompts. Compare ITL p99.
  5. Speculation. Enable ngram speculation and benchmark a summarisation workload and a creative-writing workload. Explain the difference in acceptance rate.

Sources​