Local AI Basics — Hardware, Quantization, and Runtimes
Service: Sovereign & Private AI, On-Premise LLM Deployment, GenAI Proof of Concept (PoC) · Typical shape: the foundation every private-AI engagement builds on; read it before a sizing workshop or a PoC kickoff
"Local AI" means running open-weight models on hardware the customer controls — a laptop, a workstation under a desk, a GPU server in their rack, or a GPU VM on a sovereign cloud such as STACKIT. Prompts, documents, and outputs never leave that boundary. For our customers this is what makes GDPR, NIS2, and data-residency requirements straightforward, and it turns per-token pricing into a predictable hardware or reservation cost.
This page gives you the three skills you need in every sizing conversation: estimate memory, pick a quantization, and pick a runtime. Then it walks you through serving a model and calling it like any OpenAI endpoint. The next page, Inference Engineering, explains why the numbers behave as they do; Serving at Scale takes it to Kubernetes.
What we deliver
| Deliverable | Content |
|---|---|
| Sizing sheet | Model × quantization × context × concurrency → memory per GPU, expected decode tokens/s, hardware recommendation |
| Runtime choice | Documented decision (Ollama / llama.cpp / vLLM / SGLang / MLX) with the reason it fits the workload |
| Running endpoint | An OpenAI-compatible endpoint with pinned model revision, explicit context length, and a smoke test |
| Quality check | The chosen quantization compared to a higher-precision baseline on the customer's own eval set |
Reference architecture
Every local runtime we use exposes an OpenAI-compatible HTTP API. Applications are written once against that API and the backend becomes a deployment decision.
1. Hardware options
Decode speed (tokens per second for one user) is limited by memory bandwidth, capacity decides what fits, and compute decides prompt processing (prefill) speed. Keep those three numbers in front of you when comparing boxes.
| Class | Examples (as of 2026-10) | Memory | Bandwidth | Strengths | Watch for |
|---|---|---|---|---|---|
| NVIDIA consumer | RTX 5090 | 32 GB GDDR7 | 1,792 GB/s | Highest bandwidth per euro; full CUDA stack (vLLM, SGLang, llama.cpp) | 32 GB caps you at ~32B Q4 dense; no NVLink, no ECC |
| NVIDIA workstation | RTX PRO 6000 Blackwell | 96 GB GDDR7 ECC | 1,792 GB/s (Server Edition 1,597 GB/s) | Same die as 5090 with 3× memory: 70B FP8 or ~120B MoE on one card | Price; PCIe only |
| NVIDIA datacenter | H100 SXM / H200 / B200 | 80 / 141 / 192 GB HBM | 3.35 / 4.8 / 8 TB/s | NVLink, MIG, FP8 (Hopper) and FP4 (Blackwell) tensor cores; multi-tenant serving | Power, cooling, lead times — usually rented (STACKIT, AWS, OCI) |
| Apple Silicon | Mac Studio M5 Max / M5 Ultra | up to 128 / 512 GB unified | up to 614 GB/s / 1.2 TB/s | Huge models in one box, quiet, low power; MLX and llama.cpp Metal | Weaker prefill than NVIDIA; not a multi-user server |
| AMD unified | Ryzen AI Max+ 395 "Strix Halo" | up to 128 GB LPDDR5X (~96 GB+ usable as VRAM) | ~256 GB/s | Cheap way to fit 70B–120B MoE models | ROCm on gfx1151 depends on the ROCm release (check before committing); Vulkan backend in llama.cpp is the common path |
| NVIDIA unified | DGX Spark (GB10) | 128 GB LPDDR5x | 273 GB/s | CUDA + FP4 on the desk; good prefill; ConnectX-7 to pair two units | Bandwidth-bound for dense models: ~3 tok/s for 70B Q8 |
| AMD datacenter / discrete | Instinct MI300X/MI355X, Radeon | 192–288 GB HBM | 5+ TB/s | vLLM and SGLang support ROCm; MI355X has native MXFP4 | Check kernel support per model before committing |
| CPU-only | Any x86/ARM server | System RAM | 50–400 GB/s | Embeddings, small models (≤ 4B), batch jobs | Interactive chat on > 8B dense is slow |
Rule of thumb for a sizing call: dense models want bandwidth, MoE models want capacity. A 120B MoE with ~5B active parameters runs well on a 128 GB unified-memory box; a 70B dense model on the same box does not feel interactive.
2. Memory sizing
The formula
total_memory ≈ weights + kv_cache + overhead
weights = parameters × bytes_per_parameter
kv_cache = 2 × layers × kv_heads × head_dim × bytes_per_element × context_tokens × concurrent_sequences
overhead ≈ 0.5–2 GB (CUDA context, activations, runtime buffers); vLLM reserves a fraction of VRAM up front
bytes_per_parameter: BF16/FP16 = 2, FP8/INT8 ≈ 1, Q8_0 ≈ 1.06, Q4_K_M ≈ 0.61, MXFP4 ≈ 0.53, NVFP4 ≈ 0.56. The GGUF figures come from the llama.cpp quantize table (Q4_K_M = 4.89 bits per weight).
The factor 2 in the KV formula is "one K and one V". Read num_hidden_layers, num_key_value_heads, and head_dim from the model's config.json on Hugging Face.
Worked examples
| Model | Architecture (from config.json) | KV per token (FP16) | Weights | KV at stated context | Total | Fits on |
|---|---|---|---|---|---|---|
| Llama 3.1 8B, Q4_K_M | 32 layers, 8 KV heads, dim 128 | 2·32·8·128·2 = 128 KiB | 8 B × 0.61 ≈ 4.9 GB | 8k → 1 GiB | ~6.5 GB | 8 GB GPU, any 16 GB Mac |
| Llama 3.1 8B, BF16 | same | 128 KiB | 16 GB | 32k → 4 GiB | ~21 GB | 24 GB GPU |
| Qwen3-32B, Q4_K_M | 64 layers, 8 KV heads, dim 128 | 256 KiB | 32.8 B × 0.61 ≈ 20 GB | 32k → 8 GiB | ~29 GB | RTX 5090 (tight); use q8_0 KV to drop to ~25 GB |
| Llama 3.3 70B, FP8 | 80 layers, 8 KV heads, dim 128 | 320 KiB | 70 GB | 32k × 4 seqs → 40 GiB | ~112 GB | 2× H100 80 GB (TP=2), 1× H200 141 GB |
| Llama 3.3 70B, Q4_K_M | same | 320 KiB | ≈ 43 GB | 16k → 5 GiB | ~49 GB | 64 GB Mac, Strix Halo, DGX Spark, RTX PRO 6000 |
| Qwen3-30B-A3B (MoE), Q4_K_M | 48 layers, 4 KV heads, dim 128; 128 experts, 8 active (~3.3 B active) | 96 KiB | 30.5 B × 0.61 ≈ 19 GB | 32k → 3 GiB | ~23 GB | 24 GB GPU, 32 GB Mac |
Two lessons hide in that table:
- KV cache scales with concurrency. The 70B example needs as much memory for four 32k conversations as for half its weights. Ask "how many simultaneous users, how long are their prompts" before you quote hardware.
- MoE memory is set by total parameters, speed by active parameters. Qwen3-30B-A3B needs memory for 30.5 B weights but reads only ~3.3 B per token, so it decodes roughly as fast as a 3B dense model. See Inference Engineering for the bandwidth math.
def vram_gb(params_b, bits_per_weight, layers, kv_heads, head_dim, ctx, seqs=1, kv_bytes=2, overhead_gb=1.0):
weights = params_b * 1e9 * bits_per_weight / 8
kv = 2 * layers * kv_heads * head_dim * kv_bytes * ctx * seqs
return round((weights + kv) / 1e9 + overhead_gb, 1)
print(vram_gb(32.8, 4.89, 64, 8, 128, ctx=32_768)) # Qwen3-32B Q4_K_M → ~29.6
print(vram_gb(70.6, 8, 80, 8, 128, ctx=32_768, seqs=4)) # Llama 3.3 70B FP8 → ~114
3. Quantization formats
Quantization stores weights (and sometimes activations and KV cache) in fewer bits. It saves memory and increases decode speed, because decode is bandwidth-bound.
| Format | Where it runs | Bits / weight | What it is | Use when |
|---|---|---|---|---|
GGUF k-quants (Q8_0, Q6_K, Q5_K_M, Q4_K_M, Q3_K_M) | llama.cpp, Ollama, LM Studio | 8.5 / 6.6 / 5.7 / 4.9 / 4.0 | Block-wise integer quant with per-block scales; _M/_S/_L mix precisions per tensor | Laptops, Macs, CPU, single-user GPU boxes. Q4_K_M is the default sweet spot |
GGUF i-quants (IQ4_XS, IQ3_XXS, IQ2_XXS) | llama.cpp | 4.5 / 3.3 / 2.4 | Lattice/codebook quant, designed to be used with an importance matrix (imatrix) | You must fit a big model into small memory; always use an imatrix build |
| AWQ (W4A16) | vLLM, SGLang | ~4.25 | Activation-aware: protects salient weight channels | 4-bit on GPUs that lack FP8/FP4 (Ampere, consumer) |
| GPTQ (W4A16, W8A16) | vLLM, SGLang, Transformers | ~4.25 | Second-order error-compensating quant, calibrated on sample data | Same niche as AWQ; huge choice of prebuilt checkpoints |
| FP8 (W8A8, E4M3) | vLLM, SGLang, TensorRT-LLM on Hopper/Ada/Blackwell, MI300 | 8 | Weights and activations in 8-bit float; near-lossless for most models | Default for production on H100/H200/L40S. Halves memory vs BF16 |
| MXFP4 | Blackwell, MI355X; vLLM, llama.cpp | 4.25 | OCP microscaling: E2M1 values, power-of-two (E8M0) scale per 32 values | Models shipped in it (gpt-oss); AMD MI355X |
| NVFP4 | Blackwell (B200, RTX 50, RTX PRO, DGX Spark) | ~4.5 | E2M1 values, FP8 (E4M3) scale per 16 values plus per-tensor FP32 scale | Max throughput on Blackwell; NVIDIA reports ≤ 1 % accuracy loss vs FP8 on DeepSeek-R1 |
Tooling (as of 2026-10): AutoAWQ and AutoGPTQ are archived. Use llm-compressor (vLLM project; AWQ, GPTQ, FP8, NVFP4 into the compressed-tensors format) or GPTQModel for GPTQ/AWQ, and llama-quantize for GGUF.
Quality trade-off, in practice:
- 8-bit (FP8,
Q8_0): treat as lossless for business tasks. - 5–6 bit: indistinguishable from 8-bit in almost all evals.
- 4-bit (
Q4_K_M, AWQ, NVFP4): small, measurable drops — most visible in long reasoning chains, code, math, and non-English text. Hungarian and German are where we check first. - 3-bit and below: only for very large models (a 2–3-bit 120B often beats a 4-bit 30B), and only with imatrix quants.
- Prefer a bigger model at 4-bit over a smaller model at 8-bit when memory is fixed, then confirm with an eval.
- Always evaluate the exact quantized artifact on the customer's eval set. Perplexity and KL-divergence against the BF16 model are useful smoke tests; task evals decide.
# from a llama.cpp build; model.gguf is the F16/BF16 conversion
./llama-imatrix -m model-f16.gguf -f calibration.txt -o imatrix.gguf
./llama-quantize --imatrix imatrix.gguf model-f16.gguf model-Q4_K_M.gguf Q4_K_M 8
4. Runtimes compared
| Runtime | What it is | Best for | Multi-user throughput | Formats | Notes (as of 2026-10) |
|---|---|---|---|---|---|
| Ollama | Daemon + model library on top of llama.cpp (and MLX on Apple Silicon since v0.40) | Developer laptops, demos, small teams | Low–medium (OLLAMA_NUM_PARALLEL, default 1) | GGUF, MLX | Latest stable v0.40.6 (2026-10-16). Model swapping built in (keep_alive) |
| llama.cpp / llama-server | C/C++ engine with OpenAI-compatible server | Maximum control on any hardware: CUDA, Metal, ROCm, Vulkan, CPU | Medium (--parallel slots, continuous batching) | GGUF | Router mode loads/unloads many models in one server |
| LM Studio | Desktop app + llmster headless daemon, lms CLI | Workstation users, Mac users wanting MLX with a GUI | Low–medium | GGUF, MLX | Serves on localhost:1234 |
| vLLM | High-throughput GPU server: PagedAttention, continuous batching, prefix caching | Default for production GPU serving | High | Safetensors (BF16, FP8, AWQ, GPTQ, NVFP4, MXFP4), limited GGUF | Latest v0.32.0 (2026-10-18) |
| SGLang | High-throughput GPU server with RadixAttention prefix cache | Agentic / multi-turn / shared-prefix workloads, large MoE, PD disaggregation | High | Same as vLLM | Latest v0.5.21 (GitHub, 2026-10-02); v0.5.22 on PyPI (2026-10-16) |
| MLX / mlx-lm | Apple's array framework and LLM package | Apple Silicon, fine-tuning LoRA on a Mac | Low–medium | MLX (4/8-bit) | mlx_lm.server is OpenAI-compatible |
| TensorRT-LLM | NVIDIA's optimized engine, PyTorch backend, trtllm-serve | Squeezing NVIDIA datacenter GPUs at scale | Very high | HF checkpoints, FP8/NVFP4 | Advanced; see Serving at Scale |
| TGI | Hugging Face Text Generation Inference | — | — | — | Maintenance mode since Dec 2025, repo archived Mar 2026. Migrate to vLLM or SGLang |
| Docker Model Runner | docker model CLI; models as OCI artifacts; llama.cpp or vLLM backend | Teams standardised on Docker; distributing models through a registry | Low–high (depends on backend) | GGUF, Safetensors | vLLM backend Linux + NVIDIA only |
| LocalAI | OpenAI-compatible "all-in-one" with 60+ backends (LLM, TTS, ASR, images) | One API for many modalities on modest hardware | Medium | Many | Auth deny-by-default since v4.9 |
Decision in one line: laptop or Mac → Ollama (or LM Studio for GUI users); one box, full control, any hardware → llama-server; shared GPU server or anything with more than a handful of concurrent users → vLLM (SGLang when prefix reuse dominates).
Implementation steps
Step 1 — Install Ollama and pull a small model
# Linux (macOS/Windows: installer from ollama.com)
curl -fsSL https://ollama.com/install.sh | sh
ollama --version
ollama pull qwen3.5:4b # 3.3–4.0 GB, text + image
ollama run qwen3.5:4b "Explain KV cache in two sentences."
ollama ps # check PROCESSOR (GPU/CPU split) and CONTEXT columns
Step 2 — Call the OpenAI-compatible API
curl -s http://localhost:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "qwen3.5:4b",
"messages": [{"role": "user", "content": "Give three uses of local LLMs in a hospital."}],
"temperature": 0.2
}' | jq -r '.choices[0].message.content'
# pip install openai
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama") # key required by client, ignored by server
stream = client.chat.completions.create(
model="qwen3.5:4b",
messages=[{"role": "user", "content": "Summarise GDPR Article 28 in five bullets."}],
stream=True,
)
for chunk in stream:
print(chunk.choices[0].delta.content or "", end="", flush=True)
Swap base_url to http://gpu-host:8000/v1 and the same code talks to vLLM.
Step 3 — Set the context length explicitly
Ollama's default context is chosen from VRAM: 4k below 23 GiB, 32k for 23–47 GiB, 256k at 47 GiB and above (as of 2026-10, per server/routes.go in the Ollama source; older releases used 2048). Prompts longer than the window are truncated from the start without an error — a RAG prompt or an agent's system prompt silently loses its head. Set it explicitly:
# Server-wide default (applies to /v1 requests too)
OLLAMA_CONTEXT_LENGTH=32768 ollama serve
# Or bake it into a model variant (the way to control context on the OpenAI endpoint)
cat > Modelfile <<'EOF'
FROM qwen3.5:4b
PARAMETER num_ctx 32768
EOF
ollama create qwen3-4b-32k -f Modelfile
- The native
/api/chataccepts"options": {"num_ctx": 32768}per request; the OpenAI-compatible/v1endpoint has no field for it — use the server variable or a Modelfile. OLLAMA_NUM_CTXis not a server variable; it does nothing.- Verify with
ollama ps(CONTEXT column). A bigger context costs KV-cache memory: use the formula above. - Turn on KV-cache quantization when memory is tight:
OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0halves KV memory.
Step 4 — Run llama-server directly
# Prebuilt binaries or: cmake -B build -DGGML_CUDA=ON && cmake --build build -j
llama-server \
-hf ggml-org/gpt-oss-20b-GGUF \
--ctx-size 32768 \
--n-gpu-layers 999 \
--parallel 4 \
--flash-attn on \
--jinja \
--host 0.0.0.0 --port 8080
--ctx-sizeis the total across slots when KV is not unified: with--parallel 4each slot gets 8k (--kv-unifiedchanges this). Size it asper_request_ctx × parallel.--n-gpu-layers 999offloads everything; lower it to split between GPU and CPU when the model does not fit.--jinjauses the chat template embedded in the GGUF — required for correct tool calling.GET /healthfor probes,GET /metricswith--metrics.
curl -s localhost:8080/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"messages":[{"role":"user","content":"Hello"}]}' | jq -r '.choices[0].message.content'
Step 5 — Chat templates and tool calling
A chat model was trained on a specific chat template (special tokens around system/user/assistant/tool turns). The runtime renders your messages array through that template. A wrong template is the most common cause of "the model rambles", "it never stops", or "tool calls come back as plain text".
- Use templates shipped with the model:
tokenizer_config.json/chat_template.jinjafor vLLM, embedded GGUF metadata for llama.cpp (--jinja), the ModelfileTEMPLATEin Ollama. - vLLM needs a tool parser for tool calls:
vllm serve <model> --enable-auto-tool-choice --tool-call-parser hermes(the parser name depends on the model family; check the model card). - Reasoning models emit a thinking section; servers separate it with a reasoning parser (
--reasoning-parserin vLLM) so clients get cleancontent.
curl -s http://localhost:11434/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "qwen3.5:4b",
"messages": [{"role": "user", "content": "What is the temperature in Budapest?"}],
"tools": [{
"type": "function",
"function": {
"name": "get_temperature",
"description": "Current temperature for a city",
"parameters": {"type": "object", "required": ["city"],
"properties": {"city": {"type": "string"}}}
}
}]
}' | jq '.choices[0].message.tool_calls'
Your code executes the function and sends the result back as a {"role": "tool", "tool_call_id": ..., "content": ...} message in a second request. Small models (≤ 4B) call tools less reliably; validate arguments against the schema and use structured outputs (response_format with a JSON schema) where the runtime supports it.
Best practices
- Pin everything: runtime version (container tag), model repo and revision/digest, quantization file name. "latest" is not a version.
- Set context length on purpose in every runtime: Ollama
OLLAMA_CONTEXT_LENGTH, llama.cpp--ctx-size, vLLM--max-model-len. - Size for concurrency, not for one user: KV cache × concurrent sequences often exceeds weights.
- Prefer FP8 on Hopper/Ada and NVFP4/FP8 on Blackwell for GPU serving; GGUF
Q4_K_M–Q6_Kfor laptops and Macs. - Evaluate the quantized artifact on the customer's tasks and languages before signing off a hardware quote.
- Keep apps runtime-agnostic through the OpenAI API, so the PoC on Ollama moves to vLLM unchanged.
- Bind to localhost by default (
OLLAMA_HOST=127.0.0.1); expose through a gateway with auth and TLS, never directly.
Pitfalls to design around
| Symptom | Cause | What to do |
|---|---|---|
| Answers ignore the start of a long document | Default context too small, silent truncation | Set context explicitly, log prompt token counts |
| 10× slower than expected | Model partly on CPU | Check ollama ps PROCESSOR column / llama.cpp load log; pick a smaller quant or shorter context |
| Tool calls appear as text | Wrong or missing chat template / tool parser | --jinja (llama.cpp), --tool-call-parser (vLLM), official model tags |
| vLLM OOM at start | --max-model-len defaults to the model's maximum (e.g. 128k+) | Set --max-model-len to what the use case needs |
| Good English, weak Hungarian | Aggressive quant hurts low-resource languages first | Step up one quant level; eval in the target language |
Definition of done
- Sizing sheet filled with weights, KV cache at target context × concurrency, overhead
- Runtime and quantization choice documented with the reason
- Model revision and runtime version pinned
- Context length set explicitly and verified (
ollama ps, server logs) - OpenAI-compatible endpoint answers curl and Python smoke tests, including one tool call
- Quantized model compared with a higher-precision baseline on the customer eval set
- Endpoint bound to localhost or behind an authenticated gateway
Exercises
- Size it. Using the formula, compute memory for Qwen3-32B at
Q5_K_M(5.70 bpw) with 16k context and 4 parallel slots. Does it fit an RTX 5090? What changes withq8_0KV cache? - Find the truncation. Run Ollama with
OLLAMA_CONTEXT_LENGTH=2048, send a 5,000-token document with a question about its first paragraph, and observe the answer. Repeat with 16384. Checkollama psboth times. - Quant shoot-out. Pull the same model at
Q4_K_MandQ8_0. Run ten domain questions (include Hungarian or German) against both; compare answers andeval_count / eval_durationtokens/s from/api/chatresponses. - Swap the backend. Point
hello_local.pyat llama-server on port 8080 without changing anything butbase_urlandmodel. - Tool loop. Extend the tool-calling curl into a Python loop that executes
get_temperature, returns the result, and prints the final answer.
Sources
- Ollama — Context length, FAQ (keep_alive, parallelism, KV cache type), OpenAI compatibility, Tool calling, qwen3 library page, Releases, server/routes.go (VRAM-based default context), envconfig (OLLAMA_* variables)
- llama.cpp — Quantize README (bits per weight, imatrix), Model management / router mode
- vLLM — Releases, PyPI, AutoAWQ deprecation note, llm-compressor
- SGLang — Releases
- GPTQModel — Repository; Transformers — AutoAWQ deprecation issue
- NVIDIA — Introducing NVFP4; Modal — Block quants (MXFP4/NVFP4)
- Hugging Face TGI — Maintenance mode notice, Repository (archived)
- Docker — Docker Model Runner, DMR + vLLM
- LocalAI — Repository, Releases
- LM Studio — Headless / llmster
- TensorRT-LLM — Releases
- Model configs — Qwen3-30B-A3B, Qwen3-32B, gpt-oss-120b
- Hardware — DGX Spark (PNY spec sheet), Apple Mac Studio M5 Max/Ultra, Mac Studio tech specs, RTX 5090 vs RTX PRO 6000, H100/H200/B200 (NVIDIA HGX reference architecture), H200 specs, Strix Halo for local LLMs