Ugrás a fő tartalomhoz

Local AI Basics — Hardware, Quantization, and Runtimes

Basic

Service: Sovereign & Private AI, On-Premise LLM Deployment, GenAI Proof of Concept (PoC) · Typical shape: the foundation every private-AI engagement builds on; read it before a sizing workshop or a PoC kickoff

"Local AI" means running open-weight models on hardware the customer controls — a laptop, a workstation under a desk, a GPU server in their rack, or a GPU VM on a sovereign cloud such as STACKIT. Prompts, documents, and outputs never leave that boundary. For our customers this is what makes GDPR, NIS2, and data-residency requirements straightforward, and it turns per-token pricing into a predictable hardware or reservation cost.

This page gives you the three skills you need in every sizing conversation: estimate memory, pick a quantization, and pick a runtime. Then it walks you through serving a model and calling it like any OpenAI endpoint. The next page, Inference Engineering, explains why the numbers behave as they do; Serving at Scale takes it to Kubernetes.


What we deliver​

DeliverableContent
Sizing sheetModel × quantization × context × concurrency → memory per GPU, expected decode tokens/s, hardware recommendation
Runtime choiceDocumented decision (Ollama / llama.cpp / vLLM / SGLang / MLX) with the reason it fits the workload
Running endpointAn OpenAI-compatible endpoint with pinned model revision, explicit context length, and a smoke test
Quality checkThe chosen quantization compared to a higher-precision baseline on the customer's own eval set

Reference architecture​

Every local runtime we use exposes an OpenAI-compatible HTTP API. Applications are written once against that API and the backend becomes a deployment decision.


1. Hardware options​

Decode speed (tokens per second for one user) is limited by memory bandwidth, capacity decides what fits, and compute decides prompt processing (prefill) speed. Keep those three numbers in front of you when comparing boxes.

ClassExamples (as of 2026-10)MemoryBandwidthStrengthsWatch for
NVIDIA consumerRTX 509032 GB GDDR71,792 GB/sHighest bandwidth per euro; full CUDA stack (vLLM, SGLang, llama.cpp)32 GB caps you at ~32B Q4 dense; no NVLink, no ECC
NVIDIA workstationRTX PRO 6000 Blackwell96 GB GDDR7 ECC1,792 GB/s (Server Edition 1,597 GB/s)Same die as 5090 with 3× memory: 70B FP8 or ~120B MoE on one cardPrice; PCIe only
NVIDIA datacenterH100 SXM / H200 / B20080 / 141 / 192 GB HBM3.35 / 4.8 / 8 TB/sNVLink, MIG, FP8 (Hopper) and FP4 (Blackwell) tensor cores; multi-tenant servingPower, cooling, lead times — usually rented (STACKIT, AWS, OCI)
Apple SiliconMac Studio M5 Max / M5 Ultraup to 128 / 512 GB unifiedup to 614 GB/s / 1.2 TB/sHuge models in one box, quiet, low power; MLX and llama.cpp MetalWeaker prefill than NVIDIA; not a multi-user server
AMD unifiedRyzen AI Max+ 395 "Strix Halo"up to 128 GB LPDDR5X (~96 GB+ usable as VRAM)~256 GB/sCheap way to fit 70B–120B MoE modelsROCm on gfx1151 depends on the ROCm release (check before committing); Vulkan backend in llama.cpp is the common path
NVIDIA unifiedDGX Spark (GB10)128 GB LPDDR5x273 GB/sCUDA + FP4 on the desk; good prefill; ConnectX-7 to pair two unitsBandwidth-bound for dense models: ~3 tok/s for 70B Q8
AMD datacenter / discreteInstinct MI300X/MI355X, Radeon192–288 GB HBM5+ TB/svLLM and SGLang support ROCm; MI355X has native MXFP4Check kernel support per model before committing
CPU-onlyAny x86/ARM serverSystem RAM50–400 GB/sEmbeddings, small models (≤ 4B), batch jobsInteractive chat on > 8B dense is slow

Rule of thumb for a sizing call: dense models want bandwidth, MoE models want capacity. A 120B MoE with ~5B active parameters runs well on a 128 GB unified-memory box; a 70B dense model on the same box does not feel interactive.


2. Memory sizing​

The formula​

total_memory ≈ weights + kv_cache + overhead

weights = parameters × bytes_per_parameter
kv_cache = 2 × layers × kv_heads × head_dim × bytes_per_element × context_tokens × concurrent_sequences
overhead ≈ 0.5–2 GB (CUDA context, activations, runtime buffers); vLLM reserves a fraction of VRAM up front

bytes_per_parameter: BF16/FP16 = 2, FP8/INT8 ≈ 1, Q8_0 ≈ 1.06, Q4_K_M ≈ 0.61, MXFP4 ≈ 0.53, NVFP4 ≈ 0.56. The GGUF figures come from the llama.cpp quantize table (Q4_K_M = 4.89 bits per weight).

The factor 2 in the KV formula is "one K and one V". Read num_hidden_layers, num_key_value_heads, and head_dim from the model's config.json on Hugging Face.

Worked examples​

ModelArchitecture (from config.json)KV per token (FP16)WeightsKV at stated contextTotalFits on
Llama 3.1 8B, Q4_K_M32 layers, 8 KV heads, dim 1282·32·8·128·2 = 128 KiB8 B × 0.61 ≈ 4.9 GB8k → 1 GiB~6.5 GB8 GB GPU, any 16 GB Mac
Llama 3.1 8B, BF16same128 KiB16 GB32k → 4 GiB~21 GB24 GB GPU
Qwen3-32B, Q4_K_M64 layers, 8 KV heads, dim 128256 KiB32.8 B × 0.61 ≈ 20 GB32k → 8 GiB~29 GBRTX 5090 (tight); use q8_0 KV to drop to ~25 GB
Llama 3.3 70B, FP880 layers, 8 KV heads, dim 128320 KiB70 GB32k × 4 seqs → 40 GiB~112 GB2× H100 80 GB (TP=2), 1× H200 141 GB
Llama 3.3 70B, Q4_K_Msame320 KiB≈ 43 GB16k → 5 GiB~49 GB64 GB Mac, Strix Halo, DGX Spark, RTX PRO 6000
Qwen3-30B-A3B (MoE), Q4_K_M48 layers, 4 KV heads, dim 128; 128 experts, 8 active (~3.3 B active)96 KiB30.5 B × 0.61 ≈ 19 GB32k → 3 GiB~23 GB24 GB GPU, 32 GB Mac

Two lessons hide in that table:

  1. KV cache scales with concurrency. The 70B example needs as much memory for four 32k conversations as for half its weights. Ask "how many simultaneous users, how long are their prompts" before you quote hardware.
  2. MoE memory is set by total parameters, speed by active parameters. Qwen3-30B-A3B needs memory for 30.5 B weights but reads only ~3.3 B per token, so it decodes roughly as fast as a 3B dense model. See Inference Engineering for the bandwidth math.
vram_estimate.py
def vram_gb(params_b, bits_per_weight, layers, kv_heads, head_dim, ctx, seqs=1, kv_bytes=2, overhead_gb=1.0):
weights = params_b * 1e9 * bits_per_weight / 8
kv = 2 * layers * kv_heads * head_dim * kv_bytes * ctx * seqs
return round((weights + kv) / 1e9 + overhead_gb, 1)

print(vram_gb(32.8, 4.89, 64, 8, 128, ctx=32_768)) # Qwen3-32B Q4_K_M → ~29.6
print(vram_gb(70.6, 8, 80, 8, 128, ctx=32_768, seqs=4)) # Llama 3.3 70B FP8 → ~114

3. Quantization formats​

Quantization stores weights (and sometimes activations and KV cache) in fewer bits. It saves memory and increases decode speed, because decode is bandwidth-bound.

FormatWhere it runsBits / weightWhat it isUse when
GGUF k-quants (Q8_0, Q6_K, Q5_K_M, Q4_K_M, Q3_K_M)llama.cpp, Ollama, LM Studio8.5 / 6.6 / 5.7 / 4.9 / 4.0Block-wise integer quant with per-block scales; _M/_S/_L mix precisions per tensorLaptops, Macs, CPU, single-user GPU boxes. Q4_K_M is the default sweet spot
GGUF i-quants (IQ4_XS, IQ3_XXS, IQ2_XXS)llama.cpp4.5 / 3.3 / 2.4Lattice/codebook quant, designed to be used with an importance matrix (imatrix)You must fit a big model into small memory; always use an imatrix build
AWQ (W4A16)vLLM, SGLang~4.25Activation-aware: protects salient weight channels4-bit on GPUs that lack FP8/FP4 (Ampere, consumer)
GPTQ (W4A16, W8A16)vLLM, SGLang, Transformers~4.25Second-order error-compensating quant, calibrated on sample dataSame niche as AWQ; huge choice of prebuilt checkpoints
FP8 (W8A8, E4M3)vLLM, SGLang, TensorRT-LLM on Hopper/Ada/Blackwell, MI3008Weights and activations in 8-bit float; near-lossless for most modelsDefault for production on H100/H200/L40S. Halves memory vs BF16
MXFP4Blackwell, MI355X; vLLM, llama.cpp4.25OCP microscaling: E2M1 values, power-of-two (E8M0) scale per 32 valuesModels shipped in it (gpt-oss); AMD MI355X
NVFP4Blackwell (B200, RTX 50, RTX PRO, DGX Spark)~4.5E2M1 values, FP8 (E4M3) scale per 16 values plus per-tensor FP32 scaleMax throughput on Blackwell; NVIDIA reports ≤ 1 % accuracy loss vs FP8 on DeepSeek-R1

Tooling (as of 2026-10): AutoAWQ and AutoGPTQ are archived. Use llm-compressor (vLLM project; AWQ, GPTQ, FP8, NVFP4 into the compressed-tensors format) or GPTQModel for GPTQ/AWQ, and llama-quantize for GGUF.

Quality trade-off, in practice:

  • 8-bit (FP8, Q8_0): treat as lossless for business tasks.
  • 5–6 bit: indistinguishable from 8-bit in almost all evals.
  • 4-bit (Q4_K_M, AWQ, NVFP4): small, measurable drops — most visible in long reasoning chains, code, math, and non-English text. Hungarian and German are where we check first.
  • 3-bit and below: only for very large models (a 2–3-bit 120B often beats a 4-bit 30B), and only with imatrix quants.
  • Prefer a bigger model at 4-bit over a smaller model at 8-bit when memory is fixed, then confirm with an eval.
  • Always evaluate the exact quantized artifact on the customer's eval set. Perplexity and KL-divergence against the BF16 model are useful smoke tests; task evals decide.
Make your own GGUF with an importance matrix
# from a llama.cpp build; model.gguf is the F16/BF16 conversion
./llama-imatrix -m model-f16.gguf -f calibration.txt -o imatrix.gguf
./llama-quantize --imatrix imatrix.gguf model-f16.gguf model-Q4_K_M.gguf Q4_K_M 8

4. Runtimes compared​

RuntimeWhat it isBest forMulti-user throughputFormatsNotes (as of 2026-10)
OllamaDaemon + model library on top of llama.cpp (and MLX on Apple Silicon since v0.40)Developer laptops, demos, small teamsLow–medium (OLLAMA_NUM_PARALLEL, default 1)GGUF, MLXLatest stable v0.40.6 (2026-10-16). Model swapping built in (keep_alive)
llama.cpp / llama-serverC/C++ engine with OpenAI-compatible serverMaximum control on any hardware: CUDA, Metal, ROCm, Vulkan, CPUMedium (--parallel slots, continuous batching)GGUFRouter mode loads/unloads many models in one server
LM StudioDesktop app + llmster headless daemon, lms CLIWorkstation users, Mac users wanting MLX with a GUILow–mediumGGUF, MLXServes on localhost:1234
vLLMHigh-throughput GPU server: PagedAttention, continuous batching, prefix cachingDefault for production GPU servingHighSafetensors (BF16, FP8, AWQ, GPTQ, NVFP4, MXFP4), limited GGUFLatest v0.32.0 (2026-10-18)
SGLangHigh-throughput GPU server with RadixAttention prefix cacheAgentic / multi-turn / shared-prefix workloads, large MoE, PD disaggregationHighSame as vLLMLatest v0.5.21 (GitHub, 2026-10-02); v0.5.22 on PyPI (2026-10-16)
MLX / mlx-lmApple's array framework and LLM packageApple Silicon, fine-tuning LoRA on a MacLow–mediumMLX (4/8-bit)mlx_lm.server is OpenAI-compatible
TensorRT-LLMNVIDIA's optimized engine, PyTorch backend, trtllm-serveSqueezing NVIDIA datacenter GPUs at scaleVery highHF checkpoints, FP8/NVFP4Advanced; see Serving at Scale
TGIHugging Face Text Generation Inference———Maintenance mode since Dec 2025, repo archived Mar 2026. Migrate to vLLM or SGLang
Docker Model Runnerdocker model CLI; models as OCI artifacts; llama.cpp or vLLM backendTeams standardised on Docker; distributing models through a registryLow–high (depends on backend)GGUF, SafetensorsvLLM backend Linux + NVIDIA only
LocalAIOpenAI-compatible "all-in-one" with 60+ backends (LLM, TTS, ASR, images)One API for many modalities on modest hardwareMediumManyAuth deny-by-default since v4.9

Decision in one line: laptop or Mac → Ollama (or LM Studio for GUI users); one box, full control, any hardware → llama-server; shared GPU server or anything with more than a handful of concurrent users → vLLM (SGLang when prefix reuse dominates).


Implementation steps​

Step 1 — Install Ollama and pull a small model​

# Linux (macOS/Windows: installer from ollama.com)
curl -fsSL https://ollama.com/install.sh | sh
ollama --version

ollama pull qwen3.5:4b # 3.3–4.0 GB, text + image
ollama run qwen3.5:4b "Explain KV cache in two sentences."
ollama ps # check PROCESSOR (GPU/CPU split) and CONTEXT columns

Step 2 — Call the OpenAI-compatible API​

curl
curl -s http://localhost:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "qwen3.5:4b",
"messages": [{"role": "user", "content": "Give three uses of local LLMs in a hospital."}],
"temperature": 0.2
}' | jq -r '.choices[0].message.content'
hello_local.py
# pip install openai
from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama") # key required by client, ignored by server

stream = client.chat.completions.create(
model="qwen3.5:4b",
messages=[{"role": "user", "content": "Summarise GDPR Article 28 in five bullets."}],
stream=True,
)
for chunk in stream:
print(chunk.choices[0].delta.content or "", end="", flush=True)

Swap base_url to http://gpu-host:8000/v1 and the same code talks to vLLM.

Step 3 — Set the context length explicitly​

Ollama's default context is chosen from VRAM: 4k below 23 GiB, 32k for 23–47 GiB, 256k at 47 GiB and above (as of 2026-10, per server/routes.go in the Ollama source; older releases used 2048). Prompts longer than the window are truncated from the start without an error — a RAG prompt or an agent's system prompt silently loses its head. Set it explicitly:

# Server-wide default (applies to /v1 requests too)
OLLAMA_CONTEXT_LENGTH=32768 ollama serve

# Or bake it into a model variant (the way to control context on the OpenAI endpoint)
cat > Modelfile <<'EOF'
FROM qwen3.5:4b
PARAMETER num_ctx 32768
EOF
ollama create qwen3-4b-32k -f Modelfile
  • The native /api/chat accepts "options": {"num_ctx": 32768} per request; the OpenAI-compatible /v1 endpoint has no field for it — use the server variable or a Modelfile.
  • OLLAMA_NUM_CTX is not a server variable; it does nothing.
  • Verify with ollama ps (CONTEXT column). A bigger context costs KV-cache memory: use the formula above.
  • Turn on KV-cache quantization when memory is tight: OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 halves KV memory.

Step 4 — Run llama-server directly​

# Prebuilt binaries or: cmake -B build -DGGML_CUDA=ON && cmake --build build -j
llama-server \
-hf ggml-org/gpt-oss-20b-GGUF \
--ctx-size 32768 \
--n-gpu-layers 999 \
--parallel 4 \
--flash-attn on \
--jinja \
--host 0.0.0.0 --port 8080
  • --ctx-size is the total across slots when KV is not unified: with --parallel 4 each slot gets 8k (--kv-unified changes this). Size it as per_request_ctx × parallel.
  • --n-gpu-layers 999 offloads everything; lower it to split between GPU and CPU when the model does not fit.
  • --jinja uses the chat template embedded in the GGUF — required for correct tool calling.
  • GET /health for probes, GET /metrics with --metrics.
curl -s localhost:8080/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"messages":[{"role":"user","content":"Hello"}]}' | jq -r '.choices[0].message.content'

Step 5 — Chat templates and tool calling​

A chat model was trained on a specific chat template (special tokens around system/user/assistant/tool turns). The runtime renders your messages array through that template. A wrong template is the most common cause of "the model rambles", "it never stops", or "tool calls come back as plain text".

  • Use templates shipped with the model: tokenizer_config.json / chat_template.jinja for vLLM, embedded GGUF metadata for llama.cpp (--jinja), the Modelfile TEMPLATE in Ollama.
  • vLLM needs a tool parser for tool calls: vllm serve <model> --enable-auto-tool-choice --tool-call-parser hermes (the parser name depends on the model family; check the model card).
  • Reasoning models emit a thinking section; servers separate it with a reasoning parser (--reasoning-parser in vLLM) so clients get clean content.
Tool calling against Ollama (OpenAI format)
curl -s http://localhost:11434/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "qwen3.5:4b",
"messages": [{"role": "user", "content": "What is the temperature in Budapest?"}],
"tools": [{
"type": "function",
"function": {
"name": "get_temperature",
"description": "Current temperature for a city",
"parameters": {"type": "object", "required": ["city"],
"properties": {"city": {"type": "string"}}}
}
}]
}' | jq '.choices[0].message.tool_calls'

Your code executes the function and sends the result back as a {"role": "tool", "tool_call_id": ..., "content": ...} message in a second request. Small models (≤ 4B) call tools less reliably; validate arguments against the schema and use structured outputs (response_format with a JSON schema) where the runtime supports it.


Best practices​

  • Pin everything: runtime version (container tag), model repo and revision/digest, quantization file name. "latest" is not a version.
  • Set context length on purpose in every runtime: Ollama OLLAMA_CONTEXT_LENGTH, llama.cpp --ctx-size, vLLM --max-model-len.
  • Size for concurrency, not for one user: KV cache × concurrent sequences often exceeds weights.
  • Prefer FP8 on Hopper/Ada and NVFP4/FP8 on Blackwell for GPU serving; GGUF Q4_K_M–Q6_K for laptops and Macs.
  • Evaluate the quantized artifact on the customer's tasks and languages before signing off a hardware quote.
  • Keep apps runtime-agnostic through the OpenAI API, so the PoC on Ollama moves to vLLM unchanged.
  • Bind to localhost by default (OLLAMA_HOST=127.0.0.1); expose through a gateway with auth and TLS, never directly.

Pitfalls to design around​

SymptomCauseWhat to do
Answers ignore the start of a long documentDefault context too small, silent truncationSet context explicitly, log prompt token counts
10× slower than expectedModel partly on CPUCheck ollama ps PROCESSOR column / llama.cpp load log; pick a smaller quant or shorter context
Tool calls appear as textWrong or missing chat template / tool parser--jinja (llama.cpp), --tool-call-parser (vLLM), official model tags
vLLM OOM at start--max-model-len defaults to the model's maximum (e.g. 128k+)Set --max-model-len to what the use case needs
Good English, weak HungarianAggressive quant hurts low-resource languages firstStep up one quant level; eval in the target language

Definition of done​

  • Sizing sheet filled with weights, KV cache at target context × concurrency, overhead
  • Runtime and quantization choice documented with the reason
  • Model revision and runtime version pinned
  • Context length set explicitly and verified (ollama ps, server logs)
  • OpenAI-compatible endpoint answers curl and Python smoke tests, including one tool call
  • Quantized model compared with a higher-precision baseline on the customer eval set
  • Endpoint bound to localhost or behind an authenticated gateway

Exercises​

  1. Size it. Using the formula, compute memory for Qwen3-32B at Q5_K_M (5.70 bpw) with 16k context and 4 parallel slots. Does it fit an RTX 5090? What changes with q8_0 KV cache?
  2. Find the truncation. Run Ollama with OLLAMA_CONTEXT_LENGTH=2048, send a 5,000-token document with a question about its first paragraph, and observe the answer. Repeat with 16384. Check ollama ps both times.
  3. Quant shoot-out. Pull the same model at Q4_K_M and Q8_0. Run ten domain questions (include Hungarian or German) against both; compare answers and eval_count / eval_duration tokens/s from /api/chat responses.
  4. Swap the backend. Point hello_local.py at llama-server on port 8080 without changing anything but base_url and model.
  5. Tool loop. Extend the tool-calling curl into a Python loop that executes get_temperature, returns the result, and prints the final answer.

Sources​