Local AI Best Practices
Service: On-Premise LLM Deployment · Sovereign & Private AI · Sovereign AI on STACKIT · AI Productification · Typical shape: baseline applied in every PoC; full checklist enforced before production go-live
Self-hosting moves responsibilities that a hosted API normally handles onto our side: artefact integrity, runtime exposure, prompt-injection containment, observability, and regulatory role. This page is the baseline for every local or private deployment. It complements How We Build, Inference Engineering and Serving at Scale.
Reference architecture
Principles: the engine only listens on a private interface; everything external goes through the gateway; models are pulled once into an internal store by a pipeline that scans, signs, and pins them; the engine never downloads from the internet at runtime.
1. Model supply-chain security
Formats: safetensors only
| Format | Risk | Policy |
|---|---|---|
.safetensors | Pure tensor data, no code execution on load | Default; require it. |
| GGUF | Data format for llama.cpp; parser bugs possible, no pickle | Allowed from the vendor or a known quantizer; keep llama.cpp patched. |
.bin / .pt / .pth / .ckpt (pickle) | Arbitrary code execution on load | Block. Convert in an isolated sandbox only if no alternative exists. |
Custom code (trust_remote_code=True) | Executes Python from the repo | Allowed only after code review and with a pinned revision. |
Scanners are a second line, not a guarantee: the 2026 ShadowPickle preprint evaded open-source scanners (PickleScan, ModelScan, Fickling) and hub scanners with crafted pickles; ReversingLabs showed broken/7z-compressed pickles slipping past PickleScan in 2025. A clean scan means "no known signature fired".
from transformers import AutoModelForCausalLM, AutoTokenizer
REPO = "Qwen/Qwen3.8-27B"
REV = "<40-char commit sha from the model's commit history>"
tok = AutoTokenizer.from_pretrained(REPO, revision=REV)
model = AutoModelForCausalLM.from_pretrained(
REPO,
revision=REV,
use_safetensors=True,
trust_remote_code=False,
torch_dtype="auto",
)
! grep -rn "from_pretrained" src/ | grep -v "revision="
! grep -rn "trust_remote_code=True" src/
Scanning, signing, provenance
| Control | Tool | How we use it |
|---|---|---|
| Pickle / artefact scan | ModelScan, picklescan, Fickling, ModelAudit | Run in the mirror pipeline; fail on any finding; still block pickle formats by policy. |
| Signing | OpenSSF model-signing (pip install model-signing, OMS spec v1.0) | Sign the model directory after scanning; verify at deploy time. Supports Sigstore keyless, PKI certs, or key pairs. |
| Provenance | HF commit SHA + model card + licence + scan report | Store as a record in the model register next to the signature. |
| Gated models | HF gated repos | Request access with a service account owned by the customer; access is granted per user, not per org. |
| Hub hygiene | HF security features | Fine-grained read-only tokens, 2FA, org SSO; never bake tokens into images. |
set -euo pipefail
REPO=Qwen/Qwen3.8-27B
REV=<commit-sha>
DEST=/models/qwen3.8-27b/$REV
hf download "$REPO" --revision "$REV" --local-dir "$DEST" \
--include "*.safetensors" "*.json" "tokenizer*" "*.txt" "LICENSE*"
modelscan -p "$DEST"
model_signing sign sigstore "$DEST" --signature "$DEST.sig"
sha256sum "$DEST"/*.safetensors > "$DEST.sha256"
At deploy time an init container runs model_signing verify against the expected identity and fails the pod on mismatch.
2. Runtime hardening
| Layer | Do this |
|---|---|
| Bind | Keep engines on localhost or a private interface. Ollama binds 127.0.0.1:11434 by default; keep it that way and put the gateway in front. In Docker, publish as 127.0.0.1:11434:11434. |
| Authenticate | Terminate auth at an API gateway (LiteLLM, Envoy AI Gateway, Kong, or nginx with OIDC) or a zero-trust tunnel (Cloudflare Access). Ollama has no built-in API authentication. |
| Patch | Track engine releases; CVE-2026-7482 ("Bleeding Llama", heap out-of-bounds read in the GGUF loader reachable via the unauthenticated /api/create) is fixed in Ollama 0.17.1; earlier versions are affected. Exposed endpoints are also targeted by LLMjacking campaigns. |
| Network policy | Kubernetes NetworkPolicy: engine pods accept traffic only from the gateway namespace; egress denied (models come from the internal store). |
| Control-plane endpoints | vLLM dev endpoints (/sleep, /wake_up, /collective_rpc) need VLLM_SERVER_DEV_MODE=1; never route them through the public gateway. |
| Least privilege | Non-root containers, read-only model volume, no host networking, GPU via device plugin only. |
| Quotas | Per-key rate and token limits at the gateway (OWASP LLM10 Unbounded Consumption). |
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: vllm-ingress-from-gateway
namespace: inference
spec:
podSelector:
matchLabels: { app: vllm }
policyTypes: [Ingress, Egress]
ingress:
- from:
- namespaceSelector:
matchLabels: { kubernetes.io/metadata.name: ai-gateway }
ports:
- port: 8000
egress: []
3. Prompt-injection defences for local agents
Prompt injection is LLM01 in the OWASP Top 10 for LLM Applications 2025, and agent goal hijack is ASI01 in the OWASP Top 10 for Agentic Applications 2026. No model-level fix exists; design so a hijacked model cannot do damage.
| Defence | Implementation |
|---|---|
| Least-privilege tools | Each tool has its own scoped credential; read-only by default; no generic shell or HTTP tool in production. |
| Human approval | Writes, payments, emails, deletions, and external calls require explicit confirmation. |
| Separate trusted and untrusted content | Retrieved documents, web pages, tool output go in clearly delimited data blocks; never concatenate into the system prompt. |
| Output handling | Validate every tool call against a JSON schema and an allow-list; treat model output as untrusted input (OWASP LLM05). |
| Sandbox code execution | gVisor/Firecracker/container sandbox with no network and no secrets. |
| Egress control | Agents cannot reach arbitrary URLs; prevents data exfiltration via crafted links. |
| MCP hygiene | Pin MCP server versions, review tool descriptions (they are prompt text), run servers with minimal scopes. |
| Red-team before release | Run an injection suite (e.g. promptfoo red-team presets for OWASP agentic risks) in CI. |
Details: Agentic AI Platforms.
4. PII handling and logging
- Classify first: decide per use case whether prompts may contain personal data (GDPR) or secrets; document it in the DPIA.
- Log metadata by default, content by exception: request ID, user/tenant ID (pseudonymised), model + revision, token counts, latency, tool calls. Store prompt/response bodies only where the use case requires it, with a retention period and access control.
- Redact at the gateway before logging (Presidio or regex rules for IBAN, Hungarian tax ID / TAJ numbers, emails, phone numbers).
- Keep data in region: logs, traces, and vector stores in the same sovereign boundary as the model (e.g. STACKIT region).
- Keep engine-side request logging minimal in production: check the pinned engine's logging flags so prompts are not written to engine logs, and log centrally at the gateway.
5. Pin, reproduce, evaluate before upgrade
Pinning
| Artefact | Pin to |
|---|---|
| Model weights | HF repo + full commit SHA (never main, never a tag) |
| Engine | Exact image digest, e.g. vllm/vllm-openai:v0.32.0@sha256:… (v0.32.0 is the latest release as of 2026-10-18) |
| Chat template / tokenizer | Comes with the pinned revision; override only via versioned file |
| Sampling params | Versioned config next to the deployment |
Reproducibility
- Set
temperature,top_p,top_k,min_p,presence_penalty,max_tokens, andseedexplicitly per request profile. Use the vendor-recommended values as default. - Greedy decoding (
temperature=0) is still not bit-exact across batch sizes, GPU types, or engine versions; evaluate with several samples and compare score distributions, not single strings. - Record engine version, GPU type, tensor-parallel size, quantization, and KV-cache dtype with every eval run.
Eval before every upgrade
- Golden set: 100–500 real tasks with reference answers or rubric; LLM-as-judge with a fixed judge model + human spot checks.
- Tools: lm-evaluation-harness for standard benchmarks, GuideLLM for load, promptfoo or the customer's eval framework for task suites.
- Gate on per-category thresholds, not one average.
6. Structured outputs
Use constrained decoding instead of "please answer in JSON". The engine masks invalid tokens, so output always parses.
| Engine | Mechanism (as of 2026-10) |
|---|---|
| vLLM | structured_outputs request field (json, regex, choice, grammar, structural_tag); backends xgrammar, guidance (llguidance), or auto. The old guided_* fields were removed in v0.12.0. |
| SGLang | xgrammar / llguidance backends |
| llama.cpp | GBNF grammars, JSON schema (-j); llguidance when built with -DLLAMA_LLGUIDANCE=ON |
| Ollama | format = JSON schema on /api/chat and /api/generate |
from openai import OpenAI
from pydantic import BaseModel
class Invoice(BaseModel):
supplier: str
total_eur: float
due_date: str
client = OpenAI(base_url="http://vllm.inference.svc:8000/v1", api_key="from-gateway")
resp = client.chat.completions.create(
model="qwen3.8-27b",
messages=[{"role": "user", "content": "Extract the invoice fields: ..."}],
extra_body={"structured_outputs": {"json": Invoice.model_json_schema()},
"chat_template_kwargs": {"enable_thinking": False}},
temperature=0,
)
invoice = Invoice.model_validate_json(resp.choices[0].message.content)
Choose xgrammar for repeated schemas and long outputs (grammar caching); llguidance for many changing schemas per request (low startup cost, ~50 µs/token mask computation). Still validate the parsed object; constraints guarantee syntax, not truth.
7. Choosing quantization per use case
| Use case | Recommended | Why |
|---|---|---|
| Production GPU serving, Hopper | FP8 (W8A8) official checkpoint | Near-lossless, native tensor-core support |
| Production GPU serving, Blackwell | NVFP4 / MXFP4 official checkpoint, FP8 fallback | Highest throughput per GPU; validate quality on your eval |
| Workstation / laptop | GGUF Q4_K_M–Q6_K; Q8_0 for small models | Best size/quality trade-off for llama.cpp / Ollama |
| Extraction, code, maths | Prefer ≥ 8-bit | Low-bit errors show first on exact tasks |
| Long-context RAG | FP8 KV cache | Halves KV memory; check accuracy at full context |
| Embedders / rerankers | BF16 or FP16 | Small models; quantization saves little and costs recall |
Deep dive: Inference Engineering.
8. Monitoring
| Signal | Source | Alert on |
|---|---|---|
| TTFT p95 | vllm:time_to_first_token_seconds | Above SLO (e.g. 3 s) for 10 min |
| Inter-token latency | vllm:inter_token_latency_seconds | Above SLO |
| Queue depth / saturation | vllm:num_requests_waiting, vllm:request_queue_time_seconds | Sustained growth |
| KV-cache pressure | vllm:kv_cache_usage_perc, vllm:num_preemptions_total | > 90 % or preemptions rising |
| GPU utilisation | DCGM_FI_PROF_SM_ACTIVE (truer than DCGM_FI_DEV_GPU_UTIL) | Low SM activity with queued requests (mis-config) |
| GPU health | DCGM XID errors, temperature, ECC | Any XID, thermal throttling |
| Power / energy | DCGM_FI_DEV_POWER_USAGE, DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION | Budget trends |
histogram_quantile(0.95,
sum by (le, model_name) (rate(vllm:time_to_first_token_seconds_bucket[5m])))
Metric names change between versions; check curl http://host:8000/metrics for your pinned engine. Deploy dcgm-exporter via the NVIDIA GPU Operator or its Helm chart (port 9400).
9. Energy and cost tracking
- GPU energy: integrate
DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION(mJ counter) per node; attribute to models via pod labels (dcgm-exporter -k). - Whole-node and CPU: Kepler (CNCF Sandbox, re-architected in 2026, v0.12 adds MIG power attribution via dcgm-exporter) for pod-level energy.
- Unit economics: publish
€ per 1M tokens= (GPU-hour cost + energy) ÷ tokens served, per model. Compare against hosted API prices in the AI Assessment. - Right-size: scale-to-zero or sleep mode for idle models; MoE models for throughput-bound workloads.
10. Licence compliance
- Maintain a model register: repo, commit SHA, licence ID, licence file hash, attribution requirements, revenue/MAU thresholds, date reviewed.
- Check thresholds against the customer's group revenue and business model (MaaS clauses in Qwen Community, Kimi K3, GLM-5.3; revenue cap in Mistral Medium 3.5; EU multimodal exclusion in Llama 4 AUP). Full table: Open Model Landscape.
- Ship required notices (Apache
NOTICE, "Built with Llama", model-name attribution) in product docs or UI. - Non-commercial (CC BY-NC) models stay in research environments.
11. EU AI Act: deployer vs. provider when fine-tuning
GPAI obligations apply since 2 August 2025; the AI Office enforcement powers apply since 2 August 2026.
| Situation | Role | Consequence |
|---|---|---|
| Customer runs an unmodified open model in its own application | Deployer of an AI system (and possibly provider of that system) | System-level duties depend on the use case's risk class; no GPAI-model provider duties. |
| Fine-tune / LoRA / continued pretraining using ≤ 1/3 of the original model's training compute | Downstream modifier, not a new GPAI provider | Normal case for our fine-tuning projects. Document the compute used. |
| Modification using > 1/3 of original training compute (if unknown: 1/3 of 10²³ FLOP, or of 10²⁵ FLOP for systemic-risk models) | Provider of a new GPAI model | Obligations (technical documentation, copyright policy, training-data summary) limited to the modification; systemic-risk models trigger full obligations incl. notifying the Commission. |
| Model released under a free and open-source licence | Provider exemptions for some documentation duties | Not for systemic-risk models. |
Practical rules:
- Log GPU-hours and estimated FLOP for every fine-tuning run (FLOP ≈ 6 × params × training tokens) and compare against the base model's published compute.
- Keep the training-data inventory for every fine-tune; it doubles as the GDPR record.
- For high-risk use cases (Annex III), plan system-level conformity work independently of the model question.
See Fine-Tuning and AI Assessment.
Pitfalls to design around
- Scanner-only security: pair scanning with a safetensors-only policy and signing.
- "Latest" tags: pin model SHAs and image digests; upgrades go through the eval gate.
- Exposed engine ports: bind to localhost/private interfaces and route all traffic through the authenticated gateway.
- Agents with broad credentials: scope each tool and add human approval for side effects.
- GPU-util as a health signal: use SM activity and queue metrics.
- Assuming fine-tuning makes you a GPAI provider (or never does): measure compute against the one-third threshold.
Printable checklist
Use your browser's print function on this section, or copy it into the project's go-live ticket.
Supply chain
- Only safetensors (or vendor GGUF) artefacts in the model store
-
trust_remote_codedisabled, or code reviewed and revision pinned - Every model pinned by HF commit SHA; engine pinned by image digest
- Artefacts scanned (ModelScan/picklescan/Fickling) in an isolated mirror job
- Model directories signed with OpenSSF model-signing; verified at deploy
- Model register entry: licence, thresholds, attribution, provenance
Runtime
- Engine bound to localhost/private interface; no public port
- Authenticated gateway with per-key quotas in front
- NetworkPolicy: ingress from gateway only, egress denied
- Non-root, read-only model volume, dev endpoints disabled
- Engine version current with security fixes
Application
- Tools least-privilege; human approval for side effects
- Untrusted content delimited; tool calls schema-validated
- Structured outputs via constrained decoding where output is parsed
- Injection red-team suite passing in CI
Data
- DPIA covers prompts, logs, vector stores
- PII redaction at gateway; content logging off by default; retention set
- Logs and vectors stored in the same sovereign region
Quality and operations
- Sampling parameters and seeds versioned
- Regression suite (incl. Hungarian) gates every model/engine/quant change
- Quantization chosen per use case and validated on the eval
- vLLM + DCGM metrics scraped; TTFT/ITL/queue/KV alerts live
- Energy and € per 1M tokens reported per model
Compliance
- Licence obligations shipped (notices, attribution)
- Fine-tuning compute logged and compared against the one-third threshold
- AI Act role (deployer / provider) documented per use case
Sources
- Supply chain: Transformers: loading models (trust_remote_code, revision) · HF Hub security · HF gated models · ShadowPickle (arXiv 2607.17503) · nullifAI pickle evasion (CSO Online) · ModelAudit · FaceHugger diffusers TOCTOU
- Signing: OpenSSF model-signing v1.0 launch · Model signing spec · Sigstore blog: model-transparency v1.0 · Google Security blog
- Runtime: Ollama FAQ · Ollama v0.17.1 release · CVE-2026-7482 (SentinelOne) · Cyera: Bleeding Llama · LeakIX: exposed Ollama instances · CSA research note: LLMjacking evolved · vLLM sleep mode (dev endpoints)
- Prompt injection: OWASP Top 10 for LLM Applications 2025 (PDF) · OWASP GenAI resources (Agentic Top 10) · promptfoo OWASP agentic red-team
- Structured outputs: vLLM structured outputs · Red Hat: structured outputs in vLLM · llguidance · llama.cpp llguidance docs · XGrammar-2 (arXiv 2601.04426) · Ollama structured outputs
- Engine versions: vLLM releases · vLLM on PyPI
- Monitoring and energy: vLLM production metrics · vLLM metrics design · Monitoring vLLM: PromQL and alerts · DCGM-Exporter docs · Kepler · Kepler CNCF project
- EU AI Act: Commission GPAI guidelines · GPAI guidelines policy page · GPAI FAQ · GPAI Code of Practice · Overview of GPAI guidelines
- Licences: Open Model Landscape (per-model sources)