Skip to main content

Local AI Best Practices

Basic Advanced

Service: On-Premise LLM Deployment · Sovereign & Private AI · Sovereign AI on STACKIT · AI Productification · Typical shape: baseline applied in every PoC; full checklist enforced before production go-live

Self-hosting moves responsibilities that a hosted API normally handles onto our side: artefact integrity, runtime exposure, prompt-injection containment, observability, and regulatory role. This page is the baseline for every local or private deployment. It complements How We Build, Inference Engineering and Serving at Scale.

Reference architecture​

Principles: the engine only listens on a private interface; everything external goes through the gateway; models are pulled once into an internal store by a pipeline that scans, signs, and pins them; the engine never downloads from the internet at runtime.

1. Model supply-chain security​

Formats: safetensors only​

FormatRiskPolicy
.safetensorsPure tensor data, no code execution on loadDefault; require it.
GGUFData format for llama.cpp; parser bugs possible, no pickleAllowed from the vendor or a known quantizer; keep llama.cpp patched.
.bin / .pt / .pth / .ckpt (pickle)Arbitrary code execution on loadBlock. Convert in an isolated sandbox only if no alternative exists.
Custom code (trust_remote_code=True)Executes Python from the repoAllowed only after code review and with a pinned revision.

Scanners are a second line, not a guarantee: the 2026 ShadowPickle preprint evaded open-source scanners (PickleScan, ModelScan, Fickling) and hub scanners with crafted pickles; ReversingLabs showed broken/7z-compressed pickles slipping past PickleScan in 2025. A clean scan means "no known signature fired".

load_model.py — the only allowed loader pattern
from transformers import AutoModelForCausalLM, AutoTokenizer

REPO = "Qwen/Qwen3.8-27B"
REV = "<40-char commit sha from the model's commit history>"

tok = AutoTokenizer.from_pretrained(REPO, revision=REV)
model = AutoModelForCausalLM.from_pretrained(
REPO,
revision=REV,
use_safetensors=True,
trust_remote_code=False,
torch_dtype="auto",
)
CI guard: no unpinned or remote-code loads
! grep -rn "from_pretrained" src/ | grep -v "revision="
! grep -rn "trust_remote_code=True" src/

Scanning, signing, provenance​

ControlToolHow we use it
Pickle / artefact scanModelScan, picklescan, Fickling, ModelAuditRun in the mirror pipeline; fail on any finding; still block pickle formats by policy.
SigningOpenSSF model-signing (pip install model-signing, OMS spec v1.0)Sign the model directory after scanning; verify at deploy time. Supports Sigstore keyless, PKI certs, or key pairs.
ProvenanceHF commit SHA + model card + licence + scan reportStore as a record in the model register next to the signature.
Gated modelsHF gated reposRequest access with a service account owned by the customer; access is granted per user, not per org.
Hub hygieneHF security featuresFine-grained read-only tokens, 2FA, org SSO; never bake tokens into images.
mirror-model.sh (runs in an isolated CI job)
set -euo pipefail
REPO=Qwen/Qwen3.8-27B
REV=<commit-sha>
DEST=/models/qwen3.8-27b/$REV

hf download "$REPO" --revision "$REV" --local-dir "$DEST" \
--include "*.safetensors" "*.json" "tokenizer*" "*.txt" "LICENSE*"
modelscan -p "$DEST"
model_signing sign sigstore "$DEST" --signature "$DEST.sig"
sha256sum "$DEST"/*.safetensors > "$DEST.sha256"

At deploy time an init container runs model_signing verify against the expected identity and fails the pod on mismatch.

2. Runtime hardening​

LayerDo this
BindKeep engines on localhost or a private interface. Ollama binds 127.0.0.1:11434 by default; keep it that way and put the gateway in front. In Docker, publish as 127.0.0.1:11434:11434.
AuthenticateTerminate auth at an API gateway (LiteLLM, Envoy AI Gateway, Kong, or nginx with OIDC) or a zero-trust tunnel (Cloudflare Access). Ollama has no built-in API authentication.
PatchTrack engine releases; CVE-2026-7482 ("Bleeding Llama", heap out-of-bounds read in the GGUF loader reachable via the unauthenticated /api/create) is fixed in Ollama 0.17.1; earlier versions are affected. Exposed endpoints are also targeted by LLMjacking campaigns.
Network policyKubernetes NetworkPolicy: engine pods accept traffic only from the gateway namespace; egress denied (models come from the internal store).
Control-plane endpointsvLLM dev endpoints (/sleep, /wake_up, /collective_rpc) need VLLM_SERVER_DEV_MODE=1; never route them through the public gateway.
Least privilegeNon-root containers, read-only model volume, no host networking, GPU via device plugin only.
QuotasPer-key rate and token limits at the gateway (OWASP LLM10 Unbounded Consumption).
networkpolicy-vllm.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: vllm-ingress-from-gateway
namespace: inference
spec:
podSelector:
matchLabels: { app: vllm }
policyTypes: [Ingress, Egress]
ingress:
- from:
- namespaceSelector:
matchLabels: { kubernetes.io/metadata.name: ai-gateway }
ports:
- port: 8000
egress: []

3. Prompt-injection defences for local agents​

Prompt injection is LLM01 in the OWASP Top 10 for LLM Applications 2025, and agent goal hijack is ASI01 in the OWASP Top 10 for Agentic Applications 2026. No model-level fix exists; design so a hijacked model cannot do damage.

DefenceImplementation
Least-privilege toolsEach tool has its own scoped credential; read-only by default; no generic shell or HTTP tool in production.
Human approvalWrites, payments, emails, deletions, and external calls require explicit confirmation.
Separate trusted and untrusted contentRetrieved documents, web pages, tool output go in clearly delimited data blocks; never concatenate into the system prompt.
Output handlingValidate every tool call against a JSON schema and an allow-list; treat model output as untrusted input (OWASP LLM05).
Sandbox code executiongVisor/Firecracker/container sandbox with no network and no secrets.
Egress controlAgents cannot reach arbitrary URLs; prevents data exfiltration via crafted links.
MCP hygienePin MCP server versions, review tool descriptions (they are prompt text), run servers with minimal scopes.
Red-team before releaseRun an injection suite (e.g. promptfoo red-team presets for OWASP agentic risks) in CI.

Details: Agentic AI Platforms.

4. PII handling and logging​

  • Classify first: decide per use case whether prompts may contain personal data (GDPR) or secrets; document it in the DPIA.
  • Log metadata by default, content by exception: request ID, user/tenant ID (pseudonymised), model + revision, token counts, latency, tool calls. Store prompt/response bodies only where the use case requires it, with a retention period and access control.
  • Redact at the gateway before logging (Presidio or regex rules for IBAN, Hungarian tax ID / TAJ numbers, emails, phone numbers).
  • Keep data in region: logs, traces, and vector stores in the same sovereign boundary as the model (e.g. STACKIT region).
  • Keep engine-side request logging minimal in production: check the pinned engine's logging flags so prompts are not written to engine logs, and log centrally at the gateway.

5. Pin, reproduce, evaluate before upgrade​

Pinning​

ArtefactPin to
Model weightsHF repo + full commit SHA (never main, never a tag)
EngineExact image digest, e.g. vllm/vllm-openai:v0.32.0@sha256:… (v0.32.0 is the latest release as of 2026-10-18)
Chat template / tokenizerComes with the pinned revision; override only via versioned file
Sampling paramsVersioned config next to the deployment

Reproducibility​

  • Set temperature, top_p, top_k, min_p, presence_penalty, max_tokens, and seed explicitly per request profile. Use the vendor-recommended values as default.
  • Greedy decoding (temperature=0) is still not bit-exact across batch sizes, GPU types, or engine versions; evaluate with several samples and compare score distributions, not single strings.
  • Record engine version, GPU type, tensor-parallel size, quantization, and KV-cache dtype with every eval run.

Eval before every upgrade​

  • Golden set: 100–500 real tasks with reference answers or rubric; LLM-as-judge with a fixed judge model + human spot checks.
  • Tools: lm-evaluation-harness for standard benchmarks, GuideLLM for load, promptfoo or the customer's eval framework for task suites.
  • Gate on per-category thresholds, not one average.

6. Structured outputs​

Use constrained decoding instead of "please answer in JSON". The engine masks invalid tokens, so output always parses.

EngineMechanism (as of 2026-10)
vLLMstructured_outputs request field (json, regex, choice, grammar, structural_tag); backends xgrammar, guidance (llguidance), or auto. The old guided_* fields were removed in v0.12.0.
SGLangxgrammar / llguidance backends
llama.cppGBNF grammars, JSON schema (-j); llguidance when built with -DLLAMA_LLGUIDANCE=ON
Ollamaformat = JSON schema on /api/chat and /api/generate
vLLM structured output via OpenAI client
from openai import OpenAI
from pydantic import BaseModel

class Invoice(BaseModel):
supplier: str
total_eur: float
due_date: str

client = OpenAI(base_url="http://vllm.inference.svc:8000/v1", api_key="from-gateway")
resp = client.chat.completions.create(
model="qwen3.8-27b",
messages=[{"role": "user", "content": "Extract the invoice fields: ..."}],
extra_body={"structured_outputs": {"json": Invoice.model_json_schema()},
"chat_template_kwargs": {"enable_thinking": False}},
temperature=0,
)
invoice = Invoice.model_validate_json(resp.choices[0].message.content)

Choose xgrammar for repeated schemas and long outputs (grammar caching); llguidance for many changing schemas per request (low startup cost, ~50 µs/token mask computation). Still validate the parsed object; constraints guarantee syntax, not truth.

7. Choosing quantization per use case​

Use caseRecommendedWhy
Production GPU serving, HopperFP8 (W8A8) official checkpointNear-lossless, native tensor-core support
Production GPU serving, BlackwellNVFP4 / MXFP4 official checkpoint, FP8 fallbackHighest throughput per GPU; validate quality on your eval
Workstation / laptopGGUF Q4_K_M–Q6_K; Q8_0 for small modelsBest size/quality trade-off for llama.cpp / Ollama
Extraction, code, mathsPrefer ≥ 8-bitLow-bit errors show first on exact tasks
Long-context RAGFP8 KV cacheHalves KV memory; check accuracy at full context
Embedders / rerankersBF16 or FP16Small models; quantization saves little and costs recall

Deep dive: Inference Engineering.

8. Monitoring​

SignalSourceAlert on
TTFT p95vllm:time_to_first_token_secondsAbove SLO (e.g. 3 s) for 10 min
Inter-token latencyvllm:inter_token_latency_secondsAbove SLO
Queue depth / saturationvllm:num_requests_waiting, vllm:request_queue_time_secondsSustained growth
KV-cache pressurevllm:kv_cache_usage_perc, vllm:num_preemptions_total> 90 % or preemptions rising
GPU utilisationDCGM_FI_PROF_SM_ACTIVE (truer than DCGM_FI_DEV_GPU_UTIL)Low SM activity with queued requests (mis-config)
GPU healthDCGM XID errors, temperature, ECCAny XID, thermal throttling
Power / energyDCGM_FI_DEV_POWER_USAGE, DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTIONBudget trends
p95 TTFT per model
histogram_quantile(0.95,
sum by (le, model_name) (rate(vllm:time_to_first_token_seconds_bucket[5m])))

Metric names change between versions; check curl http://host:8000/metrics for your pinned engine. Deploy dcgm-exporter via the NVIDIA GPU Operator or its Helm chart (port 9400).

9. Energy and cost tracking​

  • GPU energy: integrate DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION (mJ counter) per node; attribute to models via pod labels (dcgm-exporter -k).
  • Whole-node and CPU: Kepler (CNCF Sandbox, re-architected in 2026, v0.12 adds MIG power attribution via dcgm-exporter) for pod-level energy.
  • Unit economics: publish € per 1M tokens = (GPU-hour cost + energy) ÷ tokens served, per model. Compare against hosted API prices in the AI Assessment.
  • Right-size: scale-to-zero or sleep mode for idle models; MoE models for throughput-bound workloads.

10. Licence compliance​

  • Maintain a model register: repo, commit SHA, licence ID, licence file hash, attribution requirements, revenue/MAU thresholds, date reviewed.
  • Check thresholds against the customer's group revenue and business model (MaaS clauses in Qwen Community, Kimi K3, GLM-5.3; revenue cap in Mistral Medium 3.5; EU multimodal exclusion in Llama 4 AUP). Full table: Open Model Landscape.
  • Ship required notices (Apache NOTICE, "Built with Llama", model-name attribution) in product docs or UI.
  • Non-commercial (CC BY-NC) models stay in research environments.

11. EU AI Act: deployer vs. provider when fine-tuning​

GPAI obligations apply since 2 August 2025; the AI Office enforcement powers apply since 2 August 2026.

SituationRoleConsequence
Customer runs an unmodified open model in its own applicationDeployer of an AI system (and possibly provider of that system)System-level duties depend on the use case's risk class; no GPAI-model provider duties.
Fine-tune / LoRA / continued pretraining using ≤ 1/3 of the original model's training computeDownstream modifier, not a new GPAI providerNormal case for our fine-tuning projects. Document the compute used.
Modification using > 1/3 of original training compute (if unknown: 1/3 of 10²³ FLOP, or of 10²⁵ FLOP for systemic-risk models)Provider of a new GPAI modelObligations (technical documentation, copyright policy, training-data summary) limited to the modification; systemic-risk models trigger full obligations incl. notifying the Commission.
Model released under a free and open-source licenceProvider exemptions for some documentation dutiesNot for systemic-risk models.

Practical rules:

  1. Log GPU-hours and estimated FLOP for every fine-tuning run (FLOP ≈ 6 × params × training tokens) and compare against the base model's published compute.
  2. Keep the training-data inventory for every fine-tune; it doubles as the GDPR record.
  3. For high-risk use cases (Annex III), plan system-level conformity work independently of the model question.

See Fine-Tuning and AI Assessment.

Pitfalls to design around​

  • Scanner-only security: pair scanning with a safetensors-only policy and signing.
  • "Latest" tags: pin model SHAs and image digests; upgrades go through the eval gate.
  • Exposed engine ports: bind to localhost/private interfaces and route all traffic through the authenticated gateway.
  • Agents with broad credentials: scope each tool and add human approval for side effects.
  • GPU-util as a health signal: use SM activity and queue metrics.
  • Assuming fine-tuning makes you a GPAI provider (or never does): measure compute against the one-third threshold.

Printable checklist​

Print this

Use your browser's print function on this section, or copy it into the project's go-live ticket.

Supply chain

  • Only safetensors (or vendor GGUF) artefacts in the model store
  • trust_remote_code disabled, or code reviewed and revision pinned
  • Every model pinned by HF commit SHA; engine pinned by image digest
  • Artefacts scanned (ModelScan/picklescan/Fickling) in an isolated mirror job
  • Model directories signed with OpenSSF model-signing; verified at deploy
  • Model register entry: licence, thresholds, attribution, provenance

Runtime

  • Engine bound to localhost/private interface; no public port
  • Authenticated gateway with per-key quotas in front
  • NetworkPolicy: ingress from gateway only, egress denied
  • Non-root, read-only model volume, dev endpoints disabled
  • Engine version current with security fixes

Application

  • Tools least-privilege; human approval for side effects
  • Untrusted content delimited; tool calls schema-validated
  • Structured outputs via constrained decoding where output is parsed
  • Injection red-team suite passing in CI

Data

  • DPIA covers prompts, logs, vector stores
  • PII redaction at gateway; content logging off by default; retention set
  • Logs and vectors stored in the same sovereign region

Quality and operations

  • Sampling parameters and seeds versioned
  • Regression suite (incl. Hungarian) gates every model/engine/quant change
  • Quantization chosen per use case and validated on the eval
  • vLLM + DCGM metrics scraped; TTFT/ITL/queue/KV alerts live
  • Energy and € per 1M tokens reported per model

Compliance

  • Licence obligations shipped (notices, attribution)
  • Fine-tuning compute logged and compared against the one-third threshold
  • AI Act role (deployer / provider) documented per use case

Sources​