Fine-Tuning Services
Service: Fine-Tuning Services · Typical shape: 4–8 weeks; 1–2 ML engineers plus a client domain expert for data review; GPU capacity on STACKIT, OCI, AWS, or client hardware
What we deliver
Open-weight models customised on the client's data for a specific, measurable task, delivered as versioned adapters (or merged weights) with an evaluation report and a serving configuration.
| Deliverable | Artefact |
|---|---|
| Decision record | Why fine-tuning beats prompting and RAG for this task, with baseline numbers |
| Dataset | Versioned train/validation/test splits, data card, licence and PII review |
| Training pipeline | Reproducible config (TRL, Unsloth, or Axolotl), container image, pinned versions |
| Model artefacts | LoRA adapter(s) with model card, or merged weights; base model revision hash |
| Evaluation report | Task metrics vs baseline, general-capability regression, safety checks |
| Serving | vLLM multi-LoRA config and gateway route; rollout via AI Productification practices |
Decide first: prompting → RAG → fine-tuning
Fine-tuning changes behaviour (format, style, task skill, tool-calling accuracy, latency through a smaller model). It is a poor vehicle for knowledge that changes or must be cited; that is what RAG is for.
| Signal | Points to |
|---|---|
| Model knows the domain but output format or tone is inconsistent | SFT with LoRA |
| A large model does the task well, cost or latency is too high | Distillation into a smaller model |
| Answers must cite current documents | RAG (optionally fine-tune the retriever or reranker) |
| Correctness is machine-checkable (tests pass, JSON validates, numeric answer, SQL returns expected rows) | SFT warm-up then RL with verifiable rewards |
| Reviewers can say "A is better than B" but not write the ideal answer | Preference tuning (DPO, KTO) |
| Fewer than ~200 good examples and no way to get more | Stay with prompting; invest in data first |
Always record the prompting and RAG baselines on the same test set; they are the bar the fine-tuned model has to clear.
Techniques
| Technique | What it does | When we use it | Tooling |
|---|---|---|---|
| SFT | Next-token loss on curated input/output pairs | Default first step for any behaviour change | TRL SFTTrainer |
| LoRA | Trains low-rank adapter matrices, base frozen | Default for 1–70B models; many adapters per base | PEFT LoraConfig |
| QLoRA | LoRA on a 4-bit (NF4) quantised base | Single-GPU training of larger models | bitsandbytes + PEFT; TRL quantization_config |
| DoRA | Decomposes weights into magnitude and direction; often closes part of the LoRA-vs-full gap | When LoRA plateaus below full FT quality | PEFT use_dora=True |
| Full fine-tuning | Updates all weights | Small models, large datasets, or continued pre-training | TRL + FSDP / DeepSpeed ZeRO-3 |
| DPO | Learns from chosen/rejected pairs without a reward model | Style, helpfulness, refusal calibration | TRL DPOTrainer (stable) |
| KTO | Learns from unpaired good/bad labels (thumbs up/down) | Production feedback without pairs | TRL KTOTrainer |
| ORPO | Combines SFT and preference loss in one stage, no reference model | Compute-constrained preference tuning | TRL trl.experimental.orpo (experimental) |
| RLVR (GRPO family) | Samples groups of completions, rewards them with programmatic checks, updates on group-relative advantage | Reasoning, code, SQL, extraction with checkable output, tool-calling | TRL GRPOTrainer, Unsloth, verl |
| Distillation | Student learns from teacher outputs or logits | Shrink a large model for latency, cost, or on-prem fit | TRL distillation trainers; SFT on teacher outputs |
RL with verifiable rewards: current state (as of 2026-10)
GRPO removed the critic from PPO by normalising rewards within a group of samples for the same prompt. Its successors fix specific biases, and the current TRL defaults reflect them:
| Variant | Change | TRL setting |
|---|---|---|
| DAPO | Token-level loss normalisation, clip-higher, dynamic sampling | loss_type="dapo" (the default) |
| Dr. GRPO | Removes length and std normalisation that inflates response length | loss_type="dr_grpo" |
| GSPO | Sequence-level importance ratio and clipping; more stable for long sequences and MoE | importance_sampling_level="sequence" |
| CISPO | Clips importance-sampling weights instead of token updates | loss_type="cispo" |
| SAPO | Soft, temperature-controlled gate instead of hard clipping | loss_type="sapo" |
The KL penalty defaults to beta=0.0 in TRL (no reference model loaded), following findings from DAPO and Dr. GRPO that KL is unnecessary for verifiable-reward reasoning.
Tooling: what is maintained (as of 2026-10)
| Tool | Status | Use it for |
|---|---|---|
| Hugging Face TRL | v1.16.0 (2026-10-15); stable API since v1.0; requires Python 3.11+ | Reference trainers: SFT, DPO, KTO, reward modelling, RLOO, GRPO, distillation |
| PEFT | Active | LoRA, DoRA, adapter save/load/merge |
| Unsloth | Active, unsloth>=2026.10.3 | Fastest single-GPU LoRA/QLoRA and GRPO; patched kernels, lower VRAM |
| Axolotl | Active (v0.21.0, 2026-10-16) | YAML-driven multi-GPU runs, many model families, built on TRL |
| LLaMA-Factory (now "LlamaFactory") | Active (v0.9.5, May 2026) | CLI and web UI for teams that prefer zero-code configs |
| torchtune | No longer actively developed (announced 2025; last release 0.6.1) | Do not start new projects on it; migrate to TRL or Axolotl |
| verl, prime-rl | Active | Large-scale distributed RL beyond a single node |
Our default: TRL + PEFT for anything that must be reproducible and reviewed; Unsloth when a single GPU is the constraint; Axolotl when the client team wants config-only multi-GPU runs.
Reference architecture
Training runs as a container job on Kubernetes (Kueue or Volcano for GPU queueing) or Slurm, with data and artefacts in S3-compatible object storage in the client's jurisdiction. Experiment tracking: MLflow (STACKIT offers MLflow as a service), Weights & Biases, or Trackio.
Implementation steps
1. Define the task and the test set
- Write the task contract: input, output format, what "correct" means, unacceptable outputs.
- Build a held-out test set of 200–1,000 examples before any training data, reviewed by a domain expert. Freeze it; it is never used for training or synthetic-data seeding.
- Record baselines: base model with best prompt, base model with RAG if relevant, and the strongest available model as a ceiling.
2. Curate the dataset
| Step | Practice |
|---|---|
| Collect | Real task examples from tickets, documents, logs; record provenance per row |
| Clean | Deduplicate (exact + MinHash near-duplicate), drop truncated or malformed rows, normalise whitespace and encodings |
| Decontaminate | Remove rows with high n-gram overlap to the test set |
| PII | Detect and mask with Presidio or equivalent; DPO sign-off on the legal basis (see AI Assessment) |
| Balance | Cover each sub-task and edge case; cap over-represented templates |
| Format | Conversational messages or prompt/completion JSONL that matches the base model's chat template |
| Review | Domain expert spot-checks a stratified sample (≥ 5%) and signs off |
Quality beats quantity: a few thousand clean, diverse examples typically outperform a large noisy set for behaviour changes.
{"messages": [
{"role": "system", "content": "Extract contract metadata as JSON."},
{"role": "user", "content": "<contract text>"},
{"role": "assistant", "content": "{\"party_a\": \"...\", \"effective_date\": \"2026-01-01\", \"governing_law\": \"HU\"}"}
]}
3. Generate synthetic data where real data is thin
- Seed a strong teacher (self-hosted where data must stay private) with real examples and task rules; generate variations, harder cases, and negatives.
- Filter generated rows with the same validators used in evaluation (schema checks, unit tests, judge rubric), and deduplicate against real data and the test set.
- Keep the synthetic share visible in the data card, and check the teacher model's licence terms on using outputs for training.
- For RLVR you need prompts plus a checker, not reference answers: invest in the reward function instead.
4. Size the hardware
Unsloth's published minimum VRAM for fine-tuning (absolute minimums; long sequences and larger batches need more):
| Model size | QLoRA (4-bit) | LoRA (16-bit) |
|---|---|---|
| 8B | 6 GB | 22 GB |
| 14B | 8.5 GB | 33 GB |
| 32B | 26 GB | 76 GB |
| 70B | 41 GB | 164 GB |
Full fine-tuning with AdamW in mixed precision needs roughly 16 bytes per parameter for weights, gradients, and optimizer states before activations: about 130 GB for an 8B model and over 1 TB for 70B. That means multi-GPU sharding (FSDP or DeepSpeed ZeRO-3) and is rarely justified over LoRA for task adaptation.
Practical defaults:
| Scenario | Hardware |
|---|---|
| 7–14B QLoRA, up to 4k context | 1× 24–48 GB GPU (L4, L40S, RTX 6000 Ada class) |
| 30–35B QLoRA, or 7–14B LoRA bf16 | 1× 80 GB (A100/H100 class) |
| 70B QLoRA | 1× 80 GB, or 2× 48 GB |
| GRPO on 7–14B with vLLM generation colocated | 1–2× 80 GB; generation dominates wall-clock |
See Inference Engineering for the GPU memory model and Private & Sovereign AI for where the GPUs live.
5. Train: minimal runnable SFT with QLoRA (TRL)
Pinned to TRL 1.16.0 (as of 2026-10). The dataset uses the conversational format above; for Qwen3, TRL patches the chat template so assistant_only_loss=True works.
python3.12 -m venv .venv && . .venv/bin/activate
pip install "trl==1.16.0" peft bitsandbytes datasets
import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer
data = load_dataset("json", data_files={"train": "data/train.jsonl", "eval": "data/eval.jsonl"})
trainer = SFTTrainer(
model="Qwen/Qwen3-0.6B",
train_dataset=data["train"],
eval_dataset=data["eval"],
quantization_config=BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
),
peft_config=LoraConfig(
r=16,
lora_alpha=32,
lora_dropout=0.05,
target_modules="all-linear",
task_type="CAUSAL_LM",
),
args=SFTConfig(
output_dir="out/qwen3-0.6b-contracts-lora",
model_init_kwargs={"dtype": torch.bfloat16},
max_length=2048,
assistant_only_loss=True,
learning_rate=1e-4,
num_train_epochs=2,
per_device_train_batch_size=4,
gradient_accumulation_steps=4,
eval_strategy="steps",
eval_steps=100,
save_steps=200,
logging_steps=10,
seed=42,
report_to="none",
),
)
trainer.train()
trainer.save_model()
Swap Qwen/Qwen3-0.6B for the production base once the pipeline runs end to end. Set report_to="mlflow" (or "trackio") for tracking. For DoRA, add use_dora=True to LoraConfig.
6. Same job with Unsloth (single GPU, lower VRAM)
from unsloth import FastLanguageModel
from datasets import load_dataset
from trl import SFTConfig, SFTTrainer
model, tokenizer = FastLanguageModel.from_pretrained(
model_name="unsloth/Qwen3-4B",
max_seq_length=2048,
load_in_4bit=True,
)
model = FastLanguageModel.get_peft_model(
model,
r=16,
lora_alpha=16,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
use_gradient_checkpointing="unsloth",
random_state=3407,
)
trainer = SFTTrainer(
model=model,
processing_class=tokenizer,
train_dataset=load_dataset("json", data_files="data/train.jsonl", split="train"),
args=SFTConfig(
output_dir="out/qwen3-4b-unsloth",
per_device_train_batch_size=2,
gradient_accumulation_steps=4,
learning_rate=2e-4,
num_train_epochs=1,
logging_steps=10,
report_to="none",
),
)
trainer.train()
model.save_pretrained("out/qwen3-4b-unsloth/adapter")
tokenizer.save_pretrained("out/qwen3-4b-unsloth/adapter")
Install with pip install "unsloth>=2026.10.3" in a fresh environment; Unsloth pins compatible TRL and Transformers versions.
7. Preference and RL stages
DPO on chosen/rejected pairs, starting from the SFT adapter:
from datasets import load_dataset
from peft import AutoPeftModelForCausalLM
from trl import DPOConfig, DPOTrainer
model = AutoPeftModelForCausalLM.from_pretrained("out/qwen3-0.6b-contracts-lora", is_trainable=True)
trainer = DPOTrainer(
model=model,
train_dataset=load_dataset("json", data_files="data/prefs.jsonl", split="train"), # prompt, chosen, rejected
args=DPOConfig(output_dir="out/dpo", learning_rate=5e-6, beta=0.1, num_train_epochs=1),
)
trainer.train()
GRPO with a verifiable reward (JSON validity plus field match against a label column). It starts from the SFT adapter merged into the base (AutoPeftModelForCausalLM.from_pretrained(...).merge_and_unload().save_pretrained("out/qwen3-0.6b-contracts-merged")); TRL passes extra dataset columns such as expected to the reward function as keyword arguments.
import json
from datasets import load_dataset
from trl import GRPOConfig, GRPOTrainer
def json_field_reward(completions, expected, **kwargs):
scores = []
for completion, exp in zip(completions, expected):
text = completion[0]["content"] if isinstance(completion, list) else completion
try:
got = json.loads(text)
except json.JSONDecodeError:
scores.append(-1.0)
continue
want = json.loads(exp)
scores.append(sum(got.get(k) == v for k, v in want.items()) / len(want))
return scores
trainer = GRPOTrainer(
model="out/qwen3-0.6b-contracts-merged",
reward_funcs=json_field_reward,
train_dataset=load_dataset("json", data_files="data/rl_prompts.jsonl", split="train"), # prompt, expected
args=GRPOConfig(
output_dir="out/grpo",
loss_type="dapo",
num_generations=8,
max_completion_length=512,
learning_rate=1e-6,
),
)
trainer.train()
Watch reward, completion length, and entropy curves; a rising reward with collapsing entropy or exploding length signals reward hacking. Inspect samples every few hundred steps.
8. Evaluate
| Layer | How | Gate |
|---|---|---|
| Task metrics | Held-out test set: exact match, field-level F1, unit-test pass rate, judge rubric | Beats prompting/RAG baseline by the agreed margin |
| General capability regression | lm-evaluation-harness on a small suite (e.g. MMLU-style knowledge, GSM8K-style math, IFEval-style instruction following) vs the base | No drop beyond agreed tolerance (e.g. 2 pts) |
| Safety | Red-team prompts, refusal behaviour, guard-model pass rate | No regression vs base |
| Format robustness | Prompt paraphrases, long inputs, other languages (Hungarian, German) | Pass rate stable |
| Serving parity | Same test set through vLLM with the adapter | Matches offline scores within noise |
pip install "lm-eval[vllm]"
lm-eval --model vllm \
--model_args pretrained=Qwen/Qwen3-0.6B,max_lora_rank=16,lora_local_path=out/qwen3-0.6b-contracts-lora \
--tasks gsm8k,ifeval --batch_size auto --output_path results/adapter
lm-eval --model vllm --model_args pretrained=Qwen/Qwen3-0.6B \
--tasks gsm8k,ifeval --batch_size auto --output_path results/base
For models behind a running server, use --model local-chat-completions --model_args base_url=http://localhost:8000/v1/chat/completions,model=contracts.
9. Serve LoRA adapters on vLLM
One base model, many adapters, selected per request by the model field:
vllm serve Qwen/Qwen3-4B \
--enable-lora \
--lora-modules contracts=/adapters/contracts-v3 support=/adapters/support-v1 \
--max-loras 4 \
--max-lora-rank 64 \
--max-cpu-loras 16
curl http://vllm:8000/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"model": "contracts", "messages": [{"role": "user", "content": "..."}]}'
--max-loras: adapters active in one batch on GPU;--max-cpu-loras: LRU cache of loaded adapters in host memory;--max-lora-rankmust cover the largest adapter rank.- Runtime loading via
VLLM_ALLOW_RUNTIME_LORA_UPDATING=TrueandPOST /v1/load_lora_adapterwith{"lora_name": ..., "lora_path": ...}. The flag is off by default; keep it off in production, bake adapters into the deployment and roll them out through GitOps. - Merge the adapter into the base (
model.merge_and_unload()) when a single fine-tuned model serves all traffic and you want quantised (AWQ, FP8) serving of the merged weights. - Register each adapter as its own model name at the LLM gateway and roll out with flags and canaries.
10. Check base-model licensing
The adapter inherits the base model's licence obligations. Record the licence in the model card.
| Licence family (as of 2026-10) | Examples | Key terms to check |
|---|---|---|
| Apache 2.0 | Qwen 3.x open models (check each card; some use a Qwen community licence), Gemma 4 (first Gemma under Apache 2.0), gpt-oss | Attribution and NOTICE; no trademark rights |
| Llama 4 Community License | Llama 4 Scout, Maverick | Check the current licence text for the MAU threshold, naming and display requirements, and any EU-specific multimodal terms (not confirmed on this validation pass; the licence URL redirects) |
| Custom / research-only | Various | Commercial use, output-use restrictions, field-of-use limits |
Also check the licence of every dataset and the terms of any teacher model whose outputs are used for synthetic data or distillation.
Best practices
- Freeze the test set first; measure the baseline before training anything.
- Start with LoRA r=16,
all-linear, LR around 1e-4 (adapters) and 1–3 epochs; tune only after the data is right. - Train on assistant or completion tokens only; mismatched chat templates and EOS tokens are the most common cause of runaway generations.
- Pin base model revision hashes, library versions, seeds, and dataset versions; store them on the model card.
- Keep adapters small and task-specific; serve many on one base rather than training one adapter for everything.
- Run general-capability regression on every candidate, not only the final one.
Pitfalls to design around
- Teaching facts by fine-tuning: put changing knowledge in RAG; fine-tune for behaviour.
- Test-set leakage through synthetic data: decontaminate synthetic rows against the test set by n-gram overlap.
- Catastrophic forgetting: mix 5–20% general instruction data into SFT if regression evals drop.
- Reward hacking in RLVR: make rewards strict (parse, validate, compare) and inspect samples; add length or format penalties only with evidence.
- Quantisation mismatch: an adapter trained on a 4-bit base behaves slightly differently on a bf16 or FP8 base; evaluate on the exact serving configuration.
Definition of done
- Decision record with prompting and RAG baselines on the frozen test set
- Dataset versioned with data card: provenance, licence, PII review, synthetic share, decontamination report
- Training pipeline reproducible from a single command with pinned versions and base revision hash
- Adapter (or merged model) in the registry with a model card
- Task metrics beat baseline by the agreed margin
- General-capability and safety regression within tolerance
- Serving parity confirmed on vLLM with the production configuration
- Gateway route, flag, and canary plan in place
- Base model and dataset licences reviewed and recorded
- Retraining trigger and cadence documented (data drift, new base model release)
Sources
- TRL SFT Trainer docs (v1.16.0;
quantization_config,assistant_only_loss, PEFT usage): https://huggingface.co/docs/trl/sft_trainer - TRL GRPO Trainer docs (loss types, GSPO,
betadefault): https://huggingface.co/docs/trl/grpo_trainer - TRL releases (v1.14–v1.16, Python 3.10 dropped): https://github.com/huggingface/trl/releases
- TRL v1.0 announcement (stable vs experimental,
trl.experimental.orpo): https://huggingface.co/blog/trl-v1 - GRPO variants overview: https://huggingface.co/blog/sergiopaniego/agentic-rl-2026 · http://scalable-ai.eecs.berkeley.edu/assets/lecture_slides/lecture_15.pdf · GSPO paper: https://huggingface.co/papers/2507.18071
- torchtune end of active development: https://github.com/meta-pytorch/torchtune/issues/2883 · https://github.com/meta-pytorch/torchtune
- Unsloth releases and changelog: https://github.com/unslothai/unsloth/releases · https://unsloth.ai/docs/new/changelog
- Unsloth VRAM requirements: https://unsloth.ai/docs/get-started/fine-tuning-for-beginners/unsloth-requirements
- Unsloth GRPO pattern (FastLanguageModel / get_peft_model): https://rocm.docs.amd.com/projects/ai-developer-hub/en/latest/notebooks/fine_tune/unsloth_Llama3_1_8B_GRPO.html
- Axolotl releases: https://github.com/axolotl-ai-cloud/axolotl/releases · LlamaFactory releases: https://github.com/hiyouga/LlamaFactory/releases
- lm-evaluation-harness vLLM backend (
lora_local_path,max_lora_rank): https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/models/vllm_causallms.py - Axolotl, LLaMA-Factory, Unsloth, TRL comparison (2026): https://www.marktechpost.com/2026/07/22/unsloth-vs-axolotl-vs-trl-vs-llama-factory-a-fine-tuning-framework-comparison-on-speed-vram-and-multi-gpu/ · https://dev.to/ultraduneai/eval-003-fine-tuning-in-2026-axolotl-vs-unsloth-vs-trl-vs-llama-factory-2ohg
- vLLM LoRA adapters: https://docs.vllm.ai/en/latest/features/lora.html
- lm-evaluation-harness: https://github.com/EleutherAI/lm-evaluation-harness
- Gemma 4 under Apache 2.0: https://the-decoder.com/googles-gemma-4-is-now-available-with-apache-2-0-licensing-for-the-first-time/
- Open-weight licence comparison (Llama 4 terms, Apache 2.0 models): https://dev.to/llmradar/the-open-weight-licence-trap-apache-20-vs-the-community-licence-model-5dej