Ugrás a fő tartalomhoz

Fine-Tuning Services

Advanced

Service: Fine-Tuning Services · Typical shape: 4–8 weeks; 1–2 ML engineers plus a client domain expert for data review; GPU capacity on STACKIT, OCI, AWS, or client hardware

What we deliver​

Open-weight models customised on the client's data for a specific, measurable task, delivered as versioned adapters (or merged weights) with an evaluation report and a serving configuration.

DeliverableArtefact
Decision recordWhy fine-tuning beats prompting and RAG for this task, with baseline numbers
DatasetVersioned train/validation/test splits, data card, licence and PII review
Training pipelineReproducible config (TRL, Unsloth, or Axolotl), container image, pinned versions
Model artefactsLoRA adapter(s) with model card, or merged weights; base model revision hash
Evaluation reportTask metrics vs baseline, general-capability regression, safety checks
ServingvLLM multi-LoRA config and gateway route; rollout via AI Productification practices

Decide first: prompting → RAG → fine-tuning​

Fine-tuning changes behaviour (format, style, task skill, tool-calling accuracy, latency through a smaller model). It is a poor vehicle for knowledge that changes or must be cited; that is what RAG is for.

SignalPoints to
Model knows the domain but output format or tone is inconsistentSFT with LoRA
A large model does the task well, cost or latency is too highDistillation into a smaller model
Answers must cite current documentsRAG (optionally fine-tune the retriever or reranker)
Correctness is machine-checkable (tests pass, JSON validates, numeric answer, SQL returns expected rows)SFT warm-up then RL with verifiable rewards
Reviewers can say "A is better than B" but not write the ideal answerPreference tuning (DPO, KTO)
Fewer than ~200 good examples and no way to get moreStay with prompting; invest in data first

Always record the prompting and RAG baselines on the same test set; they are the bar the fine-tuned model has to clear.

Techniques​

TechniqueWhat it doesWhen we use itTooling
SFTNext-token loss on curated input/output pairsDefault first step for any behaviour changeTRL SFTTrainer
LoRATrains low-rank adapter matrices, base frozenDefault for 1–70B models; many adapters per basePEFT LoraConfig
QLoRALoRA on a 4-bit (NF4) quantised baseSingle-GPU training of larger modelsbitsandbytes + PEFT; TRL quantization_config
DoRADecomposes weights into magnitude and direction; often closes part of the LoRA-vs-full gapWhen LoRA plateaus below full FT qualityPEFT use_dora=True
Full fine-tuningUpdates all weightsSmall models, large datasets, or continued pre-trainingTRL + FSDP / DeepSpeed ZeRO-3
DPOLearns from chosen/rejected pairs without a reward modelStyle, helpfulness, refusal calibrationTRL DPOTrainer (stable)
KTOLearns from unpaired good/bad labels (thumbs up/down)Production feedback without pairsTRL KTOTrainer
ORPOCombines SFT and preference loss in one stage, no reference modelCompute-constrained preference tuningTRL trl.experimental.orpo (experimental)
RLVR (GRPO family)Samples groups of completions, rewards them with programmatic checks, updates on group-relative advantageReasoning, code, SQL, extraction with checkable output, tool-callingTRL GRPOTrainer, Unsloth, verl
DistillationStudent learns from teacher outputs or logitsShrink a large model for latency, cost, or on-prem fitTRL distillation trainers; SFT on teacher outputs

RL with verifiable rewards: current state (as of 2026-10)​

GRPO removed the critic from PPO by normalising rewards within a group of samples for the same prompt. Its successors fix specific biases, and the current TRL defaults reflect them:

VariantChangeTRL setting
DAPOToken-level loss normalisation, clip-higher, dynamic samplingloss_type="dapo" (the default)
Dr. GRPORemoves length and std normalisation that inflates response lengthloss_type="dr_grpo"
GSPOSequence-level importance ratio and clipping; more stable for long sequences and MoEimportance_sampling_level="sequence"
CISPOClips importance-sampling weights instead of token updatesloss_type="cispo"
SAPOSoft, temperature-controlled gate instead of hard clippingloss_type="sapo"

The KL penalty defaults to beta=0.0 in TRL (no reference model loaded), following findings from DAPO and Dr. GRPO that KL is unnecessary for verifiable-reward reasoning.

Tooling: what is maintained (as of 2026-10)​

ToolStatusUse it for
Hugging Face TRLv1.16.0 (2026-10-15); stable API since v1.0; requires Python 3.11+Reference trainers: SFT, DPO, KTO, reward modelling, RLOO, GRPO, distillation
PEFTActiveLoRA, DoRA, adapter save/load/merge
UnslothActive, unsloth>=2026.10.3Fastest single-GPU LoRA/QLoRA and GRPO; patched kernels, lower VRAM
AxolotlActive (v0.21.0, 2026-10-16)YAML-driven multi-GPU runs, many model families, built on TRL
LLaMA-Factory (now "LlamaFactory")Active (v0.9.5, May 2026)CLI and web UI for teams that prefer zero-code configs
torchtuneNo longer actively developed (announced 2025; last release 0.6.1)Do not start new projects on it; migrate to TRL or Axolotl
verl, prime-rlActiveLarge-scale distributed RL beyond a single node

Our default: TRL + PEFT for anything that must be reproducible and reviewed; Unsloth when a single GPU is the constraint; Axolotl when the client team wants config-only multi-GPU runs.

Reference architecture​

Training runs as a container job on Kubernetes (Kueue or Volcano for GPU queueing) or Slurm, with data and artefacts in S3-compatible object storage in the client's jurisdiction. Experiment tracking: MLflow (STACKIT offers MLflow as a service), Weights & Biases, or Trackio.

Implementation steps​

1. Define the task and the test set​

  1. Write the task contract: input, output format, what "correct" means, unacceptable outputs.
  2. Build a held-out test set of 200–1,000 examples before any training data, reviewed by a domain expert. Freeze it; it is never used for training or synthetic-data seeding.
  3. Record baselines: base model with best prompt, base model with RAG if relevant, and the strongest available model as a ceiling.

2. Curate the dataset​

StepPractice
CollectReal task examples from tickets, documents, logs; record provenance per row
CleanDeduplicate (exact + MinHash near-duplicate), drop truncated or malformed rows, normalise whitespace and encodings
DecontaminateRemove rows with high n-gram overlap to the test set
PIIDetect and mask with Presidio or equivalent; DPO sign-off on the legal basis (see AI Assessment)
BalanceCover each sub-task and edge case; cap over-represented templates
FormatConversational messages or prompt/completion JSONL that matches the base model's chat template
ReviewDomain expert spot-checks a stratified sample (≥ 5%) and signs off

Quality beats quantity: a few thousand clean, diverse examples typically outperform a large noisy set for behaviour changes.

data/train.jsonl (one line, formatted)
{"messages": [
{"role": "system", "content": "Extract contract metadata as JSON."},
{"role": "user", "content": "<contract text>"},
{"role": "assistant", "content": "{\"party_a\": \"...\", \"effective_date\": \"2026-01-01\", \"governing_law\": \"HU\"}"}
]}

3. Generate synthetic data where real data is thin​

  • Seed a strong teacher (self-hosted where data must stay private) with real examples and task rules; generate variations, harder cases, and negatives.
  • Filter generated rows with the same validators used in evaluation (schema checks, unit tests, judge rubric), and deduplicate against real data and the test set.
  • Keep the synthetic share visible in the data card, and check the teacher model's licence terms on using outputs for training.
  • For RLVR you need prompts plus a checker, not reference answers: invest in the reward function instead.

4. Size the hardware​

Unsloth's published minimum VRAM for fine-tuning (absolute minimums; long sequences and larger batches need more):

Model sizeQLoRA (4-bit)LoRA (16-bit)
8B6 GB22 GB
14B8.5 GB33 GB
32B26 GB76 GB
70B41 GB164 GB

Full fine-tuning with AdamW in mixed precision needs roughly 16 bytes per parameter for weights, gradients, and optimizer states before activations: about 130 GB for an 8B model and over 1 TB for 70B. That means multi-GPU sharding (FSDP or DeepSpeed ZeRO-3) and is rarely justified over LoRA for task adaptation.

Practical defaults:

ScenarioHardware
7–14B QLoRA, up to 4k context1× 24–48 GB GPU (L4, L40S, RTX 6000 Ada class)
30–35B QLoRA, or 7–14B LoRA bf161× 80 GB (A100/H100 class)
70B QLoRA1× 80 GB, or 2× 48 GB
GRPO on 7–14B with vLLM generation colocated1–2× 80 GB; generation dominates wall-clock

See Inference Engineering for the GPU memory model and Private & Sovereign AI for where the GPUs live.

5. Train: minimal runnable SFT with QLoRA (TRL)​

Pinned to TRL 1.16.0 (as of 2026-10). The dataset uses the conversational format above; for Qwen3, TRL patches the chat template so assistant_only_loss=True works.

setup
python3.12 -m venv .venv && . .venv/bin/activate
pip install "trl==1.16.0" peft bitsandbytes datasets
train_sft_qlora.py
import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer

data = load_dataset("json", data_files={"train": "data/train.jsonl", "eval": "data/eval.jsonl"})

trainer = SFTTrainer(
model="Qwen/Qwen3-0.6B",
train_dataset=data["train"],
eval_dataset=data["eval"],
quantization_config=BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
),
peft_config=LoraConfig(
r=16,
lora_alpha=32,
lora_dropout=0.05,
target_modules="all-linear",
task_type="CAUSAL_LM",
),
args=SFTConfig(
output_dir="out/qwen3-0.6b-contracts-lora",
model_init_kwargs={"dtype": torch.bfloat16},
max_length=2048,
assistant_only_loss=True,
learning_rate=1e-4,
num_train_epochs=2,
per_device_train_batch_size=4,
gradient_accumulation_steps=4,
eval_strategy="steps",
eval_steps=100,
save_steps=200,
logging_steps=10,
seed=42,
report_to="none",
),
)
trainer.train()
trainer.save_model()

Swap Qwen/Qwen3-0.6B for the production base once the pipeline runs end to end. Set report_to="mlflow" (or "trackio") for tracking. For DoRA, add use_dora=True to LoraConfig.

6. Same job with Unsloth (single GPU, lower VRAM)​

train_unsloth.py
from unsloth import FastLanguageModel
from datasets import load_dataset
from trl import SFTConfig, SFTTrainer

model, tokenizer = FastLanguageModel.from_pretrained(
model_name="unsloth/Qwen3-4B",
max_seq_length=2048,
load_in_4bit=True,
)
model = FastLanguageModel.get_peft_model(
model,
r=16,
lora_alpha=16,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
use_gradient_checkpointing="unsloth",
random_state=3407,
)
trainer = SFTTrainer(
model=model,
processing_class=tokenizer,
train_dataset=load_dataset("json", data_files="data/train.jsonl", split="train"),
args=SFTConfig(
output_dir="out/qwen3-4b-unsloth",
per_device_train_batch_size=2,
gradient_accumulation_steps=4,
learning_rate=2e-4,
num_train_epochs=1,
logging_steps=10,
report_to="none",
),
)
trainer.train()
model.save_pretrained("out/qwen3-4b-unsloth/adapter")
tokenizer.save_pretrained("out/qwen3-4b-unsloth/adapter")

Install with pip install "unsloth>=2026.10.3" in a fresh environment; Unsloth pins compatible TRL and Transformers versions.

7. Preference and RL stages​

DPO on chosen/rejected pairs, starting from the SFT adapter:

train_dpo.py
from datasets import load_dataset
from peft import AutoPeftModelForCausalLM
from trl import DPOConfig, DPOTrainer

model = AutoPeftModelForCausalLM.from_pretrained("out/qwen3-0.6b-contracts-lora", is_trainable=True)
trainer = DPOTrainer(
model=model,
train_dataset=load_dataset("json", data_files="data/prefs.jsonl", split="train"), # prompt, chosen, rejected
args=DPOConfig(output_dir="out/dpo", learning_rate=5e-6, beta=0.1, num_train_epochs=1),
)
trainer.train()

GRPO with a verifiable reward (JSON validity plus field match against a label column). It starts from the SFT adapter merged into the base (AutoPeftModelForCausalLM.from_pretrained(...).merge_and_unload().save_pretrained("out/qwen3-0.6b-contracts-merged")); TRL passes extra dataset columns such as expected to the reward function as keyword arguments.

train_grpo.py
import json
from datasets import load_dataset
from trl import GRPOConfig, GRPOTrainer

def json_field_reward(completions, expected, **kwargs):
scores = []
for completion, exp in zip(completions, expected):
text = completion[0]["content"] if isinstance(completion, list) else completion
try:
got = json.loads(text)
except json.JSONDecodeError:
scores.append(-1.0)
continue
want = json.loads(exp)
scores.append(sum(got.get(k) == v for k, v in want.items()) / len(want))
return scores

trainer = GRPOTrainer(
model="out/qwen3-0.6b-contracts-merged",
reward_funcs=json_field_reward,
train_dataset=load_dataset("json", data_files="data/rl_prompts.jsonl", split="train"), # prompt, expected
args=GRPOConfig(
output_dir="out/grpo",
loss_type="dapo",
num_generations=8,
max_completion_length=512,
learning_rate=1e-6,
),
)
trainer.train()

Watch reward, completion length, and entropy curves; a rising reward with collapsing entropy or exploding length signals reward hacking. Inspect samples every few hundred steps.

8. Evaluate​

LayerHowGate
Task metricsHeld-out test set: exact match, field-level F1, unit-test pass rate, judge rubricBeats prompting/RAG baseline by the agreed margin
General capability regressionlm-evaluation-harness on a small suite (e.g. MMLU-style knowledge, GSM8K-style math, IFEval-style instruction following) vs the baseNo drop beyond agreed tolerance (e.g. 2 pts)
SafetyRed-team prompts, refusal behaviour, guard-model pass rateNo regression vs base
Format robustnessPrompt paraphrases, long inputs, other languages (Hungarian, German)Pass rate stable
Serving paritySame test set through vLLM with the adapterMatches offline scores within noise
regression check with lm-evaluation-harness
pip install "lm-eval[vllm]"
lm-eval --model vllm \
--model_args pretrained=Qwen/Qwen3-0.6B,max_lora_rank=16,lora_local_path=out/qwen3-0.6b-contracts-lora \
--tasks gsm8k,ifeval --batch_size auto --output_path results/adapter
lm-eval --model vllm --model_args pretrained=Qwen/Qwen3-0.6B \
--tasks gsm8k,ifeval --batch_size auto --output_path results/base

For models behind a running server, use --model local-chat-completions --model_args base_url=http://localhost:8000/v1/chat/completions,model=contracts.

9. Serve LoRA adapters on vLLM​

One base model, many adapters, selected per request by the model field:

serve
vllm serve Qwen/Qwen3-4B \
--enable-lora \
--lora-modules contracts=/adapters/contracts-v3 support=/adapters/support-v1 \
--max-loras 4 \
--max-lora-rank 64 \
--max-cpu-loras 16
curl http://vllm:8000/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"model": "contracts", "messages": [{"role": "user", "content": "..."}]}'
  • --max-loras: adapters active in one batch on GPU; --max-cpu-loras: LRU cache of loaded adapters in host memory; --max-lora-rank must cover the largest adapter rank.
  • Runtime loading via VLLM_ALLOW_RUNTIME_LORA_UPDATING=True and POST /v1/load_lora_adapter with {"lora_name": ..., "lora_path": ...}. The flag is off by default; keep it off in production, bake adapters into the deployment and roll them out through GitOps.
  • Merge the adapter into the base (model.merge_and_unload()) when a single fine-tuned model serves all traffic and you want quantised (AWQ, FP8) serving of the merged weights.
  • Register each adapter as its own model name at the LLM gateway and roll out with flags and canaries.

10. Check base-model licensing​

The adapter inherits the base model's licence obligations. Record the licence in the model card.

Licence family (as of 2026-10)ExamplesKey terms to check
Apache 2.0Qwen 3.x open models (check each card; some use a Qwen community licence), Gemma 4 (first Gemma under Apache 2.0), gpt-ossAttribution and NOTICE; no trademark rights
Llama 4 Community LicenseLlama 4 Scout, MaverickCheck the current licence text for the MAU threshold, naming and display requirements, and any EU-specific multimodal terms (not confirmed on this validation pass; the licence URL redirects)
Custom / research-onlyVariousCommercial use, output-use restrictions, field-of-use limits

Also check the licence of every dataset and the terms of any teacher model whose outputs are used for synthetic data or distillation.

Best practices​

  • Freeze the test set first; measure the baseline before training anything.
  • Start with LoRA r=16, all-linear, LR around 1e-4 (adapters) and 1–3 epochs; tune only after the data is right.
  • Train on assistant or completion tokens only; mismatched chat templates and EOS tokens are the most common cause of runaway generations.
  • Pin base model revision hashes, library versions, seeds, and dataset versions; store them on the model card.
  • Keep adapters small and task-specific; serve many on one base rather than training one adapter for everything.
  • Run general-capability regression on every candidate, not only the final one.

Pitfalls to design around​

  • Teaching facts by fine-tuning: put changing knowledge in RAG; fine-tune for behaviour.
  • Test-set leakage through synthetic data: decontaminate synthetic rows against the test set by n-gram overlap.
  • Catastrophic forgetting: mix 5–20% general instruction data into SFT if regression evals drop.
  • Reward hacking in RLVR: make rewards strict (parse, validate, compare) and inspect samples; add length or format penalties only with evidence.
  • Quantisation mismatch: an adapter trained on a 4-bit base behaves slightly differently on a bf16 or FP8 base; evaluate on the exact serving configuration.

Definition of done​

  • Decision record with prompting and RAG baselines on the frozen test set
  • Dataset versioned with data card: provenance, licence, PII review, synthetic share, decontamination report
  • Training pipeline reproducible from a single command with pinned versions and base revision hash
  • Adapter (or merged model) in the registry with a model card
  • Task metrics beat baseline by the agreed margin
  • General-capability and safety regression within tolerance
  • Serving parity confirmed on vLLM with the production configuration
  • Gateway route, flag, and canary plan in place
  • Base model and dataset licences reviewed and recorded
  • Retraining trigger and cadence documented (data drift, new base model release)

Sources​