Open Model Landscape (as of 2026-10)
Service: On-Premise LLM Deployment · Sovereign & Private AI · RAG Systems on Private Infrastructure · Sovereign AI on STACKIT · Typical shape: model shortlist + eval run in the first week of every PoC; quarterly refresh for running platforms
Every model, size, date, and licence on this page was checked against the Hugging Face API and model cards, Ollama library pages, or vendor announcements on 18 October 2026. Benchmark scores are vendor-reported in model cards unless marked otherwise. Open-weight releases ship monthly; re-verify against the LICENSE file of the exact repo before you commit a customer to a model. The update procedure is at the end of this page.
What we deliver
- A shortlist of 2–4 candidate models per use case, justified against the selection criteria below.
- A hardware fit calculation (weights + KV cache at target context and concurrency) for the customer's GPUs. See Inference Engineering.
- A licence review for every model artefact (LLM, embedder, reranker, quantized variant) that goes to production.
- An eval result on the customer's own data in the languages they use. Leaderboards narrow the field; the customer eval picks the winner.
How to choose a model
Work through the criteria in this order. The first three remove most candidates.
| # | Criterion | Question to answer | How to check |
|---|---|---|---|
| 1 | Licence | Can the customer use it commercially in their setting (internal tool, SaaS, MaaS)? Are there revenue/MAU thresholds or regional restrictions? | LICENSE file in the HF repo, not the press release. See the licence table. |
| 2 | Hardware fit | Do weights + KV cache fit at the target context and concurrency? | Weights ≈ total params × bytes per param (BF16 = 2, FP8 = 1, 4-bit ≈ 0.55). MoE models need all experts in memory; only compute scales with active params. |
| 3 | Task | Chat, coding agent, RAG answerer, extraction, vision, tool use? | Model card benchmarks for the task family; the customer's own eval set. |
| 4 | Languages | Does it handle the customer's languages at the needed quality? | Run an eval set in each target language. Check tokenizer fertility: more tokens per word means higher latency and cost. |
| 5 | Context length | Longest realistic prompt (RAG chunks + history + tool results)? | Native vs. extended context on the card. Extended context can reduce short-text quality. |
| 6 | Tool calling | Native tool-call format supported by the serving engine's parser? | vLLM/SGLang tool parser for the model; test with your real tool schemas. |
| 7 | Reasoning modes | Need a thinking mode? Can it be switched off per request for latency? | Chat template flags and reasoning_effort on the card. Reasoning multiplies output tokens. |
| 8 | Ecosystem | vLLM / SGLang / llama.cpp / Ollama support, official FP8/NVFP4/GGUF quants, speculative-decoding heads? | Model card "Quickstart" + engine release notes. |
Rules of thumb (as of 2026-10):
- Small dense vs. small MoE: a 30B-class MoE with ~3B active (Nemotron 3.5 Lightning 30B-A3B, Qwen3-Coder-30B-A3B, Gemma 4 26B A4B) gives much higher throughput than a 27–31B dense model, at the cost of memory for all experts. Dense 27–31B models (Qwen3.8-27B, Gemma 4 31B) are simpler to fine-tune and quantize.
- Thinking is a cost lever: reasoning-on models emit many more output tokens. Budget them explicitly and disable thinking for extraction and classification.
- Frontier open MoE is not self-hostable everywhere: 750B–2.8T models need 8+ datacentre GPUs or multi-node. For most private deployments, the 27–128B band is the sweet spot.
Top-tier open-weight models (as of 2026-10)
Verified against the HF repo (safetensors total, licence tag, README). Min hardware assumes 4-bit weights (≈ 0.55 bytes per param) unless noted, weights only; add KV cache for your context.
| Model (HF repo) | Org | Total / active params | Context | Licence | Best for | Min hardware |
|---|---|---|---|---|---|---|
Qwen/Qwen3.8-27B (Aug 2026) | Alibaba Qwen | 27.8B dense | 262,144 native, to 1M | Apache-2.0 | General assistant, vision, coding | 1 × 24 GB (Ollama qwen3.8:27b is 18 GB) |
Qwen/Qwen3-Coder-30B-A3B-Instruct (Jul 2025) | Alibaba Qwen | 30.5B / ~3.3B | 262,144 native | Apache-2.0 | Coding agent on one GPU; newest ~30B Qwen coder | 1 × 24 GB (~17 GB) |
Qwen/Qwen3-Coder-Next (Jan 2026) | Alibaba Qwen | 80B / 3B | 262,144 native | Apache-2.0 | Newer coder for agent loops; not ~30B | 1 × 48 GB (~45 GB) |
google/gemma-4-31B-it (Mar 2026) | 30.7B dense | 256K (medium models) | Apache-2.0 | RAG answerer, vision | 1 × 24 GB (~17 GB) | |
google/gemma-4-26B-A4B-it (Mar 2026) | 25.8B / ~4B | 256K (medium models) | Apache-2.0 | Fast RAG answerer | 1 × 24 GB (~14 GB) | |
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 (Aug 2026) | NVIDIA | 30B / 3B | not checked | OpenMDW-1.1 | Tool calling, agent execution | 1 × 24 GB (~17 GB); NVFP4 repo also published |
openai/gpt-oss-120b (Aug 2025) | OpenAI | 117B / 5.1B | not checked | Apache-2.0 | Reasoning + tool use; no successor found | 1 × 80 GB (MXFP4 weights) |
mistralai/Mistral-Small-4-119B-2603 (2603 build) | Mistral | 119B / 6.5B | 256K | Apache-2.0 | General + reasoning + vision; EU vendor | 1 × 80 GB (~65 GB); NVFP4 repo also published |
mistralai/Mistral-Medium-3.5-128B (Mar 2026) | Mistral | 128B dense | not checked | Modified MIT (revenue exception) | Coding agent | 1–2 × 80 GB (~70 GB at 4-bit) |
zai-org/GLM-5.3 (Aug 2026) | Z.ai | 753B / not stated on card | 1M (eval setting) | Custom GLM-5.3 licence | Flagship agentic coding; reasoning_effort low/high/max | Multi-node; ~750 GB at FP8 |
zai-org/GLM-5.3-Flash (Aug 2026) | Z.ai | 320B / 18B | not stated on card | MIT | Smaller GLM; natively multimodal | Multi-GPU; ~180 GB at 4-bit |
moonshotai/Kimi-K3 (Jun 2026) | Moonshot | 2.8T / 104B | 1M | Kimi K3 License | Frontier coding and agentic work; native vision | Multi-node; ~1.5 TB at 4-bit |
deepseek-ai/DeepSeek-V4.1-Flash (Sep 2026) | DeepSeek | 552B backbone (763B in files); 8B prefill / 16B decode active | 1M | MIT | Long-context RAG, low cost per token; image input | Multi-node |
deepseek-ai/DeepSeek-V4-Flash (Apr 2026) | DeepSeek | 284B / 13B | 1M | MIT | Long-context agents, text | Multi-GPU server (~290 GB files) |
deepseek-ai/DeepSeek-V4-Pro (preview, Apr 2026) | DeepSeek | 1.6T / 49B | 1M | MIT | Frontier general | Multi-node |
XiaomiMiMo/MiMo-V2.6-Pro-MOPD (Sep 2026) | Xiaomi | 1.02T / not checked | not checked | MIT | Frontier general, agentic | Multi-node |
MiniMaxAI/MiniMax-M3 (Jun 2026) | MiniMax | 428B / 23B | 1M | MiniMax community licence | Long-context agents; native image and video input | ~240 GB at 4-bit; 4 × 80 GB class |
Not yet shortlistable: Mistral Large 4 (announced 6 Oct 2026: 1T total / 52B active, multimodal). Public API preview only; Mistral says weights follow by end of October 2026, and no HF repo or licence was found as of 2026-10-18. Re-check before the next review.
Which to pick (as of 2026-10):
| Need | First pick | Next option | Why |
|---|---|---|---|
| General assistant | Qwen3.8-27B | Gemma 4 31B | One GPU, Apache-2.0, native vision |
| Coding agent | Qwen3-Coder-30B-A3B-Instruct (1 GPU) | Qwen3-Coder-Next (1–2 GPUs); Kimi K3 or GLM-5.3 (frontier) | Low active params for agent loops |
| RAG answerer | Gemma 4 26B A4B | Qwen3.8-27B with thinking off | Fast MoE, 256K context |
| Small / edge | Qwen3.5-0.8B / 2B / 4B / 9B | Gemma 4 E2B / E4B | Smallest current generation |
| Tool-calling agent | Nemotron 3.5 Lightning 30B-A3B | gpt-oss-120b (1 × 80 GB) | Built for agent execution; configurable reasoning |
Current open-weight families
Latest versions only. Dates are the HF createdAt or the vendor announcement. Rows were verified against HF; older generations are dropped unless they are still the best choice for a niche.
| Family (vendor) | Latest open weights (as of 2026-10) | Sizes (total / active) | Licence | Notes |
|---|---|---|---|---|
| Qwen (Alibaba) | Qwen3.8-27B (Aug 2026); Qwen3.8-Flash-Next (Aug 2026); Qwen3.8-2.4T-A95B (Aug 2026); Qwen3.6-27B and 35B-A3B (Apr 2026); Qwen3.5 0.8B / 2B / 4B / 9B (Feb 2026) | 27.8B dense; ~180B (Flash-Next, active not verified); 2.45T / 95B; 27B; 35B / 3B; 0.8–9B | Apache-2.0 except Flash-Next (Qwen Community License 1.0) and 2.4T (Qwen licence) | 262K native context (to 1M). Thinking on by default. No Qwen3.8 small sizes found on HF or Ollama. No Qwen 4 open weights found. |
| Gemma (Google) | Gemma 4 E2B, E4B, 12B (May 2026), 26B A4B, 31B (Mar 2026) | E2B 5.1B and E4B 8B with embeddings (2.3B / 4.5B effective); 12B 11.95B; 26B A4B 25.8B; 31B 30.7B | Apache-2.0 | 128K (small) / 256K (medium) context; thinking toggle; native function calling. |
| gpt-oss (OpenAI) | gpt-oss-120b, gpt-oss-20b (Aug 2025); gpt-oss-safeguard-120b / 20b (Sep 2025) | 117B / 5.1B; 21B / 3.6B | Apache-2.0 | No successor found on HF as of 2026-10. |
| Mistral | Mistral Small 4 (2603 build); Mistral Medium 3.5 (Mar 2026); Mistral Large 3 (Dec 2025); Devstral Small 2 (Dec 2025); Ministral 3 3B / 8B / 14B (Dec 2025) | 119B / 6.5B; 128B dense; 675B total; 24B; 3–14B | Apache-2.0 (Small 4, Large 3, Devstral Small 2, Ministral 3); Modified MIT (Medium 3.5) | Mistral Large 4 is announced but not on HF (see above). |
| DeepSeek | DeepSeek-V4-Pro (preview), V4-Flash (Apr 2026), V4.1-Flash (Sep 2026) | 1.6T / 49B; 284B / 13B; 552B backbone / 8B–16B | MIT | 1M context. V4.1-Flash adds image input. |
| Kimi (Moonshot) | Kimi K3 (Jun 2026); Kimi K2.7-Code (Jun 2026); Kimi K2.6 (Apr 2026) | 2.8T / 104B; 1.03T total (K2.7-Code, active not verified) | Kimi K3 License (custom, MIT-derived); other Kimi K2 licences | Native vision; 1M context. |
| GLM (Z.ai) | GLM-5.3 (Aug 2026); GLM-5.3-Flash (Aug 2026); GLM-5.2 (Jun 2026); GLM-5.1 (Apr 2026) | 753B (active not on card); 320B / 18B | GLM-5.3: custom glm-5.3 licence; GLM-5.3-Flash, GLM-5.2, GLM-5.1: MIT | Hybrid sparse + linear attention on Flash. Thinking effort via reasoning_effort. |
| MiMo (Xiaomi) | MiMo-V2.6-Pro-MOPD and Flash-MOPD (Sep 2026) | 1.02T total; 311B total | MIT | Flash-MOPD is the smaller variant. |
| MiniMax | MiniMax-M3 (Jun 2026) | 428B / 23B | MiniMax community licence | 1M context; written authorisation needed above US$20M yearly revenue. |
| NVIDIA Nemotron | Nemotron 3.5 Lightning 30B-A3B (Aug 2026); Nemotron 3.5 Bolt 74B-A9B (Oct 2026); Nemotron 3.5 Super VL 120B-A12B (Oct 2026); Nemotron 3 Super 120B-A12B (Mar 2026); Nemotron 3 Ultra 550B-A55B (Jun 2026) | 30B / 3B; 74B / 9B; 120B / 12B; 120B / 12B; 550B / 55B | OpenMDW-1.1 (3.5 family, Ultra); NVIDIA Nemotron Open Model License (Nemotron 3 Super) | 3.5 Super VL is multimodal, 1M context. Check each card's licence. |
| Granite (IBM) | Granite 4.1 8B (8.8B) and 30B (28.9B), Apr 2026 | 8B and 30B dense | Apache-2.0 | Enterprise-friendly provenance. |
| Olmo (Ai2) | Olmo 3.1 32B Instruct (Dec 2025) | 32.2B dense | Apache-2.0 | Fully open: weights, data, and training recipe. Reference when a customer needs training-data transparency. |
| Phi (Microsoft) | Phi-4-reasoning-vision-15B (Jan 2026) | 15.1B dense | MIT | No Phi-5 repo found on HF as of 2026-10. |
| Llama (Meta) | Llama 4 Scout / Maverick (Apr 2025) | Scout 108.6B total (17B active) | Llama 4 Community License | Not re-verified beyond the Scout HF repo. Check the EU multimodal terms in the licence before use. |
| EuroLLM (EU consortium) | EuroLLM-22B-Instruct-2512 (Dec 2025) | 22.6B dense | Apache-2.0 | Covers the 24 official EU languages; 32K context. |
Recommended picks by tier
Picks are starting points for the shortlist, ordered by our default preference. Memory figures are weights at 4-bit unless noted.
Tier 1: laptop / edge (≤ 9B; 8–32 GB unified memory or 8–16 GB VRAM)
| Use case | First pick | Alternatives | Why |
|---|---|---|---|
| Chat / assistant | Qwen3.5-9B | Gemma 4 E4B, Granite 4.1 8B | Apache-2.0, 256K context; Ollama qwen3.5:9b is 6.6–7.6 GB. |
| Coding | Qwen3.5-9B | gpt-oss-20b (21B / 3.6B active) | Small active footprint for fast local loops. |
| RAG answerer | Granite 4.1 8B | Qwen3.5-4B, Ministral 3 8B | Apache-2.0. Check the 8B card for context length. |
| Agent / tool use | gpt-oss-20b | Qwen3.5-9B | Trained for tool use with configurable reasoning effort. |
| Vision | Gemma 4 E4B | Qwen3.5-4B, Ministral 3 8B | Native image input at phone-class size. |
| Small / edge | Gemma 4 E2B | Qwen3.5-0.8B / 2B | Gemma 4 E2B is about 2.5–3 GB at 4-bit (estimate from 5.1B params). |
Tier 2: single workstation GPU (14–35B; 24–48 GB VRAM, e.g. RTX 4090/5090, L40S, RTX PRO 6000)
| Use case | First pick | Alternatives | Why |
|---|---|---|---|
| Chat / assistant | Qwen3.8-27B | Gemma 4 31B, Ministral 3 14B | Current Qwen generation; 18 GB at Ollama default quant. |
| Coding agent | Qwen3-Coder-30B-A3B-Instruct | Qwen3.8-27B, Devstral Small 2 (24B) | Newest ~30B Qwen coder; 3B active. |
| RAG answerer | Gemma 4 26B A4B | Qwen3.8-27B (thinking off), Granite 4.1 30B | Fast MoE, Apache-2.0. |
| Agent / tool use | Nemotron 3.5 Lightning 30B-A3B | Qwen3.6-35B-A3B (Apache-2.0) | Built for agent execution. |
| Vision | Qwen3.8-27B | Gemma 4 31B, Phi-4-reasoning-vision-15B (MIT) | Native image input. |
| EU-provenance requirement | EuroLLM-22B-Instruct-2512 | Ministral 3 14B, Olmo 3.1 32B | European training, documented data. |
Tier 3: single server (70–130B or mid-size MoE; 1–4 × 80–141 GB GPUs)
| Use case | First pick | Alternatives | Why |
|---|---|---|---|
| Chat / assistant | Mistral Small 4 (119B / 6.5B) | gpt-oss-120b, Qwen3.5-122B-A10B | Apache-2.0, reasoning + vision + coding, EU vendor. |
| Coding agent | Mistral Medium 3.5 (128B dense) | Qwen3-Coder-Next (80B / 3B), gpt-oss-120b | Modified MIT. Check the revenue exception first. |
| RAG answerer | gpt-oss-120b | Mistral Small 4, Nemotron 3 Super | Fits one 80 GB GPU; cheap per token. |
| Agent / tool use | Nemotron 3 Super (120B / 12B) | gpt-oss-120b, Mistral Small 4 | 1M context; check the licence (NVIDIA Nemotron Open Model License). |
| Vision | Mistral Small 4 | Qwen3.5-122B-A10B | Native image input. |
Tier 4: multi-GPU / multi-node frontier MoE (8+ × H100/H200/B200)
| Use case | First pick | Alternatives | Why |
|---|---|---|---|
| General frontier | DeepSeek-V4-Pro (1.6T / 49B) | MiMo-V2.6-Pro, GLM-5.3, Kimi K3 | MIT licence; 1M context. |
| Coding agent | Kimi K3 (2.8T / 104B) | GLM-5.3, MiniMax-M3 | Check the Kimi K3 licence thresholds before delivery. |
| Long-context RAG / cost | DeepSeek-V4-Flash (284B / 13B) | DeepSeek-V4.1-Flash, GLM-5.3-Flash | MIT, 1M context, low active params. |
| EU-built frontier | Mistral Large 3 (675B total, Apache-2.0) | Mistral Large 4 when weights land | European vendor; verify the Large 4 licence on release. |
Embedding and reranker models
The live MTEB leaderboard is a JavaScript app and could not be read in this pass. Scores below come from model cards (vendor-reported, snapshot date on the card). Re-check the leaderboard before you shortlist.
Embedders
| Model (HF repo) | Params | Dims | Max tokens | Licence | Score (source) | Multilingual |
|---|---|---|---|---|---|---|
microsoft/harrier-oss-v1-27b (Mar 2026) | 27.0B | 5,376 | 32,768 | MIT | MTEB v2 multilingual 74.3 (card) | Yes |
microsoft/harrier-oss-v1-0.6b | 0.6B | 1,024 | 32,768 | MIT | MTEB v2 multilingual 69.0 (card) | Yes |
nvidia/Nemotron-3.5-Embed-8B-BF16 (Oct 2026) | 8.4B | 4,096 (truncatable) | 32,768 | OpenMDW-1.1 | not extracted from card | Yes; text and page images |
nvidia/Nemotron-3-Embed-8B-BF16 (Jul 2026) | 8.0B | 4,096 (truncatable) | 32,768 | OpenMDW-1.1 | card: state of the art on multilingual RTEB as of 16 Jul 2026; no number extracted | Yes |
Qwen/Qwen3-Embedding-8B (Jun 2025) | 7.6B | 4,096 (32–4096) | 32K | Apache-2.0 | MTEB multilingual 70.58 (card, as of 5 Jun 2025) | Yes |
Qwen/Qwen3-Embedding-4B | 4B | 2,560 (32–2560) | 32K | Apache-2.0 | see card | Yes |
Qwen/Qwen3-Embedding-0.6B | 0.6B | 1,024 (32–1024) | 32K | Apache-2.0 | see card | Yes |
google/embeddinggemma-2 (Sep 2026) | 740M (270M text) | 768 (MRL 512 / 256 / 128) | 8,192 | Apache-2.0 | MTEB multilingual v2 61.36 (card; column assignment unclear) | Yes; text, code, image, video, audio |
jinaai/jina-embeddings-v5-text-small (Feb 2026) | 677M | 1,024 (MRL 32–1024) | 32,768 | CC BY-NC 4.0 | MMTEB 67.7; MTEB English v2 71.7 (card) | Yes, 119+ languages; non-commercial |
ibm-granite/granite-embedding-311m-multilingual-r2 (Apr 2026) | 311M | 768 | 32,768 | Apache-2.0 | not checked | Yes |
BAAI/bge-m3 | 568M | 1,024 | 8,192 | MIT | not checked | Yes; dense, sparse, multi-vector |
nomic-ai/nomic-embed-text-v2-moe (Feb 2025) | 475M (305M active) | 768 (Matryoshka) | not checked | Apache-2.0 | not checked | Yes |
No newer text-only Qwen embedder than Qwen3-Embedding was found. Qwen3-VL-Embedding-2B and 8B (Jan 2026, Apache-2.0) cover multimodal retrieval.
Medical and biomedical embedders
No medical embedder below has a verified MTEB medical score in this pass. Choose by training data and licence, then measure on your corpus.
| Model (HF repo) | Params | Dims | Max tokens | Licence | Training data and notes |
|---|---|---|---|---|---|
ncbi/MedCPT-Query-Encoder + ncbi/MedCPT-Article-Encoder (Oct 2023) | 110M (article encoder 109.5M) | not checked | 512 (article encoder sample) | Public domain (card) | Pre-trained on 255M query–article pairs from PubMed search logs. NCBI publishes precomputed PubMed article embeddings. Strongest default for PubMed and literature search. |
ncbi/MedCPT-Cross-Encoder (Nov 2023) | not checked | n/a | not checked | Public domain (card) | Reranker paired with the MedCPT retriever. |
FremyCompany/BioLORD-2023 (Nov 2023) | 109M | 768 | not checked | Model card: other (SNOMED CT and UMLS terms apply) | Fine-tuned from all-mpnet-base-v2 on BioLORD-Dataset plus LLM-generated definitions from AGCT. Clinical sentences and concepts. Check SNOMED CT and UMLS terms before shipping. |
abhinand/MedEmbed-large-v0.1 (Oct 2024) | 335M (large); base and small variants | not checked | not checked | Apache-2.0 | Medical retrieval fine-tune. Card lists MTEB medical tasks such as CUREv1 as evaluation sets. Training data not extracted. |
BMRetriever/BMRetriever-410M (Apr 2024) | 410M (1B, 2B, 7B variants) | not checked | 512 (card example) | MIT | Trained on MedRAG textbooks, PubMed, and StatPearls, bioRxiv/medRxiv, and a biomedical retrieval dataset (card). The 7B variant is 7.1B params. |
sentence-transformers/embeddinggemma-300m-medical (Sep 2025) | 303M | not checked | not checked | Apache-2.0 tag; base model's Gemma terms apply | Medical fine-tune on about 100k pairs. Sample training text includes CC BY-NC material, so review the dataset licences before use. |
lastmass/Qwen3-Embedding-Medical-0.6B (Jan 2026) | 596M | 1,024 | 512 | None declared (card: "Unknown") | Community fine-tune of Qwen3-Embedding-0.6B on huatuo_encyclopedia_qa (Chinese medical QA). Do not ship without a licence review. |
Evaluating medical retrieval:
- MIRAGE (MedRAG benchmark): 7,663 medical QA questions from five datasets, evaluated across 41 corpus, retriever, and LLM combinations (arXiv 2402.13178). The MedRAG toolkit ships BM25, Contriever, SPECTER, and MedCPT retrievers over PubMed, StatPearls, textbooks, and Wikipedia corpora. Use it for end-to-end RAG tests.
- MTEB medical retrieval tasks: the MedEmbed card evaluates on MTEB tasks such as CUREv1. List the medical tasks in your installed
mtebversion before you cite a score. - Customer-grade test: a 200–500 query golden set from the customer's own corpus, with recall@10 and nDCG@10 per candidate, plus a MedCPT baseline on PubMed-style text.
Rerankers
| Model (HF repo) | Params | Licence | Notes |
|---|---|---|---|
Qwen/Qwen3-Reranker-4B (Jun 2025) | 4.0B | Apache-2.0 | 32K context. Default quality pick. |
Qwen/Qwen3-Reranker-8B | 8.2B | Apache-2.0 | 32K context. |
Qwen/Qwen3-Reranker-0.6B | 0.6B | Apache-2.0 | 32K context. Cheap CPU/edge option. |
nvidia/Nemotron-3.5-Rerank-8B-BF16 (Oct 2026) | 8.4B | OpenMDW-1.1 | Newest reranker in this list; no benchmark extracted. |
mixedbread-ai/mxbai-rerank-large-v2 (Mar 2025) | not checked | Apache-2.0 | Used in the RAG Systems page. |
BAAI/bge-reranker-v2-m3 | 568M | Apache-2.0 | Widely used cross-encoder; cheap baseline. |
ncbi/MedCPT-Cross-Encoder | not checked | Public domain | Medical and PubMed reranking. |
Qwen/Qwen3-VL-Reranker-8B (Jan 2026) | 8.8B | Apache-2.0 | Visual document retrieval (scans, slides). |
jinaai/jina-reranker-v3 | 0.6B | CC BY-NC 4.0 | Non-commercial. |
Default RAG pairing (as of 2026-10): Qwen3-Embedding-0.6B or 4B + Qwen3-Reranker-4B, all Apache-2.0. Use Harrier-OSS-v1-27B when multilingual quality dominates and a GPU is free for the embedder, and EmbeddingGemma 2 for edge or multimodal corpora. Details are in RAG Systems and the Local RAG Lab.
Licences and what they mean for delivery
| Licence | Example models | Commercial use | Watch out for |
|---|---|---|---|
| Apache-2.0 | Qwen3.8-27B, Qwen3.6, Qwen3.5, Qwen3-Coder, Gemma 4, gpt-oss, Mistral Small 4, Mistral Large 3, Granite 4.1, Olmo 3.1, EuroLLM-22B, Qwen3-Embedding, Qwen3-Reranker, EmbeddingGemma 2 | Yes | Keep NOTICE and licence files; patent grant included. Safest default. |
| MIT | DeepSeek V4, MiMo-V2.6, GLM-5.3-Flash, GLM-5.2, Phi-4-reasoning-vision, Harrier-OSS-v1, BMRetriever, bge-m3 | Yes | Keep copyright notice. No patent grant. |
| Modified MIT (revenue exception) | Mistral Medium 3.5 | Yes below the revenue exception | Check the licence text's revenue threshold against the customer's group revenue. |
| Qwen Community License 1.0 | Qwen3.8-Flash-Next | Yes, with conditions | Products above 100M monthly active users must show the model name. Internal use is fine. |
| Qwen3.8-2.4T licence | Qwen3.8-2.4T-A95B | Check the licence file | Read the repo LICENSE before any use. |
| Kimi K3 License | Kimi K3 | Yes, with conditions | "Kimi K3" must be displayed above 100M MAU or US$20M monthly revenue. MaaS businesses with more than US$20M aggregate revenue over 12 months need an agreement. |
| GLM-5.3 licence | GLM-5.3 | Yes, with conditions | MaaS operators with more than US$10B total revenue over 12 months must pass Z.AI's security review. |
| MiniMax community licence | MiniMax-M3 | Yes, with conditions | Written authorisation needed above US$20M yearly revenue. |
| NVIDIA Nemotron Open Model License | Nemotron 3 Super | Yes | NVIDIA-specific terms; read before redistribution. |
| OpenMDW-1.1 | Nemotron 3.5 family, Nemotron 3 Ultra, Nemotron-3-Embed, Nemotron-3.5-Embed, Nemotron-3.5-Rerank | Yes | Permissive model-specific licence covering weights, data, and code. |
| Llama 4 Community License | Llama 4 Scout / Maverick | Yes, with conditions | Read the licence for attribution, naming, and regional terms before use. |
| CC BY-NC 4.0 | jina-embeddings-v5, jina-reranker-v3 | No | Research and PoC only unless a commercial licence is purchased. |
| Public domain (NCBI) | MedCPT encoders and cross-encoder | Yes | Still review the training-data terms for the exact artefact. |
| Model card: other (medical terminology) | BioLORD-2023 | Depends on SNOMED CT and UMLS terms | Confirm terminology licence coverage before shipping. |
| No licence declared | lastmass/Qwen3-Embedding-Medical-0.6B | Not cleared | Do not ship without a licence review. |
Quantized repacks, GGUF conversions, and fine-tunes inherit the base model's licence. Record the licence of the exact repo + commit you deploy in the delivery's model register. See Local AI Best Practices.
Dynamically loaded models
Most customers need more models than they have GPUs: a chat model, a coder, an embedder, a reranker, a vision model. Keep a model catalogue and load models on demand instead of pinning each one to a GPU.
| Mechanism | How it works | Use when |
|---|---|---|
| Ollama | Loads a model on first request, unloads after keep_alive (default 5 min); OLLAMA_MAX_LOADED_MODELS caps concurrent models, OLLAMA_NUM_PARALLEL sets per-model concurrency. | Dev boxes, small teams, PoCs. |
| llama-swap | Single Go proxy in front of llama-server / vLLM containers; starts the upstream for the requested model ID and stops the previous one; ttl unloads idle models; matrix runs sets of models concurrently. | One or two GPUs serving many GGUF or vLLM models through one OpenAI-compatible endpoint. |
| vLLM sleep mode | --enable-sleep-mode; level 1 offloads weights to CPU RAM and drops KV cache (fast wake), level 2 discards weights (reload from disk, minimal RAM). | Production GPU servers switching between a few large models. |
models:
qwen3.8-27b:
cmd: llama-server --port ${PORT} -m /models/Qwen3.8-27B-Q4_K_M.gguf -c 65536
ttl: 600
gemma-4-e4b:
cmd: llama-server --port ${PORT} -m /models/gemma-4-E4B-Q4_K_M.gguf
ttl: 300
Use the GGUF repo you verified for each entry. The paths above are placeholders.
Routing, autoscaling, and Kubernetes patterns for model catalogues are in Serving at Scale.
Best practices
- Shortlist from two sources, decide on one: the model card plus the HF repo's own eval results → the customer eval decides.
- Prefer official quantizations (FP8, NVFP4, MXFP4/MXFP8, vendor GGUF) over anonymous community repacks; they are tested against the vendor's own eval.
- Use the vendor's sampling parameters. Qwen3.8-27B recommends temperature 1.0, top_p 0.95, top_k 20, presence_penalty 0.0 in thinking mode, and temperature 0.7, top_p 0.8, top_k 20, presence_penalty 1.5 in non-thinking mode.
- Pick per role, not one model for everything: a small fast model for routing and extraction, a mid model for answers, an embedder and reranker for retrieval.
- Pin the revision (HF commit SHA) of every model in IaC; see Best Practices.
Pitfalls to design around
- Read the
LICENSEfile, not the blog post. Several 2026 releases (Qwen3.8-Flash-Next, GLM-5.3, Kimi K3) ship custom licences. - Budget memory for MoE total parameters. A 35B-A3B model computes like 3B but needs memory for 35B.
- Treat leaderboard positions as perishable. Model-card scores go stale; check the live board before you rank.
- Plan for reasoning-token cost. Turn thinking off where the task does not benefit.
- Validate every target language separately. English benchmark wins do not transfer automatically.
- Verify model names before you quote them. Names such as "Phi-5", "gpt-oss-2", and "Llama 5" have no public HF repo as of 2026-10-18 (the HF API returned 401 for each repo id).
Definition of done (model selection)
- Use cases, languages, context length, concurrency and latency targets written down
- Licence of every artefact (LLM, embedder, reranker, quant) checked against the customer's business model
- Hardware fit calculated for weights + KV cache at target context and concurrency
- 2–4 candidates evaluated on customer data in each target language
- Winner pinned by HF repo + commit SHA in IaC, with sampling parameters
- Re-evaluation date set (≤ 3 months)
How to keep this page updated
Review monthly; update the "as of" date when you do.
| Source | What to watch |
|---|---|
| Hugging Face org pages (Qwen, google, mistralai, deepseek-ai, moonshotai, zai-org, XiaomiMiMo, MiniMaxAI, nvidia, ibm-granite, allenai) | New repos, LICENSE changes |
| MTEB leaderboard | Embedders and rerankers |
| Ollama library | Tags and sizes actually pullable |
| Vendor changelogs (Mistral, Gemma releases, NVIDIA Nemotron) | Releases, deprecations, licence changes |
| vLLM releases | Day-0 model support, new quant formats |
Sources
Fetched in this review (2026-10-18):
- Hugging Face model cards and metadata: Qwen3.8-27B · Qwen3.8-Flash-Next · Qwen3.8-2.4T-A95B · Qwen3.6-27B · Qwen3.6-35B-A3B · Qwen3.5-9B · Qwen3.5-4B · Qwen3.5-2B · Qwen3.5-0.8B · Qwen3-Coder-30B-A3B-Instruct · Qwen3-Coder-Next · Qwen3-Embedding-8B · Qwen3-Embedding-0.6B · Qwen3-Reranker-4B · Qwen3-Reranker-8B · Qwen3-Reranker-0.6B · Qwen3-VL-Reranker-8B · Qwen3-VL-Embedding-8B
- Qwen licence text: Qwen Community License 1.0 (Flash-Next)
- GLM: GLM-5.3 · GLM-5.3 licence · GLM-5.3-Flash
- Kimi: Kimi-K3 · Kimi K3 licence · Kimi-K2.7-Code
- DeepSeek: DeepSeek-V4-Pro · DeepSeek-V4-Flash · DeepSeek-V4.1-Flash
- MiniMax: MiniMax-M3 · MiniMax licence
- MiMo: MiMo-V2.6-Pro-MOPD · MiMo-V2.6-Flash-MOPD
- Gemma 4: gemma-4-26B-A4B-it · gemma-4-31B-it · EmbeddingGemma 2 · EmbeddingGemma 300M
- OpenAI: gpt-oss-120b · gpt-oss-20b
- Mistral: Mistral-Small-4-119B-2603 · Mistral-Medium-3.5-128B · Mistral Large 4 announcement · Ministral-3-14B-Instruct-2512 · Devstral-Small-2-24B-Instruct-2512
- NVIDIA: Nemotron-3.5-Lightning-30B-A3B-BF16 · Nemotron-3.5-Super-VL-120B-A12B-BF16 · Nemotron-3.5-Bolt-74B-A9B-BF16 · Nemotron-3-Super-120B-A12B-BF16 · Nemotron-3-Ultra-550B-A55B-BF16 · Nemotron-3.5-Rerank-8B-BF16 · Nemotron-3.5-Embed-8B-BF16 · Nemotron-3-Embed-8B-BF16
- Embedders and rerankers: harrier-oss-v1-27b · harrier-oss-v1-0.6b · jina-embeddings-v5-text-small · jina-reranker-v3 · granite-embedding-311m-multilingual-r2 · BAAI/bge-m3 · nomic-embed-text-v2-moe · bge-reranker-v2-m3 · mxbai-rerank-large-v2
- Medical: MedCPT Article Encoder · MedCPT Query Encoder · MedCPT Cross-Encoder · BioLORD-2023 · MedEmbed-large-v0.1 · BMRetriever-410M · embeddinggemma-300m-medical · Qwen3-Embedding-Medical-0.6B · MIRAGE / MedRAG paper (arXiv 2402.13178) · MedRAG toolkit
- Other families: Granite 4.1 30B · Granite 4.1 8B · Olmo-3.1-32B-Instruct · Phi-4-reasoning-vision-15B · EuroLLM-22B-Instruct-2512 · Llama-4-Scout-17B-16E-Instruct
- Ollama: qwen3.8 · qwen3.5
Dynamic loading and infrastructure (not re-fetched in this pass):