Ugrás a fő tartalomhoz

Open Model Landscape (as of 2026-10)

Basic Advanced

Service: On-Premise LLM Deployment · Sovereign & Private AI · RAG Systems on Private Infrastructure · Sovereign AI on STACKIT · Typical shape: model shortlist + eval run in the first week of every PoC; quarterly refresh for running platforms

Snapshot date

Every model, size, date, and licence on this page was checked against the Hugging Face API and model cards, Ollama library pages, or vendor announcements on 18 October 2026. Benchmark scores are vendor-reported in model cards unless marked otherwise. Open-weight releases ship monthly; re-verify against the LICENSE file of the exact repo before you commit a customer to a model. The update procedure is at the end of this page.

What we deliver​

  • A shortlist of 2–4 candidate models per use case, justified against the selection criteria below.
  • A hardware fit calculation (weights + KV cache at target context and concurrency) for the customer's GPUs. See Inference Engineering.
  • A licence review for every model artefact (LLM, embedder, reranker, quantized variant) that goes to production.
  • An eval result on the customer's own data in the languages they use. Leaderboards narrow the field; the customer eval picks the winner.

How to choose a model​

Work through the criteria in this order. The first three remove most candidates.

#CriterionQuestion to answerHow to check
1LicenceCan the customer use it commercially in their setting (internal tool, SaaS, MaaS)? Are there revenue/MAU thresholds or regional restrictions?LICENSE file in the HF repo, not the press release. See the licence table.
2Hardware fitDo weights + KV cache fit at the target context and concurrency?Weights ≈ total params × bytes per param (BF16 = 2, FP8 = 1, 4-bit ≈ 0.55). MoE models need all experts in memory; only compute scales with active params.
3TaskChat, coding agent, RAG answerer, extraction, vision, tool use?Model card benchmarks for the task family; the customer's own eval set.
4LanguagesDoes it handle the customer's languages at the needed quality?Run an eval set in each target language. Check tokenizer fertility: more tokens per word means higher latency and cost.
5Context lengthLongest realistic prompt (RAG chunks + history + tool results)?Native vs. extended context on the card. Extended context can reduce short-text quality.
6Tool callingNative tool-call format supported by the serving engine's parser?vLLM/SGLang tool parser for the model; test with your real tool schemas.
7Reasoning modesNeed a thinking mode? Can it be switched off per request for latency?Chat template flags and reasoning_effort on the card. Reasoning multiplies output tokens.
8EcosystemvLLM / SGLang / llama.cpp / Ollama support, official FP8/NVFP4/GGUF quants, speculative-decoding heads?Model card "Quickstart" + engine release notes.

Rules of thumb (as of 2026-10):

  • Small dense vs. small MoE: a 30B-class MoE with ~3B active (Nemotron 3.5 Lightning 30B-A3B, Qwen3-Coder-30B-A3B, Gemma 4 26B A4B) gives much higher throughput than a 27–31B dense model, at the cost of memory for all experts. Dense 27–31B models (Qwen3.8-27B, Gemma 4 31B) are simpler to fine-tune and quantize.
  • Thinking is a cost lever: reasoning-on models emit many more output tokens. Budget them explicitly and disable thinking for extraction and classification.
  • Frontier open MoE is not self-hostable everywhere: 750B–2.8T models need 8+ datacentre GPUs or multi-node. For most private deployments, the 27–128B band is the sweet spot.

Top-tier open-weight models (as of 2026-10)​

Verified against the HF repo (safetensors total, licence tag, README). Min hardware assumes 4-bit weights (≈ 0.55 bytes per param) unless noted, weights only; add KV cache for your context.

Model (HF repo)OrgTotal / active paramsContextLicenceBest forMin hardware
Qwen/Qwen3.8-27B (Aug 2026)Alibaba Qwen27.8B dense262,144 native, to 1MApache-2.0General assistant, vision, coding1 × 24 GB (Ollama qwen3.8:27b is 18 GB)
Qwen/Qwen3-Coder-30B-A3B-Instruct (Jul 2025)Alibaba Qwen30.5B / ~3.3B262,144 nativeApache-2.0Coding agent on one GPU; newest ~30B Qwen coder1 × 24 GB (~17 GB)
Qwen/Qwen3-Coder-Next (Jan 2026)Alibaba Qwen80B / 3B262,144 nativeApache-2.0Newer coder for agent loops; not ~30B1 × 48 GB (~45 GB)
google/gemma-4-31B-it (Mar 2026)Google30.7B dense256K (medium models)Apache-2.0RAG answerer, vision1 × 24 GB (~17 GB)
google/gemma-4-26B-A4B-it (Mar 2026)Google25.8B / ~4B256K (medium models)Apache-2.0Fast RAG answerer1 × 24 GB (~14 GB)
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 (Aug 2026)NVIDIA30B / 3Bnot checkedOpenMDW-1.1Tool calling, agent execution1 × 24 GB (~17 GB); NVFP4 repo also published
openai/gpt-oss-120b (Aug 2025)OpenAI117B / 5.1Bnot checkedApache-2.0Reasoning + tool use; no successor found1 × 80 GB (MXFP4 weights)
mistralai/Mistral-Small-4-119B-2603 (2603 build)Mistral119B / 6.5B256KApache-2.0General + reasoning + vision; EU vendor1 × 80 GB (~65 GB); NVFP4 repo also published
mistralai/Mistral-Medium-3.5-128B (Mar 2026)Mistral128B densenot checkedModified MIT (revenue exception)Coding agent1–2 × 80 GB (~70 GB at 4-bit)
zai-org/GLM-5.3 (Aug 2026)Z.ai753B / not stated on card1M (eval setting)Custom GLM-5.3 licenceFlagship agentic coding; reasoning_effort low/high/maxMulti-node; ~750 GB at FP8
zai-org/GLM-5.3-Flash (Aug 2026)Z.ai320B / 18Bnot stated on cardMITSmaller GLM; natively multimodalMulti-GPU; ~180 GB at 4-bit
moonshotai/Kimi-K3 (Jun 2026)Moonshot2.8T / 104B1MKimi K3 LicenseFrontier coding and agentic work; native visionMulti-node; ~1.5 TB at 4-bit
deepseek-ai/DeepSeek-V4.1-Flash (Sep 2026)DeepSeek552B backbone (763B in files); 8B prefill / 16B decode active1MMITLong-context RAG, low cost per token; image inputMulti-node
deepseek-ai/DeepSeek-V4-Flash (Apr 2026)DeepSeek284B / 13B1MMITLong-context agents, textMulti-GPU server (~290 GB files)
deepseek-ai/DeepSeek-V4-Pro (preview, Apr 2026)DeepSeek1.6T / 49B1MMITFrontier generalMulti-node
XiaomiMiMo/MiMo-V2.6-Pro-MOPD (Sep 2026)Xiaomi1.02T / not checkednot checkedMITFrontier general, agenticMulti-node
MiniMaxAI/MiniMax-M3 (Jun 2026)MiniMax428B / 23B1MMiniMax community licenceLong-context agents; native image and video input~240 GB at 4-bit; 4 × 80 GB class

Not yet shortlistable: Mistral Large 4 (announced 6 Oct 2026: 1T total / 52B active, multimodal). Public API preview only; Mistral says weights follow by end of October 2026, and no HF repo or licence was found as of 2026-10-18. Re-check before the next review.

Which to pick (as of 2026-10):

NeedFirst pickNext optionWhy
General assistantQwen3.8-27BGemma 4 31BOne GPU, Apache-2.0, native vision
Coding agentQwen3-Coder-30B-A3B-Instruct (1 GPU)Qwen3-Coder-Next (1–2 GPUs); Kimi K3 or GLM-5.3 (frontier)Low active params for agent loops
RAG answererGemma 4 26B A4BQwen3.8-27B with thinking offFast MoE, 256K context
Small / edgeQwen3.5-0.8B / 2B / 4B / 9BGemma 4 E2B / E4BSmallest current generation
Tool-calling agentNemotron 3.5 Lightning 30B-A3Bgpt-oss-120b (1 × 80 GB)Built for agent execution; configurable reasoning

Current open-weight families​

Latest versions only. Dates are the HF createdAt or the vendor announcement. Rows were verified against HF; older generations are dropped unless they are still the best choice for a niche.

Family (vendor)Latest open weights (as of 2026-10)Sizes (total / active)LicenceNotes
Qwen (Alibaba)Qwen3.8-27B (Aug 2026); Qwen3.8-Flash-Next (Aug 2026); Qwen3.8-2.4T-A95B (Aug 2026); Qwen3.6-27B and 35B-A3B (Apr 2026); Qwen3.5 0.8B / 2B / 4B / 9B (Feb 2026)27.8B dense; ~180B (Flash-Next, active not verified); 2.45T / 95B; 27B; 35B / 3B; 0.8–9BApache-2.0 except Flash-Next (Qwen Community License 1.0) and 2.4T (Qwen licence)262K native context (to 1M). Thinking on by default. No Qwen3.8 small sizes found on HF or Ollama. No Qwen 4 open weights found.
Gemma (Google)Gemma 4 E2B, E4B, 12B (May 2026), 26B A4B, 31B (Mar 2026)E2B 5.1B and E4B 8B with embeddings (2.3B / 4.5B effective); 12B 11.95B; 26B A4B 25.8B; 31B 30.7BApache-2.0128K (small) / 256K (medium) context; thinking toggle; native function calling.
gpt-oss (OpenAI)gpt-oss-120b, gpt-oss-20b (Aug 2025); gpt-oss-safeguard-120b / 20b (Sep 2025)117B / 5.1B; 21B / 3.6BApache-2.0No successor found on HF as of 2026-10.
MistralMistral Small 4 (2603 build); Mistral Medium 3.5 (Mar 2026); Mistral Large 3 (Dec 2025); Devstral Small 2 (Dec 2025); Ministral 3 3B / 8B / 14B (Dec 2025)119B / 6.5B; 128B dense; 675B total; 24B; 3–14BApache-2.0 (Small 4, Large 3, Devstral Small 2, Ministral 3); Modified MIT (Medium 3.5)Mistral Large 4 is announced but not on HF (see above).
DeepSeekDeepSeek-V4-Pro (preview), V4-Flash (Apr 2026), V4.1-Flash (Sep 2026)1.6T / 49B; 284B / 13B; 552B backbone / 8B–16BMIT1M context. V4.1-Flash adds image input.
Kimi (Moonshot)Kimi K3 (Jun 2026); Kimi K2.7-Code (Jun 2026); Kimi K2.6 (Apr 2026)2.8T / 104B; 1.03T total (K2.7-Code, active not verified)Kimi K3 License (custom, MIT-derived); other Kimi K2 licencesNative vision; 1M context.
GLM (Z.ai)GLM-5.3 (Aug 2026); GLM-5.3-Flash (Aug 2026); GLM-5.2 (Jun 2026); GLM-5.1 (Apr 2026)753B (active not on card); 320B / 18BGLM-5.3: custom glm-5.3 licence; GLM-5.3-Flash, GLM-5.2, GLM-5.1: MITHybrid sparse + linear attention on Flash. Thinking effort via reasoning_effort.
MiMo (Xiaomi)MiMo-V2.6-Pro-MOPD and Flash-MOPD (Sep 2026)1.02T total; 311B totalMITFlash-MOPD is the smaller variant.
MiniMaxMiniMax-M3 (Jun 2026)428B / 23BMiniMax community licence1M context; written authorisation needed above US$20M yearly revenue.
NVIDIA NemotronNemotron 3.5 Lightning 30B-A3B (Aug 2026); Nemotron 3.5 Bolt 74B-A9B (Oct 2026); Nemotron 3.5 Super VL 120B-A12B (Oct 2026); Nemotron 3 Super 120B-A12B (Mar 2026); Nemotron 3 Ultra 550B-A55B (Jun 2026)30B / 3B; 74B / 9B; 120B / 12B; 120B / 12B; 550B / 55BOpenMDW-1.1 (3.5 family, Ultra); NVIDIA Nemotron Open Model License (Nemotron 3 Super)3.5 Super VL is multimodal, 1M context. Check each card's licence.
Granite (IBM)Granite 4.1 8B (8.8B) and 30B (28.9B), Apr 20268B and 30B denseApache-2.0Enterprise-friendly provenance.
Olmo (Ai2)Olmo 3.1 32B Instruct (Dec 2025)32.2B denseApache-2.0Fully open: weights, data, and training recipe. Reference when a customer needs training-data transparency.
Phi (Microsoft)Phi-4-reasoning-vision-15B (Jan 2026)15.1B denseMITNo Phi-5 repo found on HF as of 2026-10.
Llama (Meta)Llama 4 Scout / Maverick (Apr 2025)Scout 108.6B total (17B active)Llama 4 Community LicenseNot re-verified beyond the Scout HF repo. Check the EU multimodal terms in the licence before use.
EuroLLM (EU consortium)EuroLLM-22B-Instruct-2512 (Dec 2025)22.6B denseApache-2.0Covers the 24 official EU languages; 32K context.

Picks are starting points for the shortlist, ordered by our default preference. Memory figures are weights at 4-bit unless noted.

Tier 1: laptop / edge (≤ 9B; 8–32 GB unified memory or 8–16 GB VRAM)​

Use caseFirst pickAlternativesWhy
Chat / assistantQwen3.5-9BGemma 4 E4B, Granite 4.1 8BApache-2.0, 256K context; Ollama qwen3.5:9b is 6.6–7.6 GB.
CodingQwen3.5-9Bgpt-oss-20b (21B / 3.6B active)Small active footprint for fast local loops.
RAG answererGranite 4.1 8BQwen3.5-4B, Ministral 3 8BApache-2.0. Check the 8B card for context length.
Agent / tool usegpt-oss-20bQwen3.5-9BTrained for tool use with configurable reasoning effort.
VisionGemma 4 E4BQwen3.5-4B, Ministral 3 8BNative image input at phone-class size.
Small / edgeGemma 4 E2BQwen3.5-0.8B / 2BGemma 4 E2B is about 2.5–3 GB at 4-bit (estimate from 5.1B params).

Tier 2: single workstation GPU (14–35B; 24–48 GB VRAM, e.g. RTX 4090/5090, L40S, RTX PRO 6000)​

Use caseFirst pickAlternativesWhy
Chat / assistantQwen3.8-27BGemma 4 31B, Ministral 3 14BCurrent Qwen generation; 18 GB at Ollama default quant.
Coding agentQwen3-Coder-30B-A3B-InstructQwen3.8-27B, Devstral Small 2 (24B)Newest ~30B Qwen coder; 3B active.
RAG answererGemma 4 26B A4BQwen3.8-27B (thinking off), Granite 4.1 30BFast MoE, Apache-2.0.
Agent / tool useNemotron 3.5 Lightning 30B-A3BQwen3.6-35B-A3B (Apache-2.0)Built for agent execution.
VisionQwen3.8-27BGemma 4 31B, Phi-4-reasoning-vision-15B (MIT)Native image input.
EU-provenance requirementEuroLLM-22B-Instruct-2512Ministral 3 14B, Olmo 3.1 32BEuropean training, documented data.

Tier 3: single server (70–130B or mid-size MoE; 1–4 × 80–141 GB GPUs)​

Use caseFirst pickAlternativesWhy
Chat / assistantMistral Small 4 (119B / 6.5B)gpt-oss-120b, Qwen3.5-122B-A10BApache-2.0, reasoning + vision + coding, EU vendor.
Coding agentMistral Medium 3.5 (128B dense)Qwen3-Coder-Next (80B / 3B), gpt-oss-120bModified MIT. Check the revenue exception first.
RAG answerergpt-oss-120bMistral Small 4, Nemotron 3 SuperFits one 80 GB GPU; cheap per token.
Agent / tool useNemotron 3 Super (120B / 12B)gpt-oss-120b, Mistral Small 41M context; check the licence (NVIDIA Nemotron Open Model License).
VisionMistral Small 4Qwen3.5-122B-A10BNative image input.

Tier 4: multi-GPU / multi-node frontier MoE (8+ × H100/H200/B200)​

Use caseFirst pickAlternativesWhy
General frontierDeepSeek-V4-Pro (1.6T / 49B)MiMo-V2.6-Pro, GLM-5.3, Kimi K3MIT licence; 1M context.
Coding agentKimi K3 (2.8T / 104B)GLM-5.3, MiniMax-M3Check the Kimi K3 licence thresholds before delivery.
Long-context RAG / costDeepSeek-V4-Flash (284B / 13B)DeepSeek-V4.1-Flash, GLM-5.3-FlashMIT, 1M context, low active params.
EU-built frontierMistral Large 3 (675B total, Apache-2.0)Mistral Large 4 when weights landEuropean vendor; verify the Large 4 licence on release.

Embedding and reranker models​

The live MTEB leaderboard is a JavaScript app and could not be read in this pass. Scores below come from model cards (vendor-reported, snapshot date on the card). Re-check the leaderboard before you shortlist.

Embedders​

Model (HF repo)ParamsDimsMax tokensLicenceScore (source)Multilingual
microsoft/harrier-oss-v1-27b (Mar 2026)27.0B5,37632,768MITMTEB v2 multilingual 74.3 (card)Yes
microsoft/harrier-oss-v1-0.6b0.6B1,02432,768MITMTEB v2 multilingual 69.0 (card)Yes
nvidia/Nemotron-3.5-Embed-8B-BF16 (Oct 2026)8.4B4,096 (truncatable)32,768OpenMDW-1.1not extracted from cardYes; text and page images
nvidia/Nemotron-3-Embed-8B-BF16 (Jul 2026)8.0B4,096 (truncatable)32,768OpenMDW-1.1card: state of the art on multilingual RTEB as of 16 Jul 2026; no number extractedYes
Qwen/Qwen3-Embedding-8B (Jun 2025)7.6B4,096 (32–4096)32KApache-2.0MTEB multilingual 70.58 (card, as of 5 Jun 2025)Yes
Qwen/Qwen3-Embedding-4B4B2,560 (32–2560)32KApache-2.0see cardYes
Qwen/Qwen3-Embedding-0.6B0.6B1,024 (32–1024)32KApache-2.0see cardYes
google/embeddinggemma-2 (Sep 2026)740M (270M text)768 (MRL 512 / 256 / 128)8,192Apache-2.0MTEB multilingual v2 61.36 (card; column assignment unclear)Yes; text, code, image, video, audio
jinaai/jina-embeddings-v5-text-small (Feb 2026)677M1,024 (MRL 32–1024)32,768CC BY-NC 4.0MMTEB 67.7; MTEB English v2 71.7 (card)Yes, 119+ languages; non-commercial
ibm-granite/granite-embedding-311m-multilingual-r2 (Apr 2026)311M76832,768Apache-2.0not checkedYes
BAAI/bge-m3568M1,0248,192MITnot checkedYes; dense, sparse, multi-vector
nomic-ai/nomic-embed-text-v2-moe (Feb 2025)475M (305M active)768 (Matryoshka)not checkedApache-2.0not checkedYes

No newer text-only Qwen embedder than Qwen3-Embedding was found. Qwen3-VL-Embedding-2B and 8B (Jan 2026, Apache-2.0) cover multimodal retrieval.

Medical and biomedical embedders​

No medical embedder below has a verified MTEB medical score in this pass. Choose by training data and licence, then measure on your corpus.

Model (HF repo)ParamsDimsMax tokensLicenceTraining data and notes
ncbi/MedCPT-Query-Encoder + ncbi/MedCPT-Article-Encoder (Oct 2023)110M (article encoder 109.5M)not checked512 (article encoder sample)Public domain (card)Pre-trained on 255M query–article pairs from PubMed search logs. NCBI publishes precomputed PubMed article embeddings. Strongest default for PubMed and literature search.
ncbi/MedCPT-Cross-Encoder (Nov 2023)not checkedn/anot checkedPublic domain (card)Reranker paired with the MedCPT retriever.
FremyCompany/BioLORD-2023 (Nov 2023)109M768not checkedModel card: other (SNOMED CT and UMLS terms apply)Fine-tuned from all-mpnet-base-v2 on BioLORD-Dataset plus LLM-generated definitions from AGCT. Clinical sentences and concepts. Check SNOMED CT and UMLS terms before shipping.
abhinand/MedEmbed-large-v0.1 (Oct 2024)335M (large); base and small variantsnot checkednot checkedApache-2.0Medical retrieval fine-tune. Card lists MTEB medical tasks such as CUREv1 as evaluation sets. Training data not extracted.
BMRetriever/BMRetriever-410M (Apr 2024)410M (1B, 2B, 7B variants)not checked512 (card example)MITTrained on MedRAG textbooks, PubMed, and StatPearls, bioRxiv/medRxiv, and a biomedical retrieval dataset (card). The 7B variant is 7.1B params.
sentence-transformers/embeddinggemma-300m-medical (Sep 2025)303Mnot checkednot checkedApache-2.0 tag; base model's Gemma terms applyMedical fine-tune on about 100k pairs. Sample training text includes CC BY-NC material, so review the dataset licences before use.
lastmass/Qwen3-Embedding-Medical-0.6B (Jan 2026)596M1,024512None declared (card: "Unknown")Community fine-tune of Qwen3-Embedding-0.6B on huatuo_encyclopedia_qa (Chinese medical QA). Do not ship without a licence review.

Evaluating medical retrieval:

  • MIRAGE (MedRAG benchmark): 7,663 medical QA questions from five datasets, evaluated across 41 corpus, retriever, and LLM combinations (arXiv 2402.13178). The MedRAG toolkit ships BM25, Contriever, SPECTER, and MedCPT retrievers over PubMed, StatPearls, textbooks, and Wikipedia corpora. Use it for end-to-end RAG tests.
  • MTEB medical retrieval tasks: the MedEmbed card evaluates on MTEB tasks such as CUREv1. List the medical tasks in your installed mteb version before you cite a score.
  • Customer-grade test: a 200–500 query golden set from the customer's own corpus, with recall@10 and nDCG@10 per candidate, plus a MedCPT baseline on PubMed-style text.

Rerankers​

Model (HF repo)ParamsLicenceNotes
Qwen/Qwen3-Reranker-4B (Jun 2025)4.0BApache-2.032K context. Default quality pick.
Qwen/Qwen3-Reranker-8B8.2BApache-2.032K context.
Qwen/Qwen3-Reranker-0.6B0.6BApache-2.032K context. Cheap CPU/edge option.
nvidia/Nemotron-3.5-Rerank-8B-BF16 (Oct 2026)8.4BOpenMDW-1.1Newest reranker in this list; no benchmark extracted.
mixedbread-ai/mxbai-rerank-large-v2 (Mar 2025)not checkedApache-2.0Used in the RAG Systems page.
BAAI/bge-reranker-v2-m3568MApache-2.0Widely used cross-encoder; cheap baseline.
ncbi/MedCPT-Cross-Encodernot checkedPublic domainMedical and PubMed reranking.
Qwen/Qwen3-VL-Reranker-8B (Jan 2026)8.8BApache-2.0Visual document retrieval (scans, slides).
jinaai/jina-reranker-v30.6BCC BY-NC 4.0Non-commercial.

Default RAG pairing (as of 2026-10): Qwen3-Embedding-0.6B or 4B + Qwen3-Reranker-4B, all Apache-2.0. Use Harrier-OSS-v1-27B when multilingual quality dominates and a GPU is free for the embedder, and EmbeddingGemma 2 for edge or multimodal corpora. Details are in RAG Systems and the Local RAG Lab.

Licences and what they mean for delivery​

LicenceExample modelsCommercial useWatch out for
Apache-2.0Qwen3.8-27B, Qwen3.6, Qwen3.5, Qwen3-Coder, Gemma 4, gpt-oss, Mistral Small 4, Mistral Large 3, Granite 4.1, Olmo 3.1, EuroLLM-22B, Qwen3-Embedding, Qwen3-Reranker, EmbeddingGemma 2YesKeep NOTICE and licence files; patent grant included. Safest default.
MITDeepSeek V4, MiMo-V2.6, GLM-5.3-Flash, GLM-5.2, Phi-4-reasoning-vision, Harrier-OSS-v1, BMRetriever, bge-m3YesKeep copyright notice. No patent grant.
Modified MIT (revenue exception)Mistral Medium 3.5Yes below the revenue exceptionCheck the licence text's revenue threshold against the customer's group revenue.
Qwen Community License 1.0Qwen3.8-Flash-NextYes, with conditionsProducts above 100M monthly active users must show the model name. Internal use is fine.
Qwen3.8-2.4T licenceQwen3.8-2.4T-A95BCheck the licence fileRead the repo LICENSE before any use.
Kimi K3 LicenseKimi K3Yes, with conditions"Kimi K3" must be displayed above 100M MAU or US$20M monthly revenue. MaaS businesses with more than US$20M aggregate revenue over 12 months need an agreement.
GLM-5.3 licenceGLM-5.3Yes, with conditionsMaaS operators with more than US$10B total revenue over 12 months must pass Z.AI's security review.
MiniMax community licenceMiniMax-M3Yes, with conditionsWritten authorisation needed above US$20M yearly revenue.
NVIDIA Nemotron Open Model LicenseNemotron 3 SuperYesNVIDIA-specific terms; read before redistribution.
OpenMDW-1.1Nemotron 3.5 family, Nemotron 3 Ultra, Nemotron-3-Embed, Nemotron-3.5-Embed, Nemotron-3.5-RerankYesPermissive model-specific licence covering weights, data, and code.
Llama 4 Community LicenseLlama 4 Scout / MaverickYes, with conditionsRead the licence for attribution, naming, and regional terms before use.
CC BY-NC 4.0jina-embeddings-v5, jina-reranker-v3NoResearch and PoC only unless a commercial licence is purchased.
Public domain (NCBI)MedCPT encoders and cross-encoderYesStill review the training-data terms for the exact artefact.
Model card: other (medical terminology)BioLORD-2023Depends on SNOMED CT and UMLS termsConfirm terminology licence coverage before shipping.
No licence declaredlastmass/Qwen3-Embedding-Medical-0.6BNot clearedDo not ship without a licence review.
Licence check is per artefact

Quantized repacks, GGUF conversions, and fine-tunes inherit the base model's licence. Record the licence of the exact repo + commit you deploy in the delivery's model register. See Local AI Best Practices.

Dynamically loaded models​

Most customers need more models than they have GPUs: a chat model, a coder, an embedder, a reranker, a vision model. Keep a model catalogue and load models on demand instead of pinning each one to a GPU.

MechanismHow it worksUse when
OllamaLoads a model on first request, unloads after keep_alive (default 5 min); OLLAMA_MAX_LOADED_MODELS caps concurrent models, OLLAMA_NUM_PARALLEL sets per-model concurrency.Dev boxes, small teams, PoCs.
llama-swapSingle Go proxy in front of llama-server / vLLM containers; starts the upstream for the requested model ID and stops the previous one; ttl unloads idle models; matrix runs sets of models concurrently.One or two GPUs serving many GGUF or vLLM models through one OpenAI-compatible endpoint.
vLLM sleep mode--enable-sleep-mode; level 1 offloads weights to CPU RAM and drops KV cache (fast wake), level 2 discards weights (reload from disk, minimal RAM).Production GPU servers switching between a few large models.
llama-swap config.yaml (minimal)
models:
qwen3.8-27b:
cmd: llama-server --port ${PORT} -m /models/Qwen3.8-27B-Q4_K_M.gguf -c 65536
ttl: 600
gemma-4-e4b:
cmd: llama-server --port ${PORT} -m /models/gemma-4-E4B-Q4_K_M.gguf
ttl: 300

Use the GGUF repo you verified for each entry. The paths above are placeholders.

Routing, autoscaling, and Kubernetes patterns for model catalogues are in Serving at Scale.

Best practices​

  • Shortlist from two sources, decide on one: the model card plus the HF repo's own eval results → the customer eval decides.
  • Prefer official quantizations (FP8, NVFP4, MXFP4/MXFP8, vendor GGUF) over anonymous community repacks; they are tested against the vendor's own eval.
  • Use the vendor's sampling parameters. Qwen3.8-27B recommends temperature 1.0, top_p 0.95, top_k 20, presence_penalty 0.0 in thinking mode, and temperature 0.7, top_p 0.8, top_k 20, presence_penalty 1.5 in non-thinking mode.
  • Pick per role, not one model for everything: a small fast model for routing and extraction, a mid model for answers, an embedder and reranker for retrieval.
  • Pin the revision (HF commit SHA) of every model in IaC; see Best Practices.

Pitfalls to design around​

  • Read the LICENSE file, not the blog post. Several 2026 releases (Qwen3.8-Flash-Next, GLM-5.3, Kimi K3) ship custom licences.
  • Budget memory for MoE total parameters. A 35B-A3B model computes like 3B but needs memory for 35B.
  • Treat leaderboard positions as perishable. Model-card scores go stale; check the live board before you rank.
  • Plan for reasoning-token cost. Turn thinking off where the task does not benefit.
  • Validate every target language separately. English benchmark wins do not transfer automatically.
  • Verify model names before you quote them. Names such as "Phi-5", "gpt-oss-2", and "Llama 5" have no public HF repo as of 2026-10-18 (the HF API returned 401 for each repo id).

Definition of done (model selection)​

  • Use cases, languages, context length, concurrency and latency targets written down
  • Licence of every artefact (LLM, embedder, reranker, quant) checked against the customer's business model
  • Hardware fit calculated for weights + KV cache at target context and concurrency
  • 2–4 candidates evaluated on customer data in each target language
  • Winner pinned by HF repo + commit SHA in IaC, with sampling parameters
  • Re-evaluation date set (≤ 3 months)

How to keep this page updated​

Review monthly; update the "as of" date when you do.

SourceWhat to watch
Hugging Face org pages (Qwen, google, mistralai, deepseek-ai, moonshotai, zai-org, XiaomiMiMo, MiniMaxAI, nvidia, ibm-granite, allenai)New repos, LICENSE changes
MTEB leaderboardEmbedders and rerankers
Ollama libraryTags and sizes actually pullable
Vendor changelogs (Mistral, Gemma releases, NVIDIA Nemotron)Releases, deprecations, licence changes
vLLM releasesDay-0 model support, new quant formats

Sources​

Fetched in this review (2026-10-18):

Dynamic loading and infrastructure (not re-fetched in this pass):