Kimi K3 Goes Open-Weight: How 2.8T Activates Just 104B Parameters
Kimi K3's full weights and 47-page technical report are out. We unpack its 2.8T MoE, 104B active parameters, 1M context, agent RL, deployment demands, and license limits.
Published 52 days ago. Content may be outdated.
Kimi K3 has delivered on its promise: the full model weights, model card, and 47-page technical report are now public.
The real story is not the headline figure of 2.8 trillion parameters. It is how Moonshot AI made a 3T-class model trainable and deployable while supporting million-token, long-horizon agents. The answer spans three dimensions: KDA across sequence length, AttnRes across network depth, and Stable LatentMoE across model width.
The short version: Kimi K3 is one of the most ambitious open-weight models available, but downloadable weights do not mean it will run on a few GPUs, nor does its custom license come without conditions.
30-Second Summary
| Item | Kimi K3 |
|---|---|
| Total parameters | 2.8T (2.78T in the technical report) |
| Active parameters per token | 104B (104.2B precisely) |
| Network depth | 93 layers |
| Expert structure | 896 routed experts, 16 selected per token, plus 2 shared experts |
| Attention composition | 69 KDA layers + 24 Gated MLA layers |
| Context window | 1,048,576 tokens |
| Vision encoder | MoonViT-V2, 401M parameters, 27 layers |
| Native modalities | Text, image, and video understanding |
| Weight format | MXFP4 MoE expert weights, MXFP8 activations |
| Recommended deployment | vLLM, SGLang, or TokenSpeed; a supernode with at least 64 accelerators |
The parameter count is easy to misread. The 2.8T figure is total model capacity; 104B is the parameter count activated for each token. That active footprint is still larger than most open models. K3’s weight release is therefore aimed first at research labs, cloud providers, and infrastructure teams with large clusters, not desktop deployment.
The Architecture Scales Sequence, Depth, and Width
Moonshot reports roughly a 2.5x improvement in overall scaling efficiency over Kimi K2. That gain does not come from one isolated trick. It comes from changes along three axes.
1. KDA Handles Long Sequences While MLA Preserves Global Attention
Each main K3 block contains three Kimi Delta Attention (KDA) layers followed by one Gated MLA layer. One additional Gated MLA layer closes the backbone, producing a 69:24 hybrid attention mix.
KDA follows the linear-attention path. It replaces a full KV cache that grows with sequence length with a fixed-size recurrent state, reducing memory pressure at million-token scale. Periodic Gated MLA layers retain global attention capacity and compensate for the limitations of purely linear attention.
The tradeoff is practical: most layers optimize long-context efficiency, while a smaller number preserve high-capacity global interaction. K3 also uses NoPE. Positional information is represented implicitly through KDA’s recurrent gating, so extending the model to 1M tokens does not require RoPE interpolation or rescaling.
2. AttnRes Lets Deep Layers Retrieve Earlier Representations
A conventional Transformer residual stream works much like a relay: each layer mainly receives the output of the layer before it. K3’s Attention Residuals (AttnRes) allow layers to selectively retrieve representations from the embedding and earlier blocks.
K3 uses Block AttnRes, grouping its 93 layers into eight 12-layer blocks plus one partial block. This preserves cross-depth retrieval while reducing state overhead from a per-layer cost to a per-block cost.
The design targets information dilution in very deep networks. A key feature no longer has to survive dozens of sequential residual updates before the final layers can use it.
3. Stable LatentMoE Expands the Expert Pool to 896
K2 used 384 routed experts and activated eight per token. K3 expands that to 896 routed experts with 16 active per token, for a sparsity factor of 56. Routed experts operate in a 3,584-dimensional latent space before projecting back to the 7,168-dimensional backbone, preventing communication and weight traffic from scaling directly with the expert count.
Nearly 1,000 experts create two new problems: internal activations can explode, and expert loads become harder to balance. Stable LatentMoE addresses them with three components:
- Normalized LatentMoE applies RMSNorm after expert aggregation and before projection back to the backbone
- SiTU-GLU smoothly caps large activations to reduce overflow risk in low-precision arithmetic
- Quantile Balancing adjusts expert biases from router-score quantiles so loads converge more directly on their targets
Quantile Balancing is an infrastructure feature as much as a modeling feature. It helps prevent undertrained experts while giving large-scale expert-parallel training more stable computation shapes and throughput.
Native Multimodality Starts From Pretraining
K3 does not attach a vision tower after language-model pretraining. Text and visual tokens share the same next-token prediction objective from the start.
More unusually, the 401M-parameter MoonViT-V2 encoder is trained from scratch rather than initialized from SigLIP. The report says a pretrained vision encoder produced frequent gradient spikes when jointly optimized with the large language model. MoonViT-V2 remained more stable and matched the SigLIP-initialized baseline on vision evaluations.
MoonViT-V2 processes images and video with shared parameters, combining spatial attention, temporal attention, and temporal pooling. A 2x2 pixel-shuffle operation reduces visual-token count by a factor of four, keeping images up to 3,584 x 3,584 pixels manageable.
This is the foundation for K3’s “vision in the loop” behavior: it can write frontend or game code, inspect a live screenshot, and continue refining the result without handing visual inspection to another model.
Training: The Report Does Not Disclose the Total Token Count
K3’s pretraining corpus spans four text domains - web, code, mathematics, and knowledge - plus captions, interleaved image-text documents, OCR, perception, video, and visual coding data. The visual coding set includes SVG, 3D assets, websites, games, and CAD, directly pairing code with rendered output.
The report explains the data processing and training methods, but does not disclose total pretraining tokens, GPU count, or total training cost. Any precise-looking figure for K3’s training-token budget is therefore unsupported by the report.
The published training recipe includes:
- Per-Head Muon, which orthogonalizes each attention head separately so high-gradient heads do not dominate the update
- Cosine learning-rate decay, a 1% linear warmup, and 0.1 weight decay
- Progressive context extension from 8K to 64K, followed by 256K and 1M during cooldown
- Multi-style rephrasing of knowledge and mathematics data with source-fidelity verification
- Quantization-aware training beginning at SFT, with the main MoE expert weights ultimately stored in MXFP4
Long-context training is not just document concatenation. Moonshot synthesizes multimodal subtasks that can only be solved by retrieving information scattered across the full 1M-token sequence, preventing the model from learning to focus only on local regions inside a long input.
Post-Training: Nine Expert Policies Distilled Into One K3
K3’s post-training pipeline has three stages: an SFT cold start, domain-specific reinforcement learning, and Multi-Teacher On-Policy Distillation (MOPD).
RL spans three domains: general tasks, general agents, and coding agents. Each domain is trained at low, high, and max reasoning effort, producing nine expert policies before their capabilities are consolidated into one model.
This is more deliberate than truncating a single deep-reasoning model at inference time. Low, high, and max are separately optimized through per-problem budget constraints during RL, with penalties for trajectories that exceed their assigned budgets.
The training environments also look more like real work than conventional coding benchmarks:
- They simulate different agent harnesses, including Kimi Code, Claude Code, Codex, and OpenClaw, reducing overfitting to one tool schema
- Agents operate across mock Gmail, Notion, and Slack applications in multi-day, cross-tool tasks
- A single trajectory can span thousands of tool calls and millions of accumulated context tokens
- RL tasks include real GPU kernels, web development, professional research, and verifiable autonomous execution
- Hidden verifiers and sandbox isolation help limit reward hacking
The report says training and evaluation created 51,219,741 sandbox instances across 1,505,678 images. That figure says more about K3’s focus than the parameter count: the goal is an agent that keeps acting inside an environment, not merely a chatbot that scores well on static questions.
Benchmarks: Open-Weight Frontier, Not an Overall Proprietary Win
Representative results from the technical report include:
| Benchmark | Kimi K3 | Claude Fable 5 | GPT-5.6 Sol |
|---|---|---|---|
| ProgramBench | 77.8 | 76.8 | 77.6 |
| Terminal-Bench 2.1 | 88.3 | 88.0 | 88.8 |
| FrontierSWE | 81.2 | 86.6 | 71.3 |
| BrowseComp | 91.2 | 88.0 | 90.4 |
| AutomationBench | 30.8 | 29.1 | 29.7 |
| OmniDocBench | 91.1 | 89.8 | 85.8 |
Independent results support the same “near the frontier, not first everywhere” conclusion. At the report’s cutoff, K3 ranked fourth on Artificial Analysis Intelligence Index v4.1, second on the Vals Index, and first on WebDev Arena.
These tables are not absolute rankings. Models were tested with different harnesses such as Kimi Code, Claude Code, and Codex. Some Claude Fable 5 runs used fallback behavior, while GPT-5.6 Sol could trigger cyber safeguards. The defensible conclusion is narrower: K3 has entered the proprietary flagship performance range and leads on some coding, search, and automation tasks, but still trails the strongest closed models in overall experience.
”Open” Has Two Layers: Public Weights, Conditional License
K3’s full weights are available on Hugging Face for research, deployment, fine-tuning, modification, and derivative works. But the model uses the custom Kimi K3 License, not Apache 2.0 or MIT.
Two commercial conditions deserve particular attention:
- A company operating a Model as a Service business whose aggregate revenue with affiliates exceeds $20 million over any consecutive 12 months must sign a separate agreement with Moonshot AI before commercial use
- A commercial product using K3 with more than 100 million monthly active users, or more than $20 million in monthly revenue, must prominently display “Kimi K3” in its user interface
Those two conditions do not apply to internal use or use through Moonshot AI’s official products and certified inference partners.
”Open-weight model” is therefore the most accurate description. The license is permissive for researchers and most smaller teams, but it sets explicit boundaries for large MaaS providers.
Deployment Reality: Easy to Download, Expensive to Run
Moonshot recommends vLLM, SGLang, and TokenSpeed, and suggests a supernode with at least 64 accelerators. The challenge is not only the 2.8T weight footprint. Deployment must also handle communication across 896 experts, joint management of KDA states and MLA KV caches, and scheduling for million-token requests.
MXFP4 reduces the theoretical storage for the primary expert weights to roughly 1.4TB, but non-expert modules remain at higher precision. A real deployment also needs room for runtime state, caches, communication buffers, and redundancy. The 104B active-parameter count lowers compute per token; it does not make the other weights disappear from memory.
K3’s open-weight value is therefore not “everyone can run it locally.” It gives the community its first opportunity to inspect, adapt, and deploy a frontier 3T-class agent model in full. Its immediate impact may be on MoE communication, FP4 inference, hybrid linear attention, and supernode scheduling rather than desktop chat applications.
Conclusion
Kimi K3’s most important breakthrough is not model size by itself. It places a 2.8T model, 1M context, native multimodality, and long-horizon agent RL inside one downloadable weight release.
Its limits are equally clear: it does not beat the strongest proprietary models across the board, deployment is a data-center engineering project, and the custom license constrains some large commercial services.
Even so, K3 moves the open-model ceiling forward. The community is no longer limited to chasing benchmark scores from a distance; it can now study how a frontier 3T-class system is trained, quantized, scheduled, and served.
Sources:
More Articles
