Kimi K3: Open Frontier Intelligence
Authors: Kimi Team
Paper: https://github.com/MoonshotAI/Kimi-K3/blob/main/k3_tech_report.pdf
Blog: https://www.kimi.com/blog/kimi-k3
Code: https://github.com/MoonshotAI/Kimi-K3
Model: https://huggingface.co/moonshotai/Kimi-K3
TL;DR
WHAT was done? The Kimi Team introduces Kimi K3, an open-weights native multimodal Mixture-of-Experts model containing 2.8 trillion total parameters and 104 billion activated parameters per token, supporting a 1-million-token context window. Built upon a 3:1 layerwise hybrid of Kimi Delta Attention and Gated Multi-Head Latent Attention, Block Attention Residuals across depth, and Stable LatentMoE with 896 routed experts, Kimi K3 incorporates a vision encoder trained from scratch via next-token prediction alongside multi-effort reinforcement learning distilled across general, agentic, and coding domains.
WHY it matters? While open-source research has rapidly adopted test-time reasoning scaling, pre-trained foundation scale in open weights has remained largely capped near the 1-trillion parameter mark, widening the performance gap against proprietary frontier systems. Kimi K3 demonstrates that simultaneous scaling of foundation parameters to 3T-class size and test-time agentic reinforcement learning at a 1M context length yields a 2.5× pre-training scaling efficiency gain over Kimi K2, establishing a competitive open alternative to elite proprietary models like Claude Fable 5 and GPT-5.6 Sol.
Executive summary: Kimi K3 delivers a major leap in open-weights model capabilities by scaling foundation architecture and reinforcement learning concurrently. For technical leaders and domain experts, it offers a production-ready frontier model with exceptional cost efficiency across long-horizon software engineering, complex tool-use, and native vision workflows. By open-sourcing the 2.8T weights and associated infrastructure libraries, this work provides the broader community with high-capacity architectures, auxiliary-loss-free routing mechanics, and microVM-based agentic evaluation paradigms necessary for frontier AI deployment.
Details
Bridging the Trillion-Parameter Scale Gap in Open Foundation Models
In recent frontier developments, model scaling has expanded along two distinct axes: pre-training compute scaling and test-time reasoning compute scaling. While the open-source community has rapidly integrated reinforcement learning techniques to scale test-time compute, open foundation models have remained largely stalled around or slightly above the 1-trillion parameter threshold, as seen in baselines like DeepSeek-V3 [review] and GLM-5. Applying complex agentic reinforcement learning onto foundation models of similar scale risks performance convergence while elite closed-source systems—such as Claude Fable 5 and GPT-5.6 Sol—pull further ahead. Kimi K3 addresses this structural disparity by pursuing both scaling axes simultaneously, elevating the pre-trained foundation to 2.8 trillion total parameters while scaling long-horizon agentic reinforcement learning across a 1-million-token context window.
The primary architectural bottleneck in scaling models to multi-trillion parameter scales lies in balancing information flow across sequence length, depth, and channel width without triggering numerical instabilities or memory communication wall-clock penalties. Traditional Transformer architectures scale quadratic attention memory requirements and experience activation spikes when scaling Mixture-of-Experts (MoE) sparsity. Kimi K3 establishes its performance delta over previous baselines like Kimi K2 through a 2.5× overall scaling efficiency gain, achieved by unifying efficient linear context mixing, deep non-sequential residual retrieval, bounded activation functions, and dual linear programming load balancing.
Mathematical Foundational Substrates: Hybrid Attention and Non-Sequential State Spaces
The sequence backbone of Kimi K3 relies on a structural synthesis of sequence, depth, and channel mixing. Along the sequence dimension, token representations xt∈Rd (where d=7168 denotes the hidden dimension and t indexes position) are processed through a hybrid sequence mixer combining Kimi Delta Attention (KDA) and Gated Multi-Head Latent Attention (MLA). KDA models linear retention recurrence over a recurrent state space matrix St∈Rdk×dv, driven by head-specific query qt∈Rdk, key kt∈Rdk, and value vt∈Rdv projections, where dk and dv represent key and value head dimensions. Channel-wise retention is governed by a decay vector αt∈(0,1)dk and write strength scalar βt∈(0,1). Periodic global content interaction is sustained via Gated MLA layers using No Position Encoding (NoPE), where key-value states are compressed into a low-dimensional latent vector ct=Wcxt and expanded on demand, delegating positional and recency-aware mixing entirely to intervening KDA layers.
Along the depth dimension, standard sequential residual accumulation hl=hl−1+fl(hl−1) is replaced with Block Attention Residuals (AttnRes). The network’s 93 layers are partitioned into N=8 blocks of 12 layers each. For a layer l, a layer-specific pseudo-query ql=wl∈Rd executes softmax attention over block-level representation keys ki=vi=bi, where bn=∑j∈Bnfj(hj) compresses inter-layer outputs within block Bn. This allows each layer to selectively retrieve representations directly from the token embedding and preceding depth blocks, reducing cross-stage memory and communication overhead from O(Ld) to O(Nd).
Operational Execution Flow: From Native Multimodal Inputs to Sparse Channel Activations
To trace the transformation of a single input through the architecture, consider a user query containing interleaved high-resolution images and text processed through the pipeline illustrated in Figure 2. Visual inputs are first encoded by MoonViT-V2, a 27-layer vision transformer comprising 0.4B parameters optimized with RMSNorm and zero bias terms. The visual features undergo a 2×2 pixel-shuffle downsampling step, reducing visual token length by 4×, before being projected by an MLP into the LLM embedding space. Text and visual tokens are interleaved seamlessly into a unified token sequence without any post-hoc modal alignment stage.
The embedded sequence enters the backbone blocks, structured in a repeating 3:1 pattern (three KDA layers followed by one Gated MLA layer), with each attention layer paired with a Stable LatentMoE layer. Within a KDA layer, chunkwise parallel execution partitions sequence length into chunks of size C. In prior formulations like Kimi Linear [review], unbounded log-decay mappings forced diagonal 16-token tiles to execute slow, explicit position-pair computations. Kimi K3 resolves this by bounding the per-step log-decay from below using a scaled sigmoid function:
where gmin=−5 and Ah is a learnable scale. As depicted in Figure 3, bounding the cumulative decay guarantees that reciprocal rescaling factors remain within the dynamic range of BF16 arithmetic, enabling both diagonal and off-diagonal tiles to execute via dense Tensor Core matrix multiplications.
The output of each attention mechanism passes to a Stable LatentMoE layer for sparse channel mixing. For an input vector x∈Rd, two full-width shared experts process x directly, while the routed path projects x down to a compact latent space z=W↓x∈Rℓ (where ℓ=3584). Tokens are dispatched to 16 active routed experts chosen from a total pool of 896 experts (representing a sparsity ratio of 56). The aggregated routed expert output u=∑i∈Tk(x)piEirouted(z) passes through an explicit RMSNorm layer before up-projection back to Rd via W↑:
To prevent activation explosion across the chained matrix multiplications of extreme MoE sparsity, feed-forward expert layers use the Sigmoid Tanh Unit GLU (SiTU-GLU) activation function, shown in Figure 4.
SiTU-GLU caps large-value growth while preserving SwiGLU’s positive linear regime:
Setting soft-cap hyperparameters to β1=4 and β2=25 guarantees an absolute coordinate upper bound ∥SiTU-GLU(x)∥∞≤β1β2=100, suppressing activation outliers and stabilizing low-precision training.
Systems Optimization, Load Balancing, and Post-Training Alignment
To balance expert dispatch across 896 experts without introducing auxiliary loss penalties that degrade model capacity, Kimi K3 implements Quantile Balancing (QB), illustrated in Figure 5.
QB derives router score bias adjustments by formulating load balancing as a dual linear program. For a batch of m tokens routed to n experts with Top-k selection (target load q=mk/n), expert biases b∈Rn are updated using the (1−k/n)-quantile of margins s:,j−α(t) across tokens:
Here, si,j represents the router score for token i and expert j, while αi(t) is the cutoff threshold from Top-(k+1) routing. At scale, gathering exact quantiles across millions of tokens incurs massive communication costs. Kimi K3 solves this by maintaining a per-expert binned histogram matrix H∈Nn×B (with B=1000 bins) across data-parallel ranks, recovering the whole-batch quantile via a single integer all-reduce per layer step.
Distributed pre-training over 2.8T parameters relies on MoonEP, an expert-parallel framework that guarantees perfectly balanced execution. By proving that a balanced assignment plan always exists with at most E/R redundant expert slots per rank (where E is total experts and R is expert-parallel size), MoonEP fixes sequence computation shapes to S×K tokens per rank. This enables zero-copy communication and eliminates host-device synchronization stalls caused by dynamic token counts. Matrix parameters are optimized using Per-Head Muon, partitioning momentum matrices along the head dimension to perform block-wise Newton-Schulz orthogonalization.
Post-training consolidates multi-domain capabilities through Supervised Fine-Tuning (SFT) using an XTML markup chat template (Figure 16), followed by Reinforcement Learning (RL) across general, agentic, and coding domains spanning three reasoning effort levels (effort∈{low,high,max}).
These specialized domain-effort policies are distilled into a single model via Multi-Teacher On-Policy Distillation (MOPD). For domain d and effort e, the student policy πθ receives a clipped advantage reward based on the teacher policy πteacher(d,e):
where sg(⋅) is the stop-gradient operator and Rmax limits reward signal variance. Serving efficiency is optimized via MXFP4 quantization-aware post-training for MoE weights (MXFP8 activations) and speculative decoding using an EAGLE-3-style draft model. The draft model directly minimizes the likelihood-based LK loss—the negative logarithm of the target-draft acceptance rate over vocabulary V:
Empirical Validation and Scalability Insights
As established in the empirical scaling law curves shown in Figure 7, Kimi K3 achieves a 2.5× gain in scaling efficiency over Kimi K2, sustained by the structural parameters detailed in Table 1.
The model expands total parameters from 1.04T to 2.78T and activated parameters from 32.6B to 104.2B, while extending the context window from 128K to 1M tokens.
A central mechanistic finding during pre-training ablation concerns the visual encoder. As plotted in Figure 6, initializing the vision tower from a contrastively pre-trained SigLIP model (MoonViT-3D) induced severe gradient norm spikes. In contrast, training MoonViT-V2 entirely from scratch via next-token prediction stabilized gradient norms while matching downstream visual benchmark performance, demonstrating that contrastive pre-training is unnecessary for scaling multimodal LLMs.
On global benchmark evaluations reported in Table 2, Kimi K3 consistently outperforms open-weights models like GLM-5.2 and previous-generation proprietary models (Claude Opus 4.8, GPT-5.5), while closing the gap on Claude Fable 5 and GPT-5.6 Sol. Kimi K3 achieves 93.5% on GPQA Diamond, 88.3% on Terminal-Bench 2.1, 81.2% on FrontierSWE, 91.2% on BrowseComp, 95.0% F1 on DeepSearchQA, and 97.8% on Math-Vision (with Python tools).
Cost-efficiency analyses in Figure 13 highlight that on BrowseComp, Kimi K3 achieves 91.2% accuracy at $2.03 per task—half the cost of GPT-5.6 Sol (90.4%) and an order of magnitude cheaper than Claude models at maximum effort.
Independent third-party evaluations (Table 5) confirm these results, ranking Kimi K3 first on the crowdsourced WebDev Arena (1,678 Elo) and fourth on the Artificial Analysis Intelligence Index v4.1.
In qualitative technical case studies, Kimi K3 demonstrated advanced agentic execution by independently developing MiniTriton (Figure 15)—a compact Triton-like GPU compiler with MLIR passes and PTX codegen—and optimizing AttnRes CUDA kernels to cut execution latency from 283.6 ms to 114.4 ms (Figure 14).
Strategic Context and Theoretical Lineage
Kimi K3 represents a convergence of several recent architectural lines in deep learning research. Its linear sequence mixing inherits directly from delta-rule state-space recurrences like Mamba-2 [review] and Gated Delta Networks, specifically modifying the negative-Softplus decay of Kimi Linear [review] into a bounded sigmoid log-decay to unlock dense Tensor Core tiling. Its periodic global attention relies on the compressed key-value representations of Multi-Head Latent Attention (MLA) introduced in DeepSeek-V2, but eliminates positional encodings (NoPE) entirely, leaving sequence positioning to KDA.
The model’s depth architecture adopts the Attention Residuals (AttnRes) formulation, extending data-dependent softmax attention over layer representations as pioneered in recent depth-routing literature. Its sparse channel layer builds upon DeepSeekMoE and LatentMoE, expanding expert counts to 896 while introducing SiTU-GLU soft-capping to resolve activation growth issues observed in scaling SwiGLU. For post-training, Kimi K3 aligns with test-time reasoning models like DeepSeek-R1 [review], extending on-policy distillation and budget-controlled agentic RL into long-horizon 1M-token environments using Firecracker-based microVM sandboxes via AgentENV.
Structural Bottlenecks and Domain Vulnerabilities
Despite its frontier-level performance among open models, Kimi K3 exhibits clear performance boundaries and structural limitations. On research-level scientific reasoning tasks such as Humanity’s Last Exam (HLE-Full), Kimi K3 scores 43.5% without tools and 56.0% with tools, lagging behind Claude Fable 5 (63.0%) and GPT-5.6 Sol (58.0%). Similar deficits appear on CritPt (23.4% vs 32.3% for GPT-5.6 Sol), indicating that high-level scientific conceptual reasoning remains an area requiring further improvement.
In cybersecurity exploit evaluations, Kimi K3 demonstrates strong Tier 1 defensive vulnerability discovery (identifying 16 previously unknown zero-days in open-source systems), but encounters significant bottlenecks in Tier 2 end-to-end exploit development. On kernel-level privilege escalation tasks, the model failed to produce working exploits on 75% of test cases. Trajectory analysis reveals four primary failure modes: inability to complete final exploit chains after acquiring primitives, poor strategy selection under OS mitigations, getting trapped in unproductive debugging loops, and premature submission without sufficient local verification. Furthermore, serving a 2.8T MoE model with 104B active parameters introduces severe infrastructure demands, requiring high-bandwidth interconnects and sophisticated fleet scheduling to manage 1M-token prefix caches.
Architectural Trajectory and Industry Implications
Kimi K3 establishes a critical milestone in open-source artificial intelligence by proving that non-standard Transformer backbones—combining hybrid linear-attention (KDA-MLA), non-sequential depth residuals (AttnRes), and high-sparsity latent MoEs—can scale successfully to 2.8 trillion parameters. Its ability to achieve frontier agentic performance while maintaining low per-task inference costs challenges the necessity of traditional dense global attention architectures for ultra-long context modeling.
For research institutions and enterprise engineering teams, the release of the full 2.8T Kimi K3 weights, alongside open-source infrastructure tools like MoonEP, FlashKDA, and MiniTriton, provides a robust foundation for advanced agentic deployment. Teams focusing on long-horizon software engineering, complex tool orchestration, and high-throughput visual workflows should evaluate Kimi K3 as a cost-effective, open-weights foundation capable of executing sophisticated multi-step reasoning.