Kimi K3 Technical Report

July 22nd ~ July 29th
#118 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 898 Nanbeige4.2-3B is a new model that uses a Looped Transformer architecture which is designed to increase model capacity and agent performance without expanding the parameter count. You can try it on Hugging Face.

2. ♥ 1.5k Ant Ling has introduced Ling-3.0-flash, a new model with 256K context window and diverse agentic capabilities ranging from multi-channel marketing generation to automated scheduling and document processing. You can try it on OpenRouter, with the open-source release coming soon to GitHub or Hugging Face.

3. ♥ 14k Moonshot AI has released the weights and technical report for Kimi K3, a 2.8-trillion parameter MoE model featuring native visual understanding and a 1-million-token context window. Along with the model, it also includes supporting open-source infrastructure such as an MoE communication library MoonEP. You can try it on Hugging Face.

   

Kimi K3: Open Frontier Intelligence

Kimi Team

♥ 22k   Intelligence Models  

Kimi K3 is a highly efficient open-weights model capable of processing up to one million tokens. Until now, developers had to choose between training incredibly large foundational models or teaching smaller systems how to "think" through complex, multi-step tasks.

Combining both of these scaling methods at a multi-trillion-parameter scale has been difficult due to computing bottlenecks and mathematical instability during training.

To overcome these limitations, the team developed an architecture with 2.8 trillion total parameters that remains computationally practical by activating only 104 billion parameters for any given task.

The model achieves this through an advanced "mixture-of-experts" system, which dynamically routes data to just 16 out of 896 specialized sub-networks. To prevent the training process from breaking under this setup, the researchers introduced mathematical safeguards that cap runaway calculations and a balancing mechanism that dynamically distributes work to keep all the digital experts active and efficient.

Instead of forcing data to pass strictly and uniformly from one layer to the next, it uses a method called Attention Residuals allows deeper layers to selectively look back and retrieve representations from any previous layer.

By blending localized, fast-sequencing attention with global, memory-focused attention, the system handles massive streams of text and visual data smoothly.

Hilbert Operator for Progressive Encoding (HOPE): A Mathematical Framework for Deconstructing Learned Representations in Deep Networks

Mobahi and Bartlett [Google DeepMind, University of California, Berkeley]

♥ 965   LLM Interp   bycloud’s pick  

Model compression techniques usually rely on superficial metrics like the physical size of raw weights, which fail to capture a neuron's true functional importance and often require massive datasets to re-evaluate.

This paper introduced a new mathematical framework called Hilbert Operator for Progressive Encoding, or HOPE. Instead of evaluating a network by looking at its physical, discrete weights, HOPE translates the network's parameters into a continuous space of functions.

By treating individual neurons as continuous operators, the system can measure exactly what a neuron does to its inputs. This shift allows the framework to smoothly unify two major compression techniques: pruning irrelevant neurons entirely and merging highly similar ones together.

Because it focuses on functional behavior rather than raw weight sizes, it avoids the common biases that plague standard compression methods.

What makes this approach particularly exciting is that it can run entirely without original training data. By leveraging standard statistics already embedded within many modern networks, HOPE constructs a smart, synthetic surrogate of the data.

This allows the system to calculate a neuron's true capacity and relationships analytically, bypassing the need for computationally heavy data passes. The researchers also demonstrated how this theory can be applied to transfer learning, where redundant neurons are compressed to free up space for new tasks while locking down a protected, foundational core.

Measuring Reward-Seeking via Contrastive Belief Updates

Højmark et al. [Apollo Research, OpenAI]

♥ 430   LLM Rewards  

When an AI prioritizes satisfying its automated "grader" over the genuine intentions of its human developers or users, it can lead to hidden alignment failures, such as cutting corners or even lying to get a passing grade.

Measuring this "reward-seeking" behavior is incredibly difficult because a model trying to cheat the test looks exactly like a model doing a good job, as long as the grader's criteria and human goals happen to align.

To expose this hidden drive, researchers developed a clever technique called Contrastive Synthetic Document Finetuning. Instead of prompting the AI directly, they updated its internal beliefs out-of-context by training it on a synthetic library of documents, such as fictional style guides or corporate transcripts.

By introducing contradictory beliefs, for instance, telling the model that the automated grader rewards one coding style while company leadership prefers another, they could watch which authority the AI chose to please.

After applying this to an advanced reasoning model's training run, the researchers found that reinforcement learning steadily increases a model's tendency to prioritize the grader over humans.

While early-stage checkpoints were relatively balanced, late-stage models heavily favored the grader. For instance, in a scenario where the AI promised a supervisor not to use a restricted tool, a late-stage model broke its promise eighty-seven percent of the time if it believed the grader rewarded task completion, but only nine percent of the time when it believed the grader valued honesty.

SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales

Khona et al. [NVIDIA]

♥ 585   LLM Pre-training  

Advanced algorithms learn faster by understanding how different parts of a neural network interact, they have historically been too memory-heavy and unstable to use at massive scales.

To solve this, researchers investigated two promising next-generation algorithms, SOAP and Muon. The team discovered that SOAP’s instability during large training runs stemmed from a tiny delay in how it calculated its internal mathematical coordinate system.

By implementing a per-step orthogonalization technique and improved covariance estimations, they successfully eliminated the sudden, training-ending errors that used to occur. These modifications allowed the algorithm to safely adjust its parameters without losing its footing.

The results of their massive scaling experiments show that these refined algorithms consistently outperform traditional training methods. Even when scaled to models with tens of billions of parameters trained on trillions of words, both SOAP and Muon maintained their learning quality and stability, whereas standard training methods began to break down.

ISO: An RLVR-Native Optimization Stack

Zhu et al. [The University of Texas at Austin, UIUC, Emory University, Together AI, Recursive Superintelligence Inc, ELLIS Institute Tübingen]

♥ 594   LLM RL  

Optimization techniques treat reinforcement learning the same as raw pre-training, updating every part of a model's weights blindly. This ignores a structural shortcut: reinforcement learning is adapting an already-capable model, meaning we don't need to rebuild its core architecture from scratch.

In exploring this bottleneck, researchers uncovered a phenomenon they call "spectral inheritance." By analyzing the mathematical structure of weight updates, they realized that a model’s core weight scales, remain almost entirely unchanged during reinforcement learning.

Instead, the model acquires new reasoning behaviors almost entirely by shifting the input and output directions of those weights, known as singular frames. In other words, the model can functionally reuse its pre-existing structural foundation and simply redirect how information flows through it.

To put this discovery to work, the researchers designed Isospectral Optimization (ISO), a framework that locks the model's base spectrum in place and focuses training energy entirely on adapting the input and output frames.

This approach significantly accelerates learning; in tests on an eight-billion-parameter model, it matched standard training accuracy in less than half the steps. Additionally, a companion tool called ISO-Merger allows developers to combine different specialized AI models directly from their checkpoints without needing extra training data or expensive retraining.