What even is a >< former (yes >< former)

June 17th ~ June 23rd
#113 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 12k Z.ai has announced the release of GLM-5.2, an open-weights model under the MIT license with 1-million-token context window and updates in coding, reasoning, and agentic tasks. It has two reasoning effort levels [GLM-5.2 (max) and GLM-5.2 (high)] designed to help developers balance task performance with token efficiency. You can try it on Hugging Face.

   

2. ♥ 1k Poolside has released the base and post-trained weights for Laguna M.1, a model featuring a 256K context length licensed under Apache 2.0. Alongside the model checkpoints, the team has also released "pool" (an agent harness) designed to let developers run the model locally as a coding agent. You can try it on Hugging Face.

3. ♥ 37k Sakana AI has announced Sakana Fugu, a multi-agent orchestration system accessible through a single model API. The underlying Fugu Ultra model is trained to recursively call and coordinate various LLMs in an agent pool to manage complex, multi-step technical workflows. You can try it on the Sakana AI website.

   

Looped World Models

Lu et al. [FaceMind Research Asia]

♥ 473   World Models  

AI models that can navigate the physical world are called "world models". These models allocate the same amount of computing power to every single moment, whether an object is simply sitting still or dynamically colliding with another. Moreover, these models are too slow and computationally expensive to run on smaller, everyday devices.

To solve this, this paper introduced Looped World Models, or LoopWM. Instead of using a giant, resource-heavy stack of unique layers, this architecture uses a smaller, shared set of layers and runs them repeatedly in a loop to refine its predictions.

Because the model reuses its existing components rather than adding new ones, it achieves the predictive quality of much larger networks with a fraction of the parameters.

This looping structure also allows the model to adjust its computing power on the fly. It can run fewer loops during simple, predictable events and automatically allocate more loops to complex scenarios like physical collisions.

Furthermore, the architecture introduces "deferred decoding," which allows the system to process a sequence of actions entirely in its abstract internal language. Rather than wasting energy rendering the visual details of every intermediate step, it only decodes the final result at the very end. This combination of parameter efficiency and smart resource allocation offers a promising, practical path forward for real-time AI planning.

Variable-Width Transformers

Wu et al. [MIT-IBM Watson AI Lab]

♥ 437   Transformers   bycloud’s pick  

When we scale LLMs, we assumes that every layer requires the same computational budget, even though different stages of a model's processing pipeline perform entirely different roles.

This paper introduces an hourglass-shaped transformer architecture known as the >< former. This design keeps the initial and final layers wide but significantly narrows the layers in the middle.

To handle the fluctuating widths without losing information or adding complex math, the researchers implemented a clever, parameter-free residual resizing mechanism. The model maintains a wide global residual stream where narrower middle layers only read from and write to a specific slice. The remaining inactive dimensions are simply carried forward, bypassing the narrow layers entirely so they can be restored later in the network.

Across various model sizes, the ><former consistently outperformed traditional, uniform-width baselines in language modeling. By optimizing how space is used, the architecture achieved a twenty-two percent reduction in training computations and a fifteen percent reduction in memory and input-output costs for the key-value cache.

Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers

Movahedi et al. [ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, Tübingen AI Center, ETH Zurich, Swiss Institute of Bioinformatics, Université Paris Cité, Liquid AI]

♥ 363   Transformers  

When solving a difficult puzzle like a complex maze or a challenging Sudoku, humans naturally spend more time thinking. For AI, scaling its computational effort based on the difficulty of a task is hard.

Some researchers use looped neural networks, which process information by running it through the same internal layers repeatedly. However, these looped models face a double-edged sword: looping too many times can destabilize the mathematical signals inside the network, and it is notoriously difficult for the model to naturally decide when it has "thought" enough and should stop.

To address these challenges, researchers designed a framework called the Fixed-Point Reasoning Model, or FPRM. The team resolved the stability issue that affects deeply looped networks by restructuring how the model balances its internal signals. By switching to a pre-normalization setup paired with residual scaling, they successfully kept the model's internal data stable and trainable, even when running through many loops.

Most importantly, FPRM introduces a self-contained halting mechanism based on mathematical convergence. The model loops until its internal representations settle into a stable state, known as a fixed point instead of relying on an external, complex module to decide when to stop.

ExpRL: Exploratory RL for LLM Mid-Training

Xiang et al. [Stanford University, Carnegie Mellon University, OpenAI,]

♥ 855   LLM RL  

RL helps AI models solve complex reasoning problems, but training collapses when problems are too difficult because the model rarely stumbles upon the correct final answer to receive any feedback. Without a way to reward partial progress and smart intermediate steps, AI models cannot easily discover the creative strategies needed to solve challenging math and science problems.

To solve this, researchers developed a method called ExpRL, which stands for Exploratory Reinforcement Learning. Instead of forcing the AI to blindly copy human solutions, this approach keeps the correct answer hidden from the AI while it attempts a problem.

An automated judge then uses that hidden answer as a customized grading rubric to evaluate the AI's step-by-step thinking. By analyzing these drafts, the judge can award points for productive intermediate progress. This turns a simple pass-fail test into a learning experience that guides the AI's exploration.

The researchers found that this supportive warm-up phase prepares the AI for subsequent, more rigorous training. Compared to traditional fine-tuning or basic right-or-wrong feedback, models prepared with this method showed a much broader diversity of problem-solving strategies.

The AI naturally began to exhibit helpful thinking habits, such as double-checking its work, self-correcting mistakes mid-calculation, and backtracking when a strategy failed.

How Transparent is DiffusionGemma?

Engels et al. [Google DeepMind]

♥ 210   DIffusionLM  

Understanding how AI arrives at its decisions is vital for ensuring safety, debugging errors, and preventing misuse. While traditional LLMs think out loud step-by-step in plain English, newer text diffusion models like DiffusionGemma perform their reasoning in a continuous mathematical space.

Researchers tried to translate these hidden mathematical states back into human-readable concepts. They discovered that the complex vector information flowing between the model's iterative denoising steps can actually be mapped into a small bottleneck of natural language tokens.

Restricting the model to just a handful of these mapped words at each step caused almost no drop in its overall performance. By showing that these intermediate states represent interpretable guesses about the final text, the researchers successfully demonstrated that the model’s hidden reasoning depth can be simplified to a level nearly identical to traditional systems.

The researchers watched the model accurately estimate its final response length before choosing its words, retroactively correct earlier mistakes after formulating its reasoning, and even temporarily weigh multiple sentences at once.

Most importantly, evaluations showed that these complex internal mechanics do not compromise safety; the model remains just as monitorable as traditional architectures.