Elhoushi et al. [Cerebras Systems, MBZUAI]

♥ 609 Dropout Training

Training AI models requires a lot of computing power and solving this roadblock is necessary for reducing the energy and financial costs required to train capable systems.

To solve this, researchers are exploring the layer dropout method on various models which range form a few million to a billion parameters. In this approach, instead of forcing data through every single layer of the network on each pass, the technique bypasses complete processing blocks.

By pairing an optimal layer distribution that increases through the network's depth with a decreasing dropout schedule over time, the network learns smoothly. As a result, models achieve lower or identical validation loss while cutting computational requirements by up to twenty-five percent during training.

Because the system learned to process information with missing internal components, it develops a structural resilience known as elastic depth. Engineers can deliberately skip intermediate layers, exit the network early, or use the model to verify its own predictions via speculative decoding without retraining.

These post-training techniques accelerate response generation speeds by up to 1.5 times with negligible loss in accuracy.

Suleymanzade et al. [EPFL, KAIST, University of Amsterdam, Carnegie Mellon University, TU Wien, AITHYRA, University of Oxford]

♥ 321 LLM Thinking bycloud’s pick

Humans take some extra time to ponder over puzzles, similarly, machines become far better problem-solvers when they can iteratively refine their internal thoughts. However, training neural networks to build on their internal scratchpad step-by-step is notoriously difficult.

To resolve this bottleneck, researchers developed looped flows which is a new framework that provides iterative reasoning as a sequence of guided denoising tasks. Instead of demanding that the network jump directly to an answer, the method starts with random noise and progressively clears it away across decreasing noise levels.

Because adjacent steps share the underlying noise and problem context, each computational stage is naturally encouraged to prepare and pass along useful clues to the next stage. This setup teaches the model to build an effective internal memory across time, even when the training signals only look a few steps ahead.

During actual problem-solving, the network constructs solutions by smoothly tracking this probability path from noise to clarity. This design allows users to scale up thinking time simply by calculating the path on a finer, more detailed grid. It also lets the network discover multiple valid solutions to flexible puzzles just by starting the journey from different initial noise samples.

When this approach was tested on complex mazes, difficult Sudoku puzzles, and abstract reasoning benchmarks, looped flows markedly outperformed prior recurrent approaches and resolved the vast majority of cases where earlier systems stalled.

DeepSeek-AI

♥ 27k LLM Architecture

Building an AI agent capable of handling complex, long-horizon tasks is compute intensive and it needs huge amounts of information. However, storing and shuffling this vast working memory, known as the key-value cache, creates severe storage and communication bottlenecks.

To overcome this hurdle, researchers have developed DeepSeek-V4.1-Flash, which is a multimodal model that supports one million tokens and drastically compresses its memory footprint without sacrificing performance.

This is achieved using a Causal Encoder-Decoder architecture that allows the model to project upper-layer memory states directly from lower-layer outputs rather than computing everything from scratch. This setup halves the compute required for digesting large prompts, and activates just eight billion parameters during initial processing and sixteen billion while generating responses.

To shrink storage even further, the team introduced Compressed Sparse Attention 2 paired with ultra-compact four-bit memory caching. Instead of forcing every layer to build and store its own independent records, layers can now share memory caches and reuse search indices across the network.

This reduces the runtime memory requirements to roughly one-fourth and persistent storage demands to one-eighth of prior designs.

Zhang [Princeton University]

♥ 2.7K LLM Recurrent Loops

Traditional LLMs have fixed structural depths, meaning the complexity of their internal computational path cannot naturally expand as a conversation. Training these models with reinforcement learning often creates computational mismatches between how prompts are read and how answers are generated.

To address this, researchers have introduced the Recurrent Looped Transformer, which is a new architecture that unlocks an unbounded path for internal computation while keeping the computational work per token strictly fixed. The design pairs a parallel causal encoder, which rapidly builds context memory, with a recurrent decoder. Instead of resetting its state at the boundary between a prompt and a response, the decoder continuously carries its internal state and a local sliding-window memory across every single token.

Since the context encoding can run in parallel, hardware accelerators can efficiently batch independent sequences together even as each sequence maintains its own sequential decoder updates. At the same time, the system also provides the same state transition across pretraining, fine-tuning, and reinforcement learning replay.