Sieberling et al. [Massachusetts Institute of Technology, MIT-IBM Computing Research Lab]

♥ 829 Attention

LLMs can process text rapidly at a constant speed, but their memory capacity limits how much context they can recall over long sequences. Expanding this memory causes an explosion in overall parameters which makes models struggle on recall-heavy tasks.

To overcome this bottleneck, researchers developed triadic linear attention. This approach expands memory capacity in a parameter-efficient way. Modern linear attention networks store information in a two-dimensional matrix formed by combining an input-dependent key vector and a value vector.

The researchers realized they could lift this framework into three dimensions by introducing a second key vector. By storing the product of two distinct keys and a single value into a third-order tensor state, and reading it back out using two matching queries, the architecture multiplies its memory volume by the dimension of that second key while requiring only two modest extra projections.

This 3D memory is fully compatible with high-performance training techniques. The architecture naturally accommodates data-dependent forgetting by applying individual gates to slices of the tensor, and it cleanly removes old associations before writing new ones.

Chng et al. [SenseTime Research, LeapLab, Tsinghua University, Nanyang Tehnological Univesity]

♥ 551 Looped Transformer bycloud’s pick

Latest image generation models generate high quality images by increasing their size to billions of parameters, which creates computational and deployment bottlenecks.

Researchers discovered that instead of stacking endless unique layers, a model can achieve greater computational depth by recycling its existing architecture. They developed the Looped Diffusion Transformer, or Looped-DiT, which routes visual data through a shared group of middle network blocks multiple times within each image-generation step.

Simply repeating layers causes models to blur spatial details and overwrite key information. The team made two innovations: deep supervision, which evaluates and guides predictions across every intermediate cycle during training, and self-modulating attention, which dynamically regulates update strengths so tokens retain critical local details.

This approach creates a form of silent visual reasoning. As input passes through deeper loops, the network iteratively refines its work. It rearranges objects to satisfy tricky spatial relationships, resolves complex constraints, and self-corrects earlier rendering mistakes without needing explicit text instructions.

In tests, a compact 260-million-parameter Looped-DiT outperformed models roughly six and a half times larger across multiple benchmarks while requiring nearly five times less computation during inference.

Liu et al. [Apple]

♥ 601 Looped Transformer

Looped Transformers save memory by recycling a single shared block across repeated processing cycles to refine their thoughts. The conventional approach wastes the model's computational power and misses an easy opportunity to catch early errors.

Researchers realized that these discarded intermediate states provide a built-in pair of rough drafts and mature predictions without requiring external training or helper models.

To take advantage of this hidden signal, they introduced LoopCD, which is a technique that contrasts the model's final prediction against an earlier, weaker loop.

Simply steering token selection away from the initial, less-developed guess sharpens its ultimate decision. The method can even adjust this contrast dynamically and applies a stronger push whenever the model hesitates between closely matched candidates to cleanly break ties on tough decisions.

This adjustment gives a lot of improvements across demanding reasoning and programming tasks, which significantly increases benchmark pass rates on complex mathematics and code generation.

Huang et al. [Institute of Foundation Models, USC, CMU]

♥ 1.2k Looped Transformer

If we can build compact language models by running data through a shared block of layers in repetitive cycles, then it can give us enormous hardware efficiency. However, recurrent architectures have severe computational debt across pretraining, text generation, and reinforcement learning.

Researchers discovered that if a looped model is guided toward a stable resting state, which is known as a fixed point where extra iterations no longer alter internal representations, then the path taken to get there stops mattering.

Once states settle into this equilibrium, the model can safely swap the entire computational history for just the final outcome. This insight unlocks major engineering shortcuts: text generation can share final memory caches across steps to shrink memory footprints threefold without sacrificing accuracy.

The researchers were able to train a distilled companion model which could process initial prompts nearly 1.8 times faster, and use RL updates to cut evaluation runtimes in half by reusing saved endpoints instead of recalculating full loops.

To reliably form these well-behaved endpoints, the researchers addressed two training flaws. First, because different words settle at different speeds, the team created a dynamic depth schedule that learns from prediction feedback, to ensure the model sees properly stabilized context while maintaining enough variation to generalize.

Second, they introduced an orthogonal input method that injects data cleanly at each step, which prevents new inputs from unintentionally canceling out or amplifying existing representations.