Baade et al. [Stanford University, Massachusetts Institute of Technology, University of Michigan]

♥ 396 Tokenization

Image generation models use visual encoders to condense raw pixels into compact representations, however, existing encoders were built for classification rather than creation.

In this paper, researchers introduced Relative Prediction Representations, which is a new approach that trains encoders to serve as generative tokenizers. RePR uses a generative model to directly estimate the shared information between different masked slices of the same scene instead of relying on traditional methods that compare separate images and drop subtle features.

By decomposing large visual scenes into incremental steps and selectively weighting noise levels, the system teaches the encoder how local regions mathematically predict one another. This allows the representation to capture both high-level meaning and fine spatial context without collapsing or ignoring crucial scene details.

The tests show that these representations scale predictabily and this proves that spending compute on an encoder makes downstream generative diffusion significantly faster and more resource-effective.

This new framework matches the representation quality of leading models like DINOv2 and WebSSL while requiring roughly seven to eight times fewer computing operations.

Zou et al. [Stanford University, Together AI, UIUC 4Meta]

♥ 192 Reasoning bycloud’s pick

Software agents solve all their problems by generating word-by-word text, which takes a lot of time and computing power. To fix this, researchers developed HybridAgent, which is a new architecture that dynamically shifts between silent internal contemplation and written text within a single continuous workflow.

Instead of forcing the model to convert every intermediate step into human vocabulary, this bridge routes the network's internal signals directly back into its input stream.

This enables the model to process complex logic in a continuous, high-speed internal mode and it returns to conventional text whenever it needs to execute a tool, observe environmental feedback, or present an answer.

The researchers discovered HybridAgent can project its internal reasoning states into readable pseudo-text, which allows developers to inspect and audit its logic in real time.

When researchers evaluated this model on benchmarks containing mathematics, medicine, and agentic search, this dual approach proved both smarter and significantly leaner than standard frontier agents. HybridAgent improved overall performance by an average of 10.4% while cutting token generation by up to half.

Zhang et al. [University of Illinois Urbana-Champaign]

♥ 298 LLM finr-tuning

Fine-tuning LLMs is an complex process where billions of internal knobs must be turned simultaneously. However small adjustments consistently cluster within a tiny fraction of the network. Researchers want to know whether a single mathematical principle could explain which parameters actually drive adaptation.

Researchers discovered that this behavior is caused by the diagonal model Fisher, a metric that measures how sensitive a model’s output distribution is to tweaks in individual parameters, which is independent of external reward signals.

The team showed mathematically that when a parameter has low sensitivity, its expected learning update is inherently bounded to near zero. In practical reasoning benchmarks, identifying this small subset of high-sensitivity weights before training began and freezing the rest allowed the model to capture the vast majority of full-parameter training gains using only a fraction of a percent of its weights.

When a model produces text, the candidate tokens it considers most plausible tend to share similar parameter sensitivities. Nudging competing options by the exact same amount does not alter their relative chances of being chosen, the model’s overall output distribution remains unchanged.

Gonzalez-Martınez et al. [ELLIS Institute Tubingen, Max Planck Institute for Intelligent Systems, Tubingen, Tubingen AI Center, ETH Zurich, Liquid AI]

♥ 262 LLM Memory

AI models need deep learning to process language, but the way they pass information from one layer to the next has remained surprisingly primitive. Older AI models add each layer's output directly into the next, which creates a bottleneck that prevents deeper layers from selectively storing and retrieving rich details from earlier stages.

This paper introduced depth processing as an explicit memory system. Rather than treating layers as a simple additive conveyor belt, their Depth Memory framework models depth as an active process of reading from and writing to distinct streams of information.

Instead of hoarding an ever-growing pile of past layer outputs, Linear Depth Memory treats cross-layer information retrieval as a regression problem that it solves on the fly using recursive least squares. This architecture manages three coordinated pathways per token: the familiar residual stream, a compact linear associative memory stream, and an inverse covariance stream that helps mathematically pinpoint which historical representations to retrieve.

Researchers tested this approach across 400-million and 1.5-billion parameter models, LDM consistently achieved lower perplexity than standard pre-normalized transformers and other recent depth-mixing architectures, while maintaining competitive zero-shot accuracy.