Belcak et al. [NVIDIA]

♥ 693 LLM Context

Running the latest AI models is very slow, memory-intensive, and it can get up to seven times more expensive as an interaction grows. Generating tokens deeper within a prompt has a steep computational penalty, which forces systems to burn resources whenever they track long conversations or read large files.

AI models do not actually need these bloated context windows to succeed. Instead of feeding entire document libraries or full multi-step histories into the prompt, this paper introduced two straightforward strategies: selective input reading and frequent compaction.

When tackling massive texts, the model uses standard command-line tools to inspect relevant passages on demand rather than ingesting everything at once. When working through lengthy tasks, the system periodically condenses the interaction history into a concise summary of the current state, which wipes the accumulated conversational transcript before resuming work.

When this approach was tested, models using these compact contexts preserved or even improved their accuracy while dramatically reducing cloud serving costs. Additionally, models continued to solve complex software issues even under a strict routine where they took only a single environment action before summarizing their progress and starting a fresh conversation.

Abhyanka et al. [Together AI Research]

♥ 957 LLM Compiler bycloud’s pick

Architectures and usage patterns of AI models change so quickly that traditional, rigid software compilers cannot keep up. This leaves hardware performance untapped on expensive chips.

This paper introduces Together Compiler, a system that creates autonomous software agents to optimize code directly. When the researchers removed human-written rulebooks, rigid intermediate abstractions, and strict tracking tools, the agents found significantly better shortcuts on their own.

In fact, seeding agents with expert knowledge bases actually slowed optimization gains by up to 31 percent. By pairing unrestricted raw code access with an analytical cost model that calculates the optimal trade-offs between chip memory, calculation speed, and network communication, the platform lets agents rapidly discover the best execution setups for any hardware configuration.

In less than 24 hours of autonomous optimization, the Together Compiler boosted token generation speeds by up to 2.6 times over the best hand-tuned baselines. The system adapted seamlessly across diverse processors from Nvidia, AMD, and AWS.

Chou et al. [NVIDIA]

♥ 102 Action Models

AI robots need to plot visual paths and spatial goals in real time. Vision-language models generate these plans one coordinate token at a time, which creates a severe latency bottleneck that limits how fast machines can react to a changing world.

Researchers introduced VFlash to overcome this sequential bottleneck through a technique called Flash Primitive Decoding. Instead of predicting coordinates one number after another, VFlash uses a lightweight drafting model to generate an entire geometric object, such as a full robot trajectory or bounding box, in a single forward pass.

The system takes advantage of the predictable structure of visual primitives, organizing and outputting all coordinates simultaneously. The researchers realized that spatial accuracy does not require exact token matching with a target model. By directly accepting structurally sound spatial predictions, VFlash bypasses tedious verification and yields complete, highly accurate geometric paths instantly.

The resulting speed and precision offer exciting possibilities for real-time robotics. Across two-dimensional and three-dimensional tasks, VFlash consistently reduced spatial errors compared to traditional sequential decoding, while generating three-dimensional motion forecasts roughly twenty-five times faster.

In robot manipulation tests on embedded hardware, the framework cut inference latency by about eighty percent. Furthermore, feeding these rapid plans into a world action model improved physical task success from 62.6% to 70.2% on complex manipulation suites, while achieving competitive results in autonomous driving.

Lufkin et al. [Zyphra Research]

♥ 2.5k LLM Attention

AI models are getting better and we need bigger context windows to process massive documents. However, standard attention mechanisms compare every token to every prior token, which creates an exploding computational bottleneck that makes processing massive sequences prohibitively expensive. Prior remedies often permanently discard historical data that future steps might need, or compress information into fixed-size containers that quickly saturate and lose accuracy.

This paper discovered that the predictive information within natural language naturally grows sublinearly, which forms a power-law structure that mirrors an organized tree. This paper developed designed Zyphra Hierarchical Attention, which arranges stored memory into a multi-tiered hierarchy of recursively compressed summaries.

Instead of scanning every past word, an incoming query begins at the top of the tree, scoring broad summaries using a small set of learned reference queries. The system then adaptively selects and descends only through the most relevant branches, and pinpoints the exact slice of detailed memory it needs while skipping the rest.

And the best part is that existing models can adopt this indexer through a lightweight training phase of just one billion tokens, leaving the base model weights untouched.