Hu et al. [NVIDIA]

♥ 485 LLM RL

Using RL to solve complex problems forces models to generate multiple attempts for the same prompt to measure progress, which wastes computational budgets. However, simply switching to a single attempt per prompt removes stabilization baselines.

To solve this, researchers developed FlashREINFORCE, which is a new framework that enables stable asynchronous learning from just one attempt per prompt without requiring an auxiliary critic model.

This new architecture uses a fresh batch of independent probelms to provide a balanced positive and negative feedback. Since async workers finish tasks at different times, system uses a sequence-level filter that removes trajectories which have drifted too far from active model.

Moreover, instead of weighting updates by raw token count, it averages updates within each trajectory first. This sample-mean approach ensures that lengthy, failed attempts do not disproportionately penalize the model or wash out correct intermediate steps.

This paper shows that models can learn faster and more reliably with far greater problem coverage. In mathematical reasoning benchmarks, FlashREINFORCE was able to sustain stable training across thousands of updates on smaller models and scaled to a 30B-MoE model.

Moreover, this approach preserved active Python tool execution and achieved a 96.5% success rate on previously unseen interactive decision-making environments.

Noukhovitch et al. [Mila, Université de Montréal, Allen Institute for AI, University of Washington, Trillium Labs, Canada CIFAR AI Chair]

♥ 512 LLM RL bycloud’s pick

RL can help LLMs solve complex problems, but it usually only polishes problems the model already knows how to solve while leaving truly difficult tasks behind. This bias, dubbed the "Matthew Effect" because the easy tasks get richer in performance while hard tasks stagnate.

Standard training algorithms generate a fixed number of response attempts for every single prompt. This fixed budget floods training batches with redundant variations of already-solved tasks, while complex tasks are discarded before the model can stumble upon a correct path.

To overcome this bottleneck, the researchers developed an adaptive sampling method called "Never Give Up." Instead of abandoning an unsolved prompt after a single round of failed attempts, the framework continues generating new attempts with a set probability until the model finally produces a correct answer.

This simple mechanism automatically filters out straightforward questions almost instantly and reroutes computational effort directly toward cracking stubborn, high-friction problems.

The system also stores recent failed attempts and usses them to create a much stronger reward contrast once the model finally discovers the right solution.

Marek and Ryabinin [Together AI]

♥ 687 LLM RL

Training LLMs via RL requires two separate engines, one generates text and another to calculates parameter updates. However, tiny numerical mismatches between them can quietly corrupt the process. This subtle divergence creates an accumulating bias that destabilizes training and can trigger complete performance collapse.

Researchers discovered that this fragility stems from a mathematical "drift" hidden inside standard reinforcement learning algorithms. When the generation engine uses faster, lower-precision math or lags slightly behind the training engine, the update formula forces the model to distill knowledge from its own imperfect copy.

Since these two engines synchronize their weights, this mimicry forms a feedback loop where small errors snowball step after step. Traditional supervised fine-tuning handles mismatched data effortlessly but, RL breaks down because the model is constantly chasing a flawed version of itself.

To solve this, the researchers introduced an correction called score centering which calculates the sampler's expected drift at each step and subtracts it. This neutralizes the artificial bias without altering the true learning signal. To keep the technique practical and memory-friendly, the team found that tracking just the top handful of likely next words provides enough force to cancel the drift completely.

The team tested LLMs scaling from hundreds of millions up to thirty billion parameters, and score centering preserved stability under severe quantization and delayed sync cycles that caused existing algorithms to fail.

Zheng et al. [Google, University of Maryland, Google Deepmind, University of Virginia]

♥ 6.4k Recursive LLM

If we want to build AI agents which can self improve, we need to teach them how to navigate massive search spaces without burning through endless computational resources. However, teaching an agent how to explore effectively is notoriously difficult because testing a new exploration strategy requires running expensive real-world trials where feedback is heavily delayed.

To solve this, researchers developed a new method called Dream-RSI. This is a new framework that allows an agent to optimize its search tactics through offline simulation.

This method uses agent’s past attempts and code execution results which have already been computed from an interactive map of the search space. Instead of running costly new experiments, the system converts this recorded history into a replay simulator.

Within this simulator, the system can essentially "dream," testing many alternative exploration strategies against pre-recorded outcomes in seconds without invoking the underlying models or running new code. A lightweight coordinator scores how well each simulated strategy balances finding high-value solutions against computation cost, immediately promoting the best-performing policy back to real-world tasks.

This creates a self-reinforcing engine: smarter online discovery uncovers richer data, which expands the simulation pool, which in turn breeds even sharper exploration strategies.

This approach was tested on different problems ranging from mathematical optimization, algorithm design, and GPU kernel engineering. This dreaming framework matched or outperformed state-of-the-art baselines while reducing computational costs by up to two orders of magnitude.