No universal winner
Method rankings change with the benchmark and model size. Gains in selected settings do not establish broad superiority over GRPO.
More Methods,
Better Agents?
A Reality Check on Self-Distillation
in Agentic Reinforcement Learning
- 11
- training methods
- 4
- agent benchmarks
- 3
- model sizes
- 132
- training runs
- 12,124.4
- recorded H800 GPU-hours
Why we did this study
As more self-distillation methods are proposed for agentic reinforcement learning, we wanted to understand which ideas are worth building on. We invested in a shared evaluation to understand where these methods help, where they struggle, and what they cost. A method need not win everywhere to be useful. We hope that making both its strengths and limitations visible helps others choose a starting point and build more reliable agents.
01 / The study
A common ground for comparison
Abstract
Self-distillation methods for agentic reinforcement learning are proliferating, but comparisons often reuse baseline scores obtained under different experimental conditions. Direct comparisons remain limited, with less attention paid to training stability and computational cost. We compare ten representative methods against GRPO across four benchmarks and three model sizes, conducting 132 training runs and repeated final evaluation over eight rollout seeds. The results show that gains over GRPO are setting-dependent, strong final performance can mask unstable training, and additional computation does not consistently translate into better performance. This independent benchmark supports a more reliable and cost-aware assessment of progress in agent training.
Endpoints are not trajectories
A strong final score can hide collapse, recovery, or late regression during training.
More compute ≠ more progress
Equal training steps do not mean equal cost, and extra teacher computation does not guarantee better results.
Experimental scope
One framework.
Twelve settings.
Three model sizes across four interactive environments, with 100 training steps per run and validation every five steps.
Benchmarks
- ALFWorld
- Household tasks Success rate · 140 games
- ScienceWorld
- Scientific experiments Task score · 200 tasks
- Search-QA
- Retrieval-assisted question answering Exact-match accuracy · 3,125 questions
- WebShop
- Web-based shopping Task score & success rate · 128 goals
GRPO and ten alternatives
GRPOGRPO+OPSDSDARSkill-SDRLSDOCSDOVCSDAgentOPSDADRSICSDPCSD
Final evaluation uses the same fixed task sets as validation, with eight rollout seeds at temperature 0.4. All methods are evaluated without privileged teacher context.
02 / Final performance
Gains depend on where you look
No method consistently outperforms GRPO across all twelve settings. Even a strong aggregate result can coexist with substantial losses on individual tasks.
ALFWorld & ScienceWorld
Full-resolution table ↗
Search-QA & WebShop
Full-resolution table ↗
03 / Training dynamics
The path matters, not just the endpoint
Validation trajectories reveal behavior that final rankings miss. Across 132 runs and 2,772 validation checkpoints, the study examines learning progress, peak-to-final declines, and low-performing outcomes.
A closer look at instability
On 4B ALFWorld, AgentOPSD undergoes a large decline before recovering, GRPO+OPSD remains low after collapse, and OCSD loses part of its earlier gains. A final score alone does not distinguish these training histories.
04 / Computational cost
What does a better score cost?
Fixed-step comparisons are not fixed-compute comparisons. Rollout length and teacher computation vary across methods, so we pair final performance with measured training time in every setting.
A useful improvement must be judged by where it works, how reliably it learns, and the resources it requires.
Explore the study
Run the comparisons
Environment setup, benchmark preparation, training, checkpoint merging, and repeated evaluation are documented in the repository.