ReflectRL: Learning from Golden Negative Trajectories via Reflective-to-Direct Reasoning
Flawed expert trajectories become material for reflection, then a policy transition transfers the benefit to direct reasoning.
The short version
ReflectRL studies expert trajectories that fail on difficult questions. Such trajectories need not be treated as correct demonstrations to contain useful information: they can provide a partially productive route that is easier to inspect and repair than to recreate from scratch. The study calls this setting a Reflection Advantage.
The framework uses these golden negative trajectories to elicit reflective reasoning, then transitions the policy back toward direct reasoning. This separates the information useful during training from the behavior desired when the trained model answers independently.
A failed expert attempt can retain useful structure when the learning objective rewards repairing it.
From flawed guidance to independent reasoning
The framework combines golden negative trajectories, reflective reasoning, and a transition back to direct reasoning.
View in the paperWhy this matters
Trajectory-guided training loses supervision precisely where expert models also struggle. Discarding every failed attempt can remove intermediate reasoning structure that might still help a learner identify and resolve the difficult part.
How it works
- Collect failed expert trajectories as material for reflection rather than as targets to imitate.
- Combine reflective and direct reasoning within on-policy training.
- Apply a reflective-to-direct policy transition so the acquired behavior transfers to answering without trajectory guidance.
Examining the reflection advantage
Behavioral, descriptive, and intervention analyses investigate what information failed trajectories provide.
View in the paperWhat the study finds
Experiments span nine benchmarks, four language-model backbones, and four on-policy training methods. Behavioral and intervention analyses examine the reflection advantage; training dynamics, ablations, and policy-transition comparisons test how the guidance affects learning and later direct reasoning.
In-distribution and held-out performance
The source table compares training methods with and without ReflectRL on ID and OOD benchmarks.
View in the paperWhat this means
This changes the role of a negative example from a scalar signal of failure to an object with potentially reusable structure. Learning can ask which parts remain informative, where the reasoning breaks, and how a successful continuation differs.
The policy transition addresses a second challenge: supervision should improve the model’s own competence rather than create permanent dependence on privileged context. Studying both stages helps distinguish gains from access to an expert attempt and gains that persist in the model’s independent reasoning.
Paper and resources
This is an introduction to the linked paper. Refer to the original for method definitions, evaluation settings, and complete results.
Cite this work
@misc{reflectrl,
title = {ReflectRL: Learning from Golden Negative Trajectories via Reflective-to-Direct Reasoning},
author = {Bi, Jinhe and Zhou, Chennan and Jin, Zengjie and Aniri and Lu, Shuo and Huang, Wenke and Cao, Hu and Xiao, Xun and Zhu, Zhihong and Tresp, Volker and Shen, Fei and Ma, Yunpu and Chua, Tat-Seng},
year = {2026},
eprint = {2608.03972},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.03972}
}