Preprint Open access
Balancing Reference Guidance and Free Generation in Trajectory Rollouts for Reasoning RL
A verified reference solution provides a correct trajectory for training a reasoning model. Alternatively, a prefix of the reference can guide the model in generating a trajectory of its own. How much reference guidance should we provide? We study this question through prefix continuation, where the model continues fro …