Preprint Open access
GAW-PO: Preference Optimization with Gradient-Aligned Token Weights
Most preference optimization methods, such as Direct Preference Optimization (DPO), apply preference supervision at the response level, although autoregressive language models are optimized token by token. As a result, all tokens in a rejected response contribute to the negative training signal, including tokens that m …