Preprint Open access
BA-DPO: Bias-Adjusted Direct Preference Optimization for Language Model Alignment
Preference-based alignment methods such as Direct Preference Optimization (DPO) use pairwise preferences labeled by human annotators to fine-tune language models. However, annotators carry systematic biases toward some attributes: a name that signals a gender or an ethnicity, a persona, a language variety, a formatting …