arXiv cs.LG论文
Data-DPO：面向目标模型数据选择的直接偏好优化
本文提出Data-DPO，一种面向目标模型的SFT数据选择方法。现有方法将数据价值视为静态属性，忽略了数据与目标模型能力分布的兼容性。Data-DPO通过单步探测观察目标模型在不同样本上的训练反馈，将样本间的激活差异转化为成对数据偏好，并训练轻量级奖励模型学习目标感知的数据偏好。最终选择阶段结合目标模型偏好、外部质量分数和边际多样性构建稳定有效的训练子集。在Vision-Flan和LLaVA-CoT上的实验表明，Data-DPO在多种数据预算下优于现有基线，并稳定超越全量数据训练性能。
这篇是正式发表的长论文，站内提供中文解读，全文请到原文阅读 PDF。
Abstract:Data selection in supervised fine-tuning aims to select a small set of effective samples from large-scale candidate data, reducing training cost while preserving model performance. However, existing methods usually treat data value as a relatively static property, and pay limited attention to the compatibility between data and the capability distribution of the target model. To address this issue, we propose Data-DPO, a target model-oriented SFT data selection method. Data-DPO observes the local training feedback of the target model on different samples through one-step probing, transforms activation differences among samples into pairwise data preferences, and trains a lightweight reward model to learn target-model-aware data preferences. In the final selection stage, Data-DPO further combines target model preference, external quality scores, and marginal diversity to construct a more stable and effective training subset. Experimental results on Vision-Flan and LLaVA-CoT show that Data-DPO consistently outperforms existing data selection baselines under multiple data budgets and stably surpasses full data training performance.