Domain-Invariant Reward Models for Transferring LLM Preference Alignment Across Domains
A framework (DIAL) that learns a reward model for aligning a large language model in a target domain that has no human preference data, by transferring from a related, easier-to-label source domain. Because human preferences encode domain-agnostic concepts, DIAL optimizes a dual objective: a source loss fitting preferences where labels exist and a domain loss minimizing the divergence between source and target feature distributions, yielding gains across cross-lingual, clean-to-noisy, few-shot-to-full, and simple-to-complex transfer settings.
D
Data
Text
Aligning LLMs with Domain Invariant Reward Models David Wu
DIAL is a framework for learning reward models that align large language models to human preferences in target domains where no preference data exists, by transferring from related source domains tha…