A training framework that makes a large language model maintain a correct answer when a user pushes back with an unfaithful counter-argument, while still conceding to genuinely faithful corrections—directly targeting the sycophancy failure in which models abandon initially correct stances under social pressure. AFICE (Alignment for Faithful Integrity with Confidence Estimation) first runs Bilateral Confidence Estimation (BCE): it gauges the model's confidence in the question from internal states during decoding and its confidence in the answer from cumulative token-probability ratios. BCE is used to construct a conversational preference dataset of (context, original statement, opposing argument) triples, on which the model is aligned via Direct Preference Optimization (DPO). Across many benchmarks this substantially improves faithful integrity under opposing arguments while keeping the model correctable, boosting utility and trustworthiness in interactive settings.
A method for making large language models hold their ground on correct answers when a user pushes back with an unfaithful counter-argument, while still conceding to genuinely faithful corrections—add…