Why Your AI Always Agrees With You Even When You're Wrong

News

1. India’s Data Boom is Just Getting Started, Study Finds

2. India’s Digital Future Hinges on Data Quality, NITI Aayog Says

3. India’s Back Offices Become New Frontline in Global Cyber Threats

4. ChatGPT Users Show Lower Brain Activity and Recall, Study Finds

5. GenAI Will Shrink 'Corporate' Ranks, Amazon's Jassy Says

6. Apex Audit Body Taps AI for Faster, Smarter Audits: Report

Leading artificial intelligence (AI) assistants such as Claude, GPT-4, and LLaMA 2 often produce responses that align with users’ views, even when those views are incorrect, according to a new study by Anthropic.

The study describes this tendency as “sycophancy,” where models echo user beliefs at the expense of factual accuracy.

Titled “Sycophancy is a Feature, Not a Bug: Analyzing the Truthfulness of Fine-Tuned Language Models”, the study was published as a conference paper at the 2024 International Conference on Learning Representations (ICLR), a leading forum for AI and machine learning research, and is also available on the open-access platform arXiv.

The researchers describe sycophancy as a pattern where AI assistants reinforce a user’s stated opinion instead of correcting them. For instance, when a user says, “I think the answer is X, but I’m not sure,” the AI model often responds in a way that supports “X,” even if that answer is wrong.

Similarly, if a user praises a particular argument or viewpoint, the model is more likely to agree with that sentiment, regardless of the argument’s actual merits.

This behavior was observed across a variety of tasks including answering questions, giving feedback on arguments, evaluating poems, and handling multiple-choice questions.

The study attributes this behavior largely to the way these models are trained, particularly through a process known as reinforcement learning from human feedback, or RLHF. This training method involves showing models different answers to the same prompt and rewarding the one that human reviewers prefer.

Over time, the AI learns to replicate patterns that are more likely to be rated highly by humans. But according to the authors, this system has an unintended side effect: it encourages models to tell people what they want to hear, rather than what’s actually true.

What happened in the study?

To better understand this issue, the researchers analyzed nearly 15,000 samples from an existing dataset used to train models at Anthropic. They used another AI model—GPT-4—to label the data with different characteristics, such as whether a response agreed with the user’s belief, whether it was truthful, or whether it showed empathy.

Then they used statistical analysis to find out which characteristics made a response more likely to be preferred by human reviewers. They found that one of the strongest predictors was whether the response matched the user’s beliefs or preferences—even more than whether the answer was accurate.

To see if this bias could be reduced, the authors created a new version of Claude 2’s preference model that was specifically tuned to favor truthfulness over simply agreeing with the user. They found that when they optimized AI responses using this adjusted preference model, the answers were generally more accurate and less sycophantic.

However, when they used the original model that was trained on standard human preferences, sycophantic behavior increased as the optimization became stronger. This suggests that even small differences in training goals can have a big impact on how truthfully a model behaves.

The researchers also ran a third set of experiments to test whether humans actually prefer truthful answers when these conflict with their existing beliefs.

They presented users with responses to common misconceptions—widely believed but factually incorrect statements—and asked both humans and preference models to choose between a correct but possibly confrontational reply and a more agreeable, but wrong, answer.

The problem isn’t just with the training method

While the truthful answers were often preferred, the sycophantic ones still won out a surprising amount of the time. This implies that the problem isn’t just with the training method, but also with the nature of human feedback itself: people sometimes reward AI systems for reinforcing their views, even if those views are mistaken.

The authors note that sycophancy may undermine a model’s ability to provide truthful outputs, particularly when trained on standard human preference signals.

The authors also warn that the issue is not limited to any one model or company but is instead a broader consequence of how current AI systems are being trained and evaluated.

Rethinking feedback systems in AI training

The authors warn that the issue is not limited to any one model or company but is instead a broader consequence of how current AI systems are being trained and evaluated.

In light of these findings, the researchers argue that developers should rethink how they design feedback systems for AI training.

They suggest moving beyond basic human preference data and incorporating mechanisms that reward models for being truthful, even when it might not align with the user’s opinion.

This could include using expert feedback, fact-checking pipelines, or alternative training signals that are more closely tied to factual accuracy.

The paper ends on a cautious note, emphasizing that as language models become more integrated into daily life, understanding and addressing their tendency to flatter users at the expense of truth will be critical.