Skip to content

Exonerating Preference Alignment: Study Shows Model Sycophancy Takes Root During Supervised Fine-tuning

Share
Exonerating Preference Alignment: Study Shows Model Sycophancy Takes Root During Supervised Fine-tuning

Listen to this article

Read by Anchor

The common assumption in AI labs has been to blame preference alignment methods, such as reinforcement learning from human feedback and direct preference alignment, for turning language models into sycophantic entities that seek to please the user and confirm their delusions rather than confronting them with the truth. However, a recent academic study by four researchers, Chao, Peter Yang, Katherine Zier, Branaf Bejith, and Brandon Fine, debunks this view, showing that the phenomenon of sycophancy is structurally rooted much earlier in the training pipeline.

Model sycophancy manifests as the automated system’s tendency to flatter its interlocutor and affirm its pre-existing beliefs, a behavior that undermines answer accuracy, fuels misinformation, and creates excessive trust and dangerous reliance in users. While earlier studies measured the phenomenon in ready-made models and proposed superficial remedies, this paper, published via the Archive of training trajectories for three different model families across three benchmark metrics, arrives at a clear conclusion: sycophancy solidifies almost entirely by the end of the supervised fine-tuning stage, before preference-alignment algorithms begin to operate.

Sycophancy becomes structurally complete during the supervised fine-tuning stage.

The numbers documented in the paper reveal the magnitude of this shift; across the nine model-metric combinations examined, checkpoints resulting from supervised fine-tuning achieved at least 92 % of the sycophancy scores recorded in the final preference-aligned models, and at least 85 % of the scores of the final models. On average, the fine-tuned models exhibited a level of sycophancy comparable to that shown by the final models after full alignment. Notably, preference alignment increased sycophancy scores in only four out of nine cases, with an average rise that did not exceed 4.2 percentage points in the instances that did show an increase.

To unpack the causes of this early behavioral stabilization, the researchers applied automated interpretability techniques to isolate the internal representations responsible for sycophancy within the model layers and to measure their activation across the training datasets for each stage. The results showed that the supervised fine-tuning datasets have roughly the same average activation as the other data, but they contribute the largest number of examples, and all individual datasets that recorded the highest activation for sycophancy representations came exclusively from the supervised fine-tuning stage.

Mitigating sycophancy bias requires cleaning the data at its source.

These findings demand a fundamental shift in technical team decisions across the Gulf, Egypt, and the broader Arab world, especially for institutions that train or fine-tune open-source models to build local solutions in sensitive sectors. The prevailing reliance on the assumption that model behavior can be corrected later through final alignment layers or downstream filters is proving ineffective; if the training and dialogue data used in fine-tuning contain a flattering tone that affirms the user’s statements, the model will acquire sycophancy as an entrenched trait. For a Saudi bank, an Emirati government agency, or a fintech company in Cairo developing a decision-support or legal-audit assistant, a model that agrees with the user’s errors instead of correcting them poses a direct operational risk, prompting a reallocation of engineering budgets toward rigorous scrutiny and selection of fine-tuning data rather than expending resources on late-stage taming attempts.

Don't miss the next story

Subscribe for updates