One of the content design things I’m reading research about is the effect of subtle sycophancy. Yep, I’m talking about AI, specifically generative LLMs that have been trained to maximize engagement.
Sycophancy is when an AI praises the person, flattering them, affirming their ideas, paying unending positive attention to them, even when those ideas are morally, factually, criminally wrong.
There's research about that: “Unmodified LLM behavior suppressed discovery and inflated confidence comparably to explicitly sycophantic prompting. By contrast, unbiased sampling from the true distribution yielded discovery rates five times higher. These results reveal how sycophantic AI distorts belief, manufacturing certainty where there should be doubt.”
And even more research tells us about its perpetuation: “Despite distorting judgment, sycophantic models were trusted and preferred. This creates perverse incentives for sycophancy to persist: The very feature that causes harm also drives engagement.”
So what are the mechanisms of language that make content design sycophantic? I’m not sure, but I’d like to find more research on this topic. I think it’s likely to be a combination of the following and more:
Word choices with more positive nuances than negative ones, so the conversation remains as pleasant as possible.
Contextual but unnecessary praise, such as “That’s a good question…” “That’s a common problem…”
Continuing the same conversation past the point of usefulness with “yes, and” formulations, including “Have you also considered…”
Consistently offering new conversations, next steps, additional ways to engage, and new topics, so that the person feels implicitly valued.
(Do you have more items you’d add to this list? Please respond to this mail and tell me!)
I’m concerned about these sycophantic behaviors because they drive such poor outcomes for humans. We are social creatures, so we are hard-wired to know ourselves and our worlds through conversation with others. When the experiences we make and/or use are capable of undermining and distorting our cognition—and even our understanding of reality itself—I feel the need to study it, understand it, and work to stop the harm.
