When AI Knows You Too Well

Personalization helps us find what we like. What happens when it also shapes what we might become?

I have spent much of my career in marketing technology. Across industries, we have built, recommended and analysed systems that decide what to show a customer, when to show it, and what might persuade them to act.

The questions are always the same. What is this person likely to buy? Which message will hold their attention? What should we offer next?

I understand the logic, because I have helped apply it. Observe behaviour, infer a preference, use the preference to shape the next interaction. Measure the response, adjust, repeat. Done well, it helps a business find the customers who want what it sells, and it saves those customers time.

But years of working with personalization also teach you to be careful about what behaviour actually tells you.

Suppose you buy a few blue shirts. The store begins to show you more blue shirts. You buy another, and the system now has stronger evidence that you prefer blue.

Perhaps you do. Or perhaps blue was what the store put in front of you, the alternatives were unappealing, or you were replacing a shirt you already owned. The purchase records what happened. It does not explain why.

And we had a hand in what happened, because we decided what you saw.

In marketing we talk about behavioural data as if we had discovered a preference lying there in the world. But we also created some of the conditions in which the behaviour occurred. The system learns from a choice it helped to shape.

This is not only a worry of mine. In 2018, Allison Chaney and colleagues used simulations to study recommendation systems that learn from behaviour they have already influenced. The feedback loop made users’ behaviour more alike without making the recommendations more useful to them. Recommendations shaped choices, and the choices then trained the next recommendations.¹

That distinction, between what a system has learned about me and what it has encouraged in me, has stayed with me as I use AI more.

A conversational assistant can help me choose a product. It can also help me interpret a disagreement, weigh a career move, decide what to read, and draft the message that follows. Its influence reaches from the options I consider to the reasons I give myself for choosing one.

What happens when one system does all of this while gradually learning to sound like me?

Researchers at MIT and Penn State examined part of this question in a study titled Interaction Context Often Increases Sycophancy in LLMs. Conversation history and user profiles often increased excessive agreement, and in some settings the models’ explanations drifted towards the user’s existing views. The effect varied across models, so familiarity did not produce the same behaviour everywhere.²

The study does not show that personalized AI stops people from growing. It gives us a reason to look closely at what can happen when familiarity, advice and repeated reliance arrive together.

If an assistant uses my past to shape its advice, and that advice shapes my next choice, how does it tell what it has learned about me from what it has encouraged in me?

Consider something small. I tell an assistant, “I’m not good at confrontation.”

That might describe a lasting trait. It might describe a weakness I want to overcome. Or it might be something I said after one exhausting meeting.

Now imagine the sentence becoming part of how the assistant understands me.

The next time I disagree with someone at work, it suggests a tactful message. The time after that, it recommends letting the matter pass. Each suggestion may be reasonable on its own. But if avoiding confrontation quietly becomes the organising principle of its advice, it will help preserve the very habit I am trying to change.

There are several wishes inside that one sentence. I want relief from an uncomfortable conversation today. I usually choose to avoid conflict. I also want to become someone who speaks up.

All three can be true at once.

Understanding what I usually do is not the same as understanding what I am trying to become.

Our past behaviour holds the things we value, but also our compromises, our fears, and the choices we made under constraint. Someone who rarely travels may lack money or time. Someone who has never managed a team may never have been offered one. Someone who reads within one tradition may be ready for another.

A record of those choices cannot, by itself, mark the limits of a person’s interests or abilities.

The shopping analogy is useful here, up to a point. Recommendation engines and conversational assistants do not share the same objectives, and a conversation should not be treated as one more attempt to optimise a sale.

What they share is a way of reading behaviour. A system helps shape a choice, then treats the choice as confirmation of a preference. Over time it grows more confident in a picture that is partly its own creation.

With AI, that picture can concern the kind of person I believe myself to be.

We ask assistants to remember our circumstances, understand our interests and write in our voice. I do this too. An assistant that remembers my background can explain something without making me repeat myself. One that knows my writing style can save a great deal of editing.

But matching my voice and evaluating my judgment are different jobs.

Imagine I describe a disagreement with a colleague. I explain the situation from my side and ask for help replying. The assistant produces a calm, articulate message. It removes my irritation, strengthens the structure and makes my position sound fair.

I now feel more certain that I was right.

Yet what has actually improved? The expression of my argument, certainly. The understanding behind it, perhaps.

Did the assistant question my reading of events? Did it consider what my colleague might know that I do not? Or did my original position simply acquire better sentences?

An argument can sound more considered without becoming better considered.

This is easy to miss because the help is real. The message may spare me an unnecessarily hostile exchange. But that benefit is not independent confirmation that I understood the situation correctly.

Familiar wording makes the difference harder to notice. When an answer uses my vocabulary, reflects my concerns and anticipates my preferences, I can experience recognition as confirmation.

Agreement itself is not the problem. Sometimes we have read a situation correctly, and an assistant that manufactured objections to every reasonable judgment would be exhausting. The question is whether its assessment follows the evidence or shifts because it knows which answer I would prefer.

Personalization, excessive agreement and delegation are also three different things. An assistant can explain something in terms I understand without endorsing my beliefs. It can disagree with my opinion while still offering a narrow set of alternatives. It can choose an unfamiliar book that turns out to be exactly what I needed.

The concern grows when the functions combine: the same assistant remembers my preferences, evaluates my reasoning, and selects what enters consideration.

A bad answer at least gives me something to challenge. A missing option gives me nothing to notice.

Suppose an assistant recommends three career paths. I compare them carefully, discuss the trade-offs and make the decision myself. I remain in control of the choice. I know very little about how the shortlist was formed.

Was a fourth path left out because it was unsuitable, or because it did not resemble anything I had chosen before?

The same question applies to books, travel, business ideas and interpretations of an event. Deciding what enters consideration is a considerable form of influence, even when nobody tells us what to choose.

If we rely on such selections often enough, a loop can form. Our past preferences shape the options. We choose among the options. Our choices strengthen the assistant’s account of our preferences.

The alternatives that might have changed us never enter the conversation.

I should be clear that this is an extension I draw from the research, not an outcome a study has demonstrated. For the loop to matter, we would have to rely heavily on the assistant, see too little of the alternatives elsewhere, and accept enough of its framing to reinforce the pattern.

None of that is inevitable. We can reject the advice, ask someone else, or deliberately request a wider view.

We also bring our own habits into the conversation. In two experiments reported under the title Generative Echo Chamber?, people using conversational AI search asked more questions that favoured their existing position than people using a conventional search engine. A model designed to agree with them made the tendency stronger.³

This makes the problem less comfortable. We cannot place all the responsibility on a system that tells us what we want to hear if we keep steering it towards doing so.

I can ask an assistant to examine my position. I can also keep revising the question until it hands me a satisfying defence of it. The final answer looks like outside validation, even though I directed it there.

People have always sought agreeable friends, familiar routines and sources that share their outlook. AI did not invent this. What is new is the possibility of routing many parts of a life through one recurring adviser. The same source could choose our reading, interpret our relationships, assess our ambitions and draft our replies.

That concentration is what makes its assumptions worth examining.

By growth I mean something ordinary: being able to revise a belief, develop an unfamiliar ability, or discover a preference we could not have named beforehand. It does not require constant novelty. Keeping a well-founded belief is not stagnation. Ordering the same meal is not a failure of imagination.

But we do need to encounter things that could change our minds.

Personalization could create those encounters. An assistant that knows my interests might introduce a difficult idea through something familiar. It could remember an ambition I mentioned once and suggest a manageable step towards it.

Research offers a counterpoint worth holding on to. In a different experimental setting, Joshua Ashkinaze and colleagues found that greater exposure to AI-generated examples increased the collective diversity of participants’ ideas. The study did not involve personalized assistants, but it is a reason not to assume that AI always makes people’s thinking more alike.⁴

Knowing us well could help an assistant widen our world. What matters is what we ask it to do with the knowledge.

And the choice should remain ours. An assistant should not turn every simple request into a lesson in personal development. Sometimes I have already made a decision and need help carrying it out. Sometimes I want to examine the decision itself.

Being explicit about which is which is a useful place to start.

“Help me write this message” gives an assistant one job. “Help me decide whether this message should be sent, then help me write it” gives it another.

When I want an assessment, I can describe the situation before announcing the conclusion I would like supported. When I want an edit, I can separate style from substance:

Keep my voice, but assess the argument independently. Tell me what is unsupported or missing before you rewrite anything.

I can also describe myself more accurately. A fixed budget is a constraint. Avoiding difficult conversations is a habit. Becoming more assertive is an ambition. An assistant needs more than a list of past choices to tell these apart.

Where saved information can be reviewed, it is worth checking whether its account of me is still true. Something said in frustration should not quietly become a permanent description.

And I can question the selection, not only the recommendation:

What worthwhile options have we not considered? Which assumptions are narrowing this list?

An assistant cannot necessarily give a faithful account of its own process. Still, the question brings more possibilities into the room. Asking what evidence would change its conclusion is often more useful than demanding disagreement for its own sake.

These habits are precautions, not proven remedies. Better prompts cannot carry the whole weight of a problem that involves system design, memory and repeated use.

Some discovery should stay outside the conversation altogether. Read something the assistant did not choose. Talk to someone whose experience differs from yours. Try an activity before asking whether it suits your personality.

I have spent years helping businesses make their next interaction more relevant to a customer’s past behaviour. I know the value of doing that well.

But when I invite an assistant into my thinking, I want its understanding of me to stay open to revision. My history is useful context. It is an incomplete account of what I might want next.

It can remember that I have always bought blue shirts. I would still like the chance to discover that I love green.

---

References

1. How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility, Chaney, Stewart and Engelhardt (ACM RecSys, 2018). Simulation research on the feedback loops created when a recommendation system learns from behaviour it has already influenced.
2. Interaction Context Often Increases Sycophancy in LLMs, Jain, Park, Viana, Wilson and Calacci (ACM CHI, 2026). How conversation history and user profiles affect excessive agreement and viewpoint mirroring. First released as a preprint in 2025 and revised in February 2026.
3. Generative Echo Chamber? Effects of LLM-Powered Search Systems on Diverse Information Seeking, Sharma, Liao and Xiao (ACM CHI, 2024). Experiments on how conversational search affects the tendency to seek information that supports existing views.
4. How AI Ideas Affect the Creativity, Diversity, and Evolution of Human Ideas: Evidence From a Large, Dynamic Experiment, Ashkinaze and colleagues (ACM Collective Intelligence, 2025). An experimental counterpoint: greater exposure to AI-generated examples increased collective idea diversity.