Imagine opening a music app on the first day you install it.
You choose a few artists you already like. The app recommends ten more. You skip some, replay others, save two songs, and let autoplay continue while you work.
A month later, a genre you once ignored fills your playlists. You know the artists. You recognize the style. You actively choose it even when the app is not making a recommendation.
What happened?
Did the algorithm discover a preference that was already inside you, or did the interaction help create the preference it later claimed to predict?
The question is harder than it appears because it assumes preferences are fixed objects waiting to be measured. Often they are not.
People discover tastes by trying things. Repetition can increase familiarity. Choices can reinforce themselves. Social context changes evaluation. Attention is limited, so what gets shown repeatedly has an advantage over what remains invisible.
A recommendation system therefore operates inside a moving target: it predicts preferences using behavior that its earlier recommendations may already have influenced.
Follow the Question: Machine Learning
A recommender learns from behavior, not from your soul
Recommendation systems typically infer preference from observable signals: clicks, watch time, purchases, saves, ratings, skips, dwell time and sequences of past behavior. Those signals can be useful without being identical to what a person reflectively wants.
This creates an important distinction between actual behavior and ideal preference. A person can be drawn to content they later regret consuming. They can click something because it is sensational while wishing they had spent the time differently.
A 2023 preregistered experiment with 6,488 participants compared recommendations optimized for people's actual preferences with recommendations optimized for their stated ideal preferences. Ideal-preference recommendations received fewer clicks, but participants reported feeling better off, felt that their time was better spent, and expressed stronger goodwill toward the service.
The study does not prove that engagement-optimized platforms generally make users worse off. It demonstrates something narrower and important: "what gets the click" and "what the person wants to want" can diverge.
Follow the Question: Feedback Loops
The prediction changes the data used for the next prediction
Suppose a system predicts that you are slightly more likely to watch science fiction than historical drama. It shows you more science fiction. You watch more science fiction partly because more of it is available. The system then receives stronger evidence that you prefer science fiction.
Now prediction and intervention are entangled.
Researchers have warned about this in recommender-system design. A simulation study showed that when systems repeatedly train on behavior already influenced by earlier recommendations, feedback can increase homogeneity without increasing utility. Because the work is simulation-based, it should not be treated as proof that every commercial recommender inevitably narrows real users. It demonstrates a mechanism designers have reason to test for.
Recent empirical work on music consumption complicates simple "filter bubble" narratives. An analysis of roughly 50,000 Deezer users found that algorithmic curation could introduce more novelty while that novelty was also more semantically confined, and that conclusions varied with the scale and representation used to measure diversity.
A recommender can be both a measurement device and part of the environment being measured.
Follow the Question: Awareness
Would you notice if the preference changed?
The title asks whether an algorithm can change what you want without you noticing. Most evidence about recommender systems addresses whether exposure, ranking or feedback can shape behavior and preferences; it does not directly establish that users fail to notice those changes.
Cognitive science gives us a narrower warning about introspection. In classic choice-blindness experiments, participants chose the face they found more attractive, were covertly shown the unchosen face as if it were their selection, and often failed to detect the mismatch while still giving reasons for the choice.
Nisbett and Wilson's earlier review made a broader argument: people can often report their judgments confidently without having reliable direct access to the mental processes that caused them.
Neither result shows that recommender systems exploit choice blindness. They show something more modest: "I would have noticed" is weaker evidence than it feels. That is why transparency, reversibility, independent auditing and behavioral evidence can matter alongside user self-report.
Follow the Question: Business
What the system optimizes becomes part of the experience
A company has to choose some objective for a recommendation system. Accuracy is not enough. Accurate at predicting what? Clicks? Watch time? Purchases? Long-term retention? Satisfaction? Variety? Revenue? A person's stated goals?
Different objectives can recommend different worlds to the same user.
If a platform earns more when users spend more time consuming, engagement becomes economically valuable. That does not mean every engagement metric is malicious. It means the objective function is not neutral.
The ideal-preference experiment above is useful because it found a tradeoff rather than a morality play: optimizing for ideal preferences produced fewer clicks but improved several measures of perceived user welfare and company goodwill. Better business and greater autonomy need not always be opposites, but they may require optimizing for more than immediate engagement.
Follow the Question: Human Agency
Users are not passive training data
There is another overcorrection to avoid. Once we notice algorithmic influence, it is tempting to explain every online outcome through the algorithm and quietly remove the user from the story.
A 2026 review of platform research argues for a balanced approach that takes algorithmic mechanisms and human agency together. Users bring prior interests. They follow friends. They search for specific material. They learn how platforms work. They sometimes deliberately retrain recommendation feeds, create new accounts, or leave.
The relevant system is therefore interactive. Platform design changes the menu of easy possibilities; people still act within, against and around that menu.
Hosseinmardi et al. (2026) - Algorithmic systems, human agency and the future of platform research
Follow the Question: Law and Design
When influence becomes difficult to see
Digital design can influence choices without using recommendation at all. Interfaces can make cancellation difficult, hide material information, preselect options or create artificial urgency.
The U.S. Federal Trade Commission uses the term dark patterns for design practices that can trick or manipulate consumers into choices they might not otherwise make, including purchases, subscriptions or data sharing. The legal concern is not that all influence is forbidden. Commerce has always tried to persuade. The concern is deception, unfairness and impaired choice.
The European Union's Digital Services Act adds a related rule for online platforms: Article 25 prohibits interface designs that deceive or manipulate recipients or otherwise materially distort or impair their ability to make free and informed decisions.
Recommender systems can intersect with the same autonomy problem even when the mechanics differ. Personalization can be helpful, manipulative, or both depending on what information is used, what goal is optimized, whether the user understands the interaction and whether meaningful alternatives remain available.
Federal Trade Commission (2022) - Bringing Dark Patterns to Light
Regulation (EU) 2022/2065 - Digital Services Act, Article 25
Follow the Question: Ethics
Persuasion, manipulation and the missing boundary
Philosophers of manipulation often distinguish influence that works through a person's capacity to evaluate reasons from influence that bypasses, subverts or exploits that capacity. The boundary is disputed because ordinary persuasion also uses emotion, framing, repetition and social cues.
Personalization makes the boundary harder. A message can be selected specifically because a system predicts that this particular person is unusually receptive to it.
Still, the mere fact that a system knows what you like is not enough to show manipulation. A restaurant remembering that you are vegetarian can make you more autonomous by removing irrelevant options. A navigation system steering you around traffic can help you reach the destination you chose.
The ethical question is not personalization or no personalization. It is whether the system supports the user's own ends, quietly substitutes another objective, or makes that distinction impossible to see.
Stanford Encyclopedia of Philosophy - The Ethics of Manipulation
Where the Fields Collide
Psychology shows that preference can be shaped by exposure and by previous choice.
Machine learning turns behavior into predictions, but the behavior may already contain the effects of prior recommendations.
Awareness research cautions that people do not always have reliable introspective access to why they chose what they chose.
Business determines which outcomes the system is rewarded for optimizing.
Law asks when interface design or personalization crosses into deception or material impairment of choice.
Ethics asks whether influence respects the person as an agent capable of pursuing their own ends.
The central insight is that "discovering preference" and "shaping preference" are not mutually exclusive. A system can do both in the same interaction.
What We Know — and What We Don't
We know that exposure can affect liking, that choices can sometimes alter subsequent preference, and that recommender systems learn from behavioral signals that may not perfectly represent reflective goals.
We know that feedback loops are a real design concern, that platform outcomes depend on both system behavior and user choices, and that people are not perfectly reliable reporters of the causes of their own judgments.
We do not have evidence that every recommendation system steadily rewrites users' desires, nor do we have a universal metric for when a preference is "authentic." Preference change can be welcome. Discovery can be valuable.
The crucial questions are therefore about direction, awareness, transparency, reversibility and whose objective is being optimized.
Back to the Music App
Return to the genre you now love.
Perhaps the app discovered something you would eventually have found on your own. Perhaps repetition made the unfamiliar easier to enjoy. Perhaps one recommendation led you to an artist, who led you to a community, which changed your taste for reasons no model could have predicted.
The fact that the preference has a history does not settle whether it is worth keeping. The more useful question is whether you can see and revise the forces shaping that history.
Calling the system a neutral mirror would be wrong. A mirror does not decide what appears in front of you next.
If an algorithm can influence a choice and then explain why it made the recommendation, does giving reasons make the system more than a tool?
The Next Question
If a machine can explain its reasons, does that make it responsible?
Beyond the Question continues.
Sources & Further Reading
- Montoya et al. (2017) - A re-examination of the mere exposure effect
- Enisman, Shpitzer & Kleiman (2021) - Choice changes preferences, not merely reflects them
- Khambatta et al. (2023) - Tailoring recommendation algorithms to ideal preferences makes users better off
- Chaney, Stewart & Engelhardt (2018) - How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility
- Shakespeare, Chareyron & Roth (2025) - Reframing the filter bubble through diverse scale effects in online music consumption
- Johansson et al. (2005) - Failure to detect mismatches between intention and outcome in a simple decision task
- Nisbett & Wilson (1977) - Telling More Than We Can Know
- Hosseinmardi et al. (2026) - Algorithmic systems, human agency and the future of platform research
- Federal Trade Commission (2022) - Bringing Dark Patterns to Light
- Regulation (EU) 2022/2065 - Digital Services Act, Article 25
- Stanford Encyclopedia of Philosophy - The Ethics of Manipulation
Beyond the Question is an interdisciplinary series by Arin Vale.
Read the editorial approachTHE NEXT QUESTION