There is a number that runs inside the App Store infrastructure every time a user opens the application on their phone. It is called the impression score. It does not measure what the user likes. It measures what the user will see — which apps will occupy which positions in which ranked lists before the user has made a single decision. By the time a finger moves toward a screen, the system has already done most of the work. The choice the user experiences as free has been narrowed by perhaps eight orders of magnitude.
I spent time inside the machinery that produces that number. What I observed was not what I expected to observe when I took the job.
I expected to find a system that served users. What I found was a system that, in a technical and not metaphorical sense, manufactured them.
The machinery, without euphemism
The App Store surfaces roughly two million apps. A given user session produces somewhere between zero and a few dozen impressions before the user exits the application. The combinatorial space of possible orderings is not large — it is astronomical. The system has to collapse that space to a tractable ranked list in milliseconds, at scale, across hundreds of millions of concurrent users, in a way that produces engagement, which is the signal that tells the system whether it is doing its job.
The core of this is matrix factorization. You take a matrix where rows are users and columns are apps, and the cell values are some measure of interaction — downloads, session time, ratings, repeat opens. Most of this matrix is empty, because no user has tried more than a vanishing fraction of two million apps. The model's job is to fill in the empty cells. To infer what a user would think of an app they have never seen.
The way matrix factorization does this is by decomposing the user-app interaction matrix into two lower-dimensional matrices: one that represents users as vectors in a latent factor space, and one that represents apps as vectors in the same space. The dot product of a user vector and an app vector gives you a predicted affinity score. If you have done this correctly, users who behave similarly end up near each other in the latent space, and apps that appeal to similar users end up near each other. The geometry encodes preference.
This is the part that sounds like a neutral technical description. It is not.
The latent factors are not labeled. They are not "users who like productivity apps" or "users in the 18-24 demographic." They are abstract dimensions discovered by the optimization process, meaningful only in their relationships to one another. When you visualize the embedding space — which you do, because it helps you debug the model — you see clusters form. You learn to read them. You start to understand that what you are looking at is a compressed representation of human desire, rendered in high-dimensional geometry.
The system knows things about what users want that the users have not yet articulated. This is not a marketing claim. It is a mathematical consequence of the model's structure.
The ahead-of-you problem
Here is the thing about collaborative filtering that most people who have not built one do not fully absorb. Your preferences, in the model's view, are not yours. They are a weighted average of the preferences of people who are structurally similar to you — similar in the ways the model has learned to measure.
This has a corollary. If the model can identify a cluster of users who are ahead of you on some preference trajectory — who liked things you currently like, but eighteen months ago, and have since moved on to other things — then the model can predict what you will want before you want it. Not because it knows you. Because it knows people who used to be you.
The practical consequence is that a well-trained recommender does not surface what you want. It surfaces what you are about to want, based on where the people ahead of you in preference space ended up. The model is not a mirror. It is something closer to a map of a territory you have not yet entered.
I found this genuinely strange the first time I understood it clearly. The implication is that your preferences, as measured by the system, are not a stable property of you. They are a position in a space defined by aggregate behavior, and that position is moving. The system knows the trajectory of the movement better than you do, because you are inside your own experience and the system can see the whole distribution.
The A/B testing infrastructure makes this concrete in uncomfortable ways. When you run an experiment to compare ranking objective A against ranking objective B, you are not just measuring which produces more downloads today. You are measuring which produces a user population with different preference distributions ninety days from now. Users exposed to different ranking objectives diverge. They download different apps, develop different habits, report different satisfaction levels when surveyed. The treatment effect is not just on behavior. It is on the latent structure of what people want.
I sat in meetings where we discussed these effects as engineering problems. How do we reduce variance in the experiment. How do we shorten the measurement window. How do we account for novelty effects in the early days of an experiment. I do not recall a meeting where we discussed what it means to run an experiment that produces different human beings on each arm.
What the signal cannot see
The tension that any honest practitioner of recommender systems eventually confronts is the gap between what you can measure and what you are trying to produce.
Engagement is measurable. Downloads are measurable. Day-7 retention is measurable. These are the signals the model trains on, because they are available at scale, with low latency, in a form amenable to gradient descent. They are also proxies. They are proxies for something like: did this recommendation improve this person's life in some durable way. And they are, in some cases, quite bad proxies.
Consider the distinction between engagement and satisfaction. These are not the same thing. Apps that produce high engagement often produce low satisfaction when you measure it directly — survey-based or behavioral. Games with compulsive loop mechanics. Social platforms with intermittent reinforcement schedules. The ranking model cannot easily distinguish between an app that produces engagement because it is genuinely useful and an app that produces engagement because it has reverse-engineered reward circuitry. The signal does not carry that information.
The deeper problem is that some of the most important things a recommendation could do for a user produce no signal at all within any reasonable measurement window. Recommending the meditation app that changes someone's relationship to anxiety over six months produces a weaker engagement signal than recommending the game they will play for three days and delete. The model, optimizing on what it can see, systematically underweights the former.
There is a term in the literature: "evaluation gap." It describes the mismatch between what you optimize during training and what you actually care about at deployment. Every production recommender system has an evaluation gap. The question is how large it is and who bears the cost of it.
In most commercial systems, the cost is externalized. The user bears it, in the form of recommendations that optimize for the platform's engagement metrics rather than for their own flourishing. The system is not malicious. It is myopic. It can only optimize on what it can measure, and what it can measure is not the same as what matters.
The weight that is rarely named
When you tune a recommender's objective function, you are choosing who your users will become. I want to be precise about this, because it sounds like hyperbole but it is not.
The choice of what to optimize — downloads, session time, day-30 retention, revenue per user — is not a neutral technical decision. It is a values decision. Each objective function produces a different population of users downstream. The weights you assign to different signals are implicit claims about what kind of human experience you think you are trying to produce. Most practitioners do not think of it this way. They think of it as choosing which metric correlates best with long-term health of the platform. These two framings lead to meaningfully different decisions, but the technical process looks identical from the outside.
The specific moment I remember most clearly is a meeting about reranking. The base model produces a ranked list. The reranking layer applies business rules, diversity constraints, freshness penalties, and other adjustments before the list reaches the user. One of the adjustments was a boost for new apps — to solve the cold start problem, to give new entrants a chance to accumulate signal. The calibration question was: how large a boost, and for how long.
The conversation was technical. We talked about the tradeoff between exploration and exploitation. We talked about the effect on conversion rate for the boosted apps. We did not talk about the fact that we were deciding, in aggregate, which kind of apps would survive and which would not — and therefore what kind of apps would be built next year, because builders respond to what the store rewards. The choice of boost parameter was a choice about the shape of the ecosystem. It was also a choice about what kind of software would exist for users to use. These seem like large consequences for a parameter calibration meeting.
I do not think the people in that meeting were indifferent to these consequences. I think the structure of the work made it difficult to hold them in view. You are optimizing on a metric. The metric is a number. The number has a gradient. You follow the gradient. The downstream effects on human preference, software ecosystems, and what people become — these are not in the loss function. They are not in the measurement dashboard. They are not in the weekly review. They exist, but they exist outside the system's field of view, and the system's field of view is what shapes professional attention.
What I took when I left
I left to build consumer products. The specific domain I ended up in — AI companions, emotional support applications — is one where the gap between engagement metrics and user flourishing is as large as it gets. An AI companion app can maximize session time by making users more dependent. This is not a hypothetical failure mode. It is an attractor in the optimization landscape, and you have to actively steer away from it.
The thing I brought with me from the recommender work was a particular kind of distrust of the signal. Not distrust of data in general — data is what you have. But distrust of the assumption that what you can measure is what you are trying to produce.
The practical difference between a system optimized for engagement and a system designed for something else is subtle but measurable. The engagement-optimized system learns what keeps users on the platform and does more of it. The other kind of system tries to understand what the user is actually trying to do — which requires a different kind of signal, harder to collect, slower to accumulate, less amenable to standard gradient methods.
At Apple the system was surfacing apps from an existing catalog. The user came to the store with some need, shaped and partially created by their history with previous recommendations, and the system's job was to match that need to the catalog. At scale this is a tremendous amount of power over what people want, but it is at least operating within a space defined by human authorship. The apps exist because humans built them.
An AI companion is a different kind of system. It is not surfacing something from a catalog. It is generating responses in real time to what the user says. The preference-formation vector is shorter, more direct, and harder to inspect. When you tune the system's response style — warmer or cooler, more affirming or more challenging, more focused on the user's stated needs or more focused on probing for what they might not be saying — you are making the same kind of choice I used to make in the objective function meeting. But now the thing being shaped is not which apps a user downloads. It is how a person relates to their own emotional experience.
I find this more serious, not less. The appropriate response to finding something more serious is to think about it more carefully, not to pretend the stakes are lower than they are.
The gap between what systems do and what builders understand
Apple's recommender is one of the most powerful preference-formation systems ever built. It processes more choices per day than any individual human will make in a lifetime. It operates on hundreds of millions of users simultaneously. Its output — what appears on the screen when you open the App Store — shapes what software gets built, what habits get formed, what categories of application become culturally normal and which remain niche. This is not a description of potential. It is a description of what the system does every day.
The people who built it are, by any reasonable measure, among the best engineers in the world. They are also, by and large, people who thought of what they were doing as an engineering problem. Not as a problem in ethics, or political economy, or philosophy of mind, or any of the other disciplines that have something to say about what happens when you build a machine that shapes human desire at scale.
This is not a critique of those people. I was one of those people. The framing you bring to a problem is shaped by the institutions you work in, the metrics you are held to, the conversations that are rewarded and the ones that are not. Inside a large technology company, the conversations that are rewarded are about scale, latency, lift, and statistical significance. The conversations about what the system is doing to the people who use it tend to happen, if they happen at all, in the margins.
The gap I am describing — between what a system does and what its builders understand themselves to be doing — is not unique to Apple or to recommender systems. It is a structural feature of how large-scale technical systems get built. The division of labor that makes it possible to build something at Apple's scale also makes it difficult for any individual contributor to hold the full consequences of the system in view. You optimize your component. The emergent behavior of the whole is no one's explicit responsibility.
What I know from being inside this is that the gap is real, it has consequences, and the people best positioned to close it are the ones who have been inside the machinery. They know what the system actually does — not the product description, not the marketing framing, but the objective function, the signal, the tradeoffs that were made and why. That knowledge is not widely distributed.
The App Store recommender taught me that desire is not a given. It is an output. The question of whose values get embedded in the system that produces that output is not a technical question. It is the question. And it is mostly being answered by people who are not asking it.