The Model Your Buyer Uses Is Not the One We Measure

No measurement taken through an API reproduces what a particular buyer sees in ChatGPT, or in any of the other three assistants, and that includes ours. It is an odd admission for a company that sells measurement to lead with. It is also true, and anyone weighing up this category is better off hearing it from a vendor than discovering it in a meeting.

The gap is not one thing. It is several modest things stacked on top of each other, and each of them is documented by the providers themselves.

What sits between the model and the buyer's screen

The first is the model. ChatGPT does not give every user the same one. OpenAI's help centre keeps separate pages for what the free tier gets, what the paid tiers get, and what Enterprise and Edu workspaces get, and it publishes model release notes, which is a polite way of saying the answer changes. A buyer on a free account and a buyer inside their employer's Enterprise workspace can put the same question to what both would call ChatGPT and be answered by different models.

The second is the account. ChatGPT's memory, when it is switched on, carries useful context from earlier chats into new ones. Custom instructions, available on every plan, are applied to every conversation. A procurement manager who has spent a month asking about one supplier's product range is not a clean sample of anything, and neither is the marketing director checking their own brand from the account they have used to discuss it all year.

The third is place. When ChatGPT searches the web, OpenAI says it may estimate a general location from the IP address and share that approximate location with its search providers to improve results. A question about suppliers asked from Lyon and the same question asked from Leeds are not, strictly, the same request.

The fourth is the call itself. An API request carries only what the caller puts in it. OpenAI's API does allow a developer to pass an approximate location to its web search tool, but that is a decision somebody makes and writes down, not something inherited from a person sitting at a desk. And even with search enabled, the model decides for each prompt whether to search at all.

Why the number still means something

Taken together, that does not make API measurement a poor copy of the real thing. It means there is no single real thing to copy. There is no one ChatGPT answer to a commercial question; there are as many as there are combinations of plan, history, settings and location, and the answer on a given buyer's screen is one draw from that spread. A screenshot of it, forwarded by a colleague in sales, is an anecdote with a timestamp.

What a measurement can be is controlled. The same prompts, in the same language, sent the same way, to the same assistants, repeated, at regular intervals, with competitors measured under identical conditions. None of that tells you what one person saw on Tuesday. It tells you what the assistants say about your category in German this month compared with last, and whether a competitor is named more often than you under conditions that did not change between the two readings. That is a narrower claim than the screenshot makes. It has the considerable advantage of surviving a second look.

The control is the whole point. If the conditions drift, a change in the number could be a change in the world or a change in the apparatus, and nobody can say which. If they hold, movement means something happened.

PSentry queries ChatGPT, Claude, Gemini and Perplexity through their official APIs, twice a month, segmented by language and market rather than by where any individual user happens to be sitting. Providers retire models on their own timetables, and when they do we move to the newer ones. The date of that move belongs next to the series, because a step change on one assistant on that date is more likely to be the model than your market.

So the limit stands, stated plainly. What we measure is not your buyer's screen. It is the nearest thing to that screen that can be measured twice and compared, which is rather more than the screen itself was ever going to offer.