LLM Answer Volatility: Why One Check Proves Nothing

Ask ChatGPT which tools lead your category. Write the names down. Close the window, open a fresh one, ask the identical question. There is a good chance the list is not the same. A brand that was there has gone. A brand you have never heard of has arrived. Nothing changed in the world between the two questions, and nothing changed on your website.

People react to this in one of two wrong ways. Either they decide the tool that reported it is broken, or they decide that something they published last week has finally started working. Both readings turn noise into decisions, the most expensive error in this category.

Answer volatility is not a defect you can file a ticket against. It is a property of how these systems produce text. Once you know where it comes from, you stop trying to eliminate it and start measuring around it.

Five reasons the answer moves

The model samples, it does not look anything up

A language model does not fetch a stored answer and hand it to you. It writes one token at a time, holding at each step a probability distribution over what could plausibly come next. Then it picks. Not always the top candidate: sampling parameters, temperature and top-p among them, deliberately let lower-ranked tokens through. That is what makes the output read like prose instead of a stuck record.

The consequence is structural. If your brand sits near the boundary of what the model treats as a plausible recommendation, it gets named on some runs and skipped on others, from an identical prompt, with nothing having changed. Brands deep inside the model's confident territory come up almost always. Brands nowhere near it almost never do. The borderline cases flicker.

The hidden instruction layer gets rewritten

Every consumer assistant wraps your question in instructions you never see. That layer tells the model how to behave, how hedged to be, whether to name commercial products at all, when to search the web before answering. Vendors edit it routinely and none publish a changelog for it. A quiet tightening of the rule about recommending named vendors can reshuffle an entire category, with no new model and no announcement.

The model underneath the product name gets swapped

The assistant keeps its name. The weights behind it do not. A new version ships, an older one is retired, a cheaper variant starts serving part of the traffic. What the system knows, and how it phrases what it knows, shifts. Your inputs did not change. They are now being read by a different reader.

Retrieval catches a different web every time

When an assistant searches before it answers, it is pulling live pages, and the live web is not a stable object. Results reorder. A page gets edited. A comparison article gets published and indexed. The documents the model reads while answering are not the ones it read yesterday, so the answer assembled from them is not yesterday's answer. On the platforms that lean hard on retrieval this is the largest source of movement, and it means your visibility can shift because of an editorial decision at a publication you have never visited.

Your own session is not a neutral instrument

Memory, saved preferences, custom instructions and the conversation already sitting above the question all condition what comes next. Ask about a competitor, then ask who else there is, and you get something different from a cold question in a clean session. Check your own brand while signed into the account you have used to discuss your own brand for a year, and you are not seeing what a stranger sees. You are seeing your reflection.

What that does to measurement

Put those five together and the conclusion is uncomfortable but simple: a single answer is an anecdote, not a data point. It is one draw from a distribution. It is not false, exactly. It just carries almost no information. Reporting it upward as a finding is reporting one coin flip as evidence about the coin.

This is where internal reporting quietly falls apart. Someone checks on a Tuesday, sees the brand named, and tells the room the content work is landing. Someone else checks the following week, sees nothing, and the room concludes visibility has collapsed. Neither event happened. The distribution did not move. It was sampled twice, and the samples disagreed, which is what samples do.

What carries information is repetition plus direction. Run the same prompt many times and you get a rate: how often, across all those answers, the brand appears. A rate is a measurement. Compare it against the rate a month ago, on the same prompts in the same language, and you get the only artefact worth showing anyone: a trend line.

How to separate noise from real movement

This requires no statistical sophistication, only discipline about what you hold still.

• Freeze the prompt set. Rewrite the questions between readings and you have changed the instrument as well as the measurement. When a better phrasing occurs to you, add it as a new prompt and keep the old one running.
• Freeze the language. The same question in English and in German is not the same question, because the models draw on different bodies of text. Never compare across languages and call it a trend.
• Run each prompt more than once. A result built on a single execution is a coin flip with a chart around it. Several executions, aggregated, turn an anecdote into a rate.
• Read direction across several readings, never the gap between two. The difference between two noisy samples is mostly noise. Several readings sketch a shape, and the shape is the signal.
• Keep competitors in the frame as a control. If your appearance rate falls and so does everyone else's on that platform, something changed at the platform, not at your company.

What a sampling cadence honestly gives you

Here we have to be straight about our own product, because the category is not otherwise honest about this. PSentry runs your frozen prompt set across ChatGPT, Claude, Gemini and Perplexity, in each language you sell in, and records where you were named, where you were cited, and who was recommended in your place. It runs on its own cadence, twice inside a month, rather than watching continuously, because a rate is built from repeated draws, not a single anxious refresh. What comes back is a reading, not an intervention, and it carries no guarantee: the sampling changes nothing about what any model actually says.

Twice a month is a sampling cadence, not surveillance. It tells you whether the trend is moving and in which direction. It will not tell you, on the afternoon it happens, that a vendor rewrote its instruction layer. If a competitor's placement surges on a Thursday, you learn about it at the next scan, not that evening. We do not sell instant alerts and we are not going to invent them, because an alert fired on a single noisy reading is a notification about nothing, dressed up as urgency.

False triumph and false panic

False triumph: you shipped a comparison page, asked the question that afternoon, saw your name, and concluded the page worked. It may have. It may equally have been the run where the sampling went your way. The honest test is whether your appearance rate across the next several readings is higher than across the previous several. That takes weeks, and it is the only version of the claim that survives someone asking how you know.

False panic: your brand vanished from one answer and a meeting got scheduled. Before that meeting, ask three things. Was the prompt identical. Was the language identical. Was it a single execution. If the answer to any of them is yes, there is nothing to discuss yet, and the useful action is to wait for the next reading rather than rewrite the website on the strength of one sentence.

The discipline this field demands is that you stop reading individual answers as verdicts. They are samples, and only the shape they make over time means anything.

Frequently Asked Questions

If the answer changes every time, is measuring it pointless?

No, but measuring it once is. Volatility does not make a system unmeasurable, it makes it unmeasurable in a single observation. Weather is volatile and perfectly measurable. What you are estimating is a rate of appearance, and a rate needs many observations before it settles.

How many runs of a prompt are enough?

There is no clean threshold, and anyone quoting one precisely is guessing. The practical rule is that you need enough runs that adding more stops changing the picture. Until then, treat prompts where you appear only sometimes as genuinely borderline.

Can I remove the randomness by setting temperature to zero?

Not in the assistants your buyers actually use, which expose no such control. Even through an API, low-temperature settings reduce variation without guaranteeing identical output, and they do nothing about the other sources of movement: an updated model, a rewritten instruction layer, a retrieval step that fetched different pages. You would be pinning down one variable out of several.

Does volatility hit all four platforms the same way?

No. Platforms that retrieve live pages before answering inherit the instability of the web on top of their own sampling, so they move more. Answers drawn mostly from what a model absorbed during training shift in steps when the model is updated, rather than drifting continuously.

Is a brand mentioned inconsistently better off than one never mentioned?

Better, and the distinction matters. Inconsistent mentions mean the model treats you as a plausible answer without being confident about it, which is a different position from sitting outside the candidate set entirely. Those flickering prompts are where change in either direction shows up first.