An AI visibility report can improve because the questions changed. If the team adds friendlier buying scenarios, drops awkward markets or rewrites the tracked wording, the next score answers a different question. Presenting that movement as a win invites the client to fund work the comparison has not tested.
Keep a stable comparison panel, and explore new questions separately. Repeating checks is useful. Preserving what those checks mean is the harder job.
Repetition fixes only part of the problem
Running the same buyer question again helps reveal whether a brand appears consistently in that particular setup. Comparing rates over weeks is more informative than celebrating an isolated answer. But collecting more answers to one chosen sentence does not establish how customers asking differently will fare.
An Unusual research-team preprint compared repeated identical questions with reworded commercial questions in a single-day API experiment. Its recommendation-set overlap score was 0.288 for cosmetic rewordings, against 0.50–0.61 for identical-question reruns. A score of one would mean identical sets; zero would mean no shared brands.
That finding challenges the comfortable response of simply running the tracker more often. The same wording can define a useful narrow benchmark while remaining a poor stand-in for all the ways buyers express a need.
Changing a city, language or company size needs a further distinction. Those edits can change who the buyer is or what they need, rather than merely paraphrasing the sentence. A weak result for one location may be a real market gap. Averaging it with a stronger location can conceal the work instead of measuring improvement.
Keep a comparison panel and an exploration panel
For the comparison panel, retain the exact wording, engine, buyer context and inclusion rule. Record when the check ran and which product or model was observable. Compare like with like within each engine, and preserve topic, persona and buying-stage breakdowns when reviewing weekly or monthly results.
Use a separate exploration panel for new phrasings, additional locations and emerging buyer questions. Its job is to discover missed opportunities. It should not silently replace the questions against which the team promised to demonstrate progress.
When the tracked mix genuinely needs updating, show the old and new panels alongside each other for a comparison period and label the break. A larger question library is useful coverage, but its size alone says nothing about whether visibility improved.
This is the strongest case for a fixed panel: it gives the team a repeatable operational question. The mistake is naming that narrow result as the brand's overall share of AI attention.
Do not buy a universal run count
How many repeats are enough depends on the comparison and the size of change worth acting on. A stable average across very different questions is not the same test as detecting a small improvement for one question.
The corrected second version of the Dice Roll Method preprint withdrew its earlier fixed repetition tiers and recommends a pilot-specific design. That is a good reason to ask a measurement provider how it established its run budget, rather than treating five, ten or fifteen checks as a universal certificate of reliability.
Ask the provider to distinguish ordinary rerun variation from wording sensitivity, retain the underlying answers and explain when the observed difference is large enough for the proposed decision. A weekly reporting cadence is a work schedule, not proof that the sample can detect the effect being sold.
Investigate the answer before rewriting the site
When comparable results weaken, inspect the cited pages as well as whether the brand appears. Did a different directory or comparison page enter the answer? Did the question move to another buyer segment? Record observable competitor profile or review changes when relevant, but leave them as possible explanations if they were not monitored.
A changed source set is a lead to investigate, not proof that the business deteriorated or that a particular page edit will restore mentions. Do not turn every disappearance into another content assignment.
The research here is bounded: these preprints have commercial affiliations, limited datasets and model settings that do not establish a universal threshold for today's consumer assistants. Nor does a before-and-after mention rate isolate the effect of your work.
Before approving the next optimisation budget, ask the team to reproduce the gain against the unchanged panel and show what the exploration panel adds. If the improvement disappears when the old questions return, the honest deliverable is a revised measurement plan—not a victory chart.