You ask ChatGPT for the best brands in your category. It names you. You celebrate. Then a colleague asks the same question and you're gone. Neither check was wrong. AI answers change every time you ask. So one check tells you almost nothing. This guide shows you a simple fix: ask each question at least 5 times, count how often you're named, and only trust a change when it's bigger than the noise.
Source: a 2026 repeat-test of 10 buying questions, asked 8 times each, logged out, in one hour.
Where to start
Write 5 neutral buying questions
In the same repeat-test, ChatGPT kept the same brand order in only 5.2% of answer pairs. It put the same brand first 60.0% of the time. Gemini repeated its exact list in 4.8% of pairs, with 51.2% brand overlap. Perplexity was steadier: 15.4% exact lists, 68.3% brand overlap and 85.2% overlap in cited sites. Every engine moved. None stood still.
Why? Three main reasons. Models pick words with some randomness. OpenAI even documents a temperature setting for this in its API. With live web search on, the assistant turns your question into searches, and the pages it finds shift. Answer length also varies, so a short answer has room for fewer brands. Memory, location, wording and language matter too.
So start with fair questions. Write them the way a buyer would. Leave your brand name out, or the assistant will name you because you asked.
Question set builder Example numbers
Runs in your browser. Nothing is sent anywhere. Describe your category and buyer.
Ask each question 5 times and tally the result
Now run the test. Ask each question at least 5 times in one engine. Each time, use a fresh chat. Test logged out, or in a Temporary Chat, so memory doesn't steer the answer. Keep the wording, country, language and engine the same for every run. Then tick each run that named your brand.
The tool turns your ticks into a share and a range. The range matters more than the number. It shows where your true share probably sits. Small samples give wide ranges. With 30 answers at a 50% share, the 95% range is roughly 17 points either side. That's why one check proves nothing.
5-run tally Example numbers
Runs in your browser. Nothing is sent anywhere. Tick each run where the answer named you. Your ticks are saved on this device only.
Track it over time and only trust real changes
Repeat the same test every week or every two weeks. Same questions, same engine, same settings. Then compare two periods. If your new share sits inside the old range, the change is probably noise. Treat it as real only when the two ranges don't overlap.
This saves you from two costly mistakes. You won't celebrate a lucky week. And you won't panic and rewrite your site after an unlucky one. If you want tighter ranges, add more runs or more questions.
Is the change real? Example numbers
Runs in your browser. Nothing is sent anywhere. Enter named answers and total answers for each period.
Why does ChatGPT give a different answer every time?
It picks words with some randomness, it may run new web searches each time, and answer length varies. Memory, location and wording also change the result.
Why leave my brand name out of the question?
If you name your brand, the assistant will almost always talk about it. That makes your share look far better than what a real buyer sees.
Is Perplexity more stable?
In the repeat-test, yes. It repeated its exact list in 15.4% of pairs and cited the same sites 85.2% of the time. But its answers still changed, so test it 5 times too.
What is the 95% range?
It's the span where your true share most likely sits, given how many answers you collected. More answers make it narrower.
Sources: A 2026 repeat-test of ChatGPT, Gemini and Perplexity (10 buying questions, 8 runs each); OpenAI API docs. Checked October 2026. Ranges use the 95% Wilson interval. Example brands (Acme, Northline, Kitebase, Orbly, Stride One) and numbers are made up.