ChatGPT answered 185 original questions 3 times each. The domains two runs cited in common, as a share of all domains either run cited, averaged 0.299, and the mean share of brands named in all 3 runs was 0.241 (over 157 questions that named brands).
Questions asked
185
Original questions, each answered 3 times.
Cited domains in common between runs
0.299
Mean across 185 questions.
Brands named in all runs
0.241
Mean per question, over 157 of 185 questions that named a brand.
Brands named in all runs, pooled
238 of 1,199
Brands pooled across the same questions, a share of 0.198.
First-run brands also in the second run
51.3%
Over 142 questions that named a brand in the first run.
Minimum detectable change
19.3 points
For a single run. It is 11.2 points for an average of 3 runs.
Across 185 original questions, the domains cited in one run and another had 0.299 in common as a share of all domains either run cited, on average (185 questions).
The mean share of brands named in all 3 runs was 0.241 per question (157 questions with brands). Pooled across questions, 238 of 1,199 brands appeared in all 3 runs (0.198). Of the brands named in the first run, 51.3% also appeared in the second (142 questions). The figures by question type are in the chart and the table below it. The "worth hiring" cell is thin (n=8).
A brand's mention rate across 10 questions would have to move by 19.3 points on a single run, or 11.2 points on an average of 3 runs, to stand out from run-to-run noise (82 eligible brands in the original set).
The brand-seeking supplement has 84 questions and 252 answers. 100% of those answers named a brand, against 56% of the original answers of the same three types (123 of 219).
Bars show the mean share of brands named in all 3 runs per question, as a decimal, for each question type and for all original questions. "Worth hiring" is thin.
Mean share of brands named in every run, for each question type and for all questions.
| Group | Brands named in all 3 runs | Observations |
|---|---|---|
| best/top | 0.275 | 51 |
| best for a need | 0.207 | 38 |
| how to choose | 0.288 | 17 |
| cost | 0.204 | 26 |
| alternatives | 0.254 | 17 |
| worth hiring | 0.188 (thin) | 8 |
| All original questions | 0.241 | 157 |
The numbers behind the chart, with how many observations each group has.
| Group | Questions | (a) mean citation-domain Jaccard | (b) share of brands in all 3 runs, mean per question | (b) pooled brands | (c) run 1 brands also in run 2 |
|---|---|---|---|---|---|
| All original questions | 185 | 0.299 (n=185) | 0.241 (n=157) | 0.198 (238/1,199) | 0.513 (n=142) |
| Sub-niche typed questions | 140 | 0.291 (n=140) | 0.208 (n=124) | 0.161 (155/960) | 0.478 (n=113) |
| Parent questions | 35 | 0.301 (n=35) | 0.307 (n=23) | 0.305 (51/167) | 0.624 (n=19) |
| Controls | 10 | 0.394 (n=10) | 0.508 (n=10) | 0.444 (32/72) | 0.693 (n=10) |
How often the same brands were named across repeated runs, for all questions.
| Group | Questions | (a) mean citation-domain Jaccard | (b) share of brands in all 3 runs, mean per question | (b) pooled brands | (c) run 1 brands also in run 2 |
|---|---|---|---|---|---|
| best/top | 51 | 0.285 (n=51) | 0.275 (n=51) | 0.208 (106/509) | 0.477 (n=51) |
| best for a need | 38 | 0.241 (n=38) | 0.207 (n=38) | 0.150 (54/359) | 0.395 (n=38) |
| how to choose | 22 | 0.350 (n=22) | 0.288 (n=17) | 0.240 (18/75) | 0.637 (n=14) |
| cost | 30 | 0.317 (n=30) | 0.204 (n=26) | 0.183 (22/120) | 0.591 (n=22) |
| alternatives | 23 | 0.305 (n=23) | 0.254 (n=17) | 0.282 (35/124) | 0.611 (n=14) |
| worth hiring | 21 | 0.351 (n=21) | 0.188 (n=8) * | 0.250 (3/12) | 1.000 (n=3) * |
The same results split by question type.
* Fewer than 10 observations, so the cell is thin.
| Brand mention rate (avg of 3 runs, 10 questions) | Comparisons | SD of 1-run difference (pts) | 95th pct of abs diff (pts) | MDC 1 run (pts) | MDC 3 runs (pts) |
|---|---|---|---|---|---|
| All eligible brands | 98,400 | 9.9 | 20.0 | 19.3 | 11.2 |
| <20% | 74,787 | 9.2 | 20.0 | 18.1 | 10.5 |
| 20-49% | 19,194 | 11.3 | 20.0 | 22.2 | 12.8 |
| 50%+ | 4,419 | 12.9 | 30.0 | 25.2 | 14.6 |
The smallest change in brand mention rate the data can reliably detect.
Brand mention rate is the average of 3 runs over 10 questions.
Peak Answer asked ChatGPT 185 original questions and 84 brand-seeking questions in October 2026. Each question was asked 3 times in separate calls, which gives 807 answers. The answers came from simulated logged out ChatGPT browser sessions, run from the United States, with web search forced.
Citation overlap (a) uses the domains each answer cited, one set per answer. The in-common score is the share of domains two runs have in common out of all the domains either run cited; none in common scores the lowest and identical sets score the highest. It is calculated for the first and second, first and third, and second and third runs, then averaged.
Brands in all runs (b) takes the brands named in any of the 3 runs for a question and gives the share named in all 3, as a per-question mean and as pooled counts. First-run brands also in the second run (c) is the share of brands named in the first run that were also named in the second. Brand names were extracted by Sonnet 5.5, one call per question, from the answer text.
Minimum detectable change comes from 400 random draws of 10 questions, comparing two different runs on the same questions, for the 82 eligible brands in the original set.
The CSV files hold the figures behind the tables. Units are in the column names.
Peak Answer, "How Much ChatGPT Answers Change Between Runs", https://peakanswer.com/research/ai-answer-stability.
Repeated runs over time are in the AI answer tracker.
Start with the free AI visibility checker.