How Much Do ChatGPT's Answers Change Between Runs?

What the numbers show

Across 185 original questions, the domains cited in one run and another overlapped with a mean Jaccard of 0.299 (185 questions).

The mean share of brands named in all 3 runs was 0.241 per question (157 questions with brands). Pooled across questions, 238 of 1199 brands appeared in all 3 runs (0.198). Of the brands named in the first run, 51.3% also appeared in the second (142 questions). The figures by question type are in the chart and the table below it. The "worth hiring" cell is thin (n=8).

A brand's mention rate across 10 questions would have to move by 19.3 points on a single run, or 11.2 points on an average of 3 runs, to stand out from run-to-run noise (82 eligible brands in the original set).

The brand-seeking supplement has 84 questions and 252 answers. 100% of those answers named a brand, against 56% of the original answers of the same three types (123 of 219).

Method

Peak Answer asked ChatGPT 185 original questions and 84 brand-seeking questions in October 2026. Each question was asked 3 times in separate calls, which gives 807 answers. The answers came from DataForSEO's ChatGPT scraper, run from the United States, with web search forced.

Citation overlap (a) uses the domains each answer cited, one set per answer. Jaccard is the share of domains two runs have in common out of all the domains either run cited. It is calculated for the first and second, first and third, and second and third runs, then averaged.

Brands in all runs (b) takes the brands named in any of the 3 runs for a question and gives the share named in all 3, as a per-question mean and as pooled counts. First-run brands also in the second run (c) is the share of brands named in the first run that were also named in the second. Brand names were extracted by Sonnet 5.5, one call per question, from the answer text.

Minimum detectable change comes from 400 random draws of 10 questions, comparing two different runs on the same questions, for the 82 eligible brands in the original set.

Limits

• Cells marked * have fewer than 10 observations.
• A pair of runs where just one run cited anything scores zero for citation overlap.
• Questions with no brand in any run are left out of the brand measure, and questions with no brand in the first run are left out of the first-run-to-second-run measure, so the number of questions differs by measure.
• Brand names were extracted by Sonnet 5.5, one call per question, from the answer text.
• The original and brand-seeking sets are different questions, so comparing them shows how the sets behaved, not what the wording caused.
• The minimum detectable change for brand-seeking questions alone is higher (27.7 points for a single run, 16.0 for an average of 3 runs). It is a rough figure, because just 12 questions per target are available and the 10-question draws overlap heavily.
• The original "worth hiring" results rest on few answers that named any brand (n=8) and are thin.
• The original figures for the three advice-style types rest on the few answers that named any brand, so the comparison with the brand-seeking set is not like for like.
• One engine and one scraper. The first pass on a different scraper gave 0.247 for citation overlap and 19.4% for brands in all runs, against 0.299 and 24.1% here.

Download the data

The CSV files hold the figures behind the tables. Units are in the column names.

• Run-to-run stability by group (23 rows)
• Minimum detectable change in brand mention rate (12 rows)

How to cite this study

Peak Answer, "How Much ChatGPT Answers Change Between Runs", https://peakanswer.com/research/ai-answer-stability.

Related

Repeated runs over time are in the AI answer tracker.

Start with the free AI visibility audit.