ChatGPT's top pick is more a coin toss than a ranking.问 ChatGPT「哪个最好」,它的首选更像一次抽签,不是一份排名。

In a study published on October 27, the marketing firm BusySeed asked the same question twice and compared 172,624 pairs of answers across 172 categories. With the model version, the web-search setting and the wording all held constant, the brand it placed first matched only 25.4% of the time — three times in four, a different brand came out on top. Asking a week apart or a month apart barely changed that.
The mechanism is plain: a large model writes by sampling tokens probabilistically, so each time you ask, a different name can float to the front. That order reflects how often a brand appears in the training text plus sampling noise, not any judgment about which fits you. There is no stable ranking underneath being read out.
Second-order: as companies start paying to be the brand named first, they are chasing a target that is three-quarters noise; and anyone who treats the top answer as authoritative has handed the comparison step to a process that cannot compare.
The strongest counter: maybe only the wording moves and the set of brands named is stable. That holds for generating a shortlist, but this study measured the first pick, and people act on the first pick.
What it means for you: when you ask an AI for "the best X," treat the answer as a shortlist to check, not a verdict, and compare it yourself against the two or three criteria you actually care about. This judgment needs revising if a replication with the same design shows the top brand holding steady in a clear majority of paired answers.
营销公司 BusySeed 在 10 月 27 日公布的研究里,把同一个问题问两遍、比对了 172,624 对回答,覆盖 172 个品类:模型版本、联网设置、措辞都不变,可排在第一位的品牌只有 25.4% 两次一致——也就是说,四次里有三次,被它推到最前面的是另一个品牌。隔一周问和隔一个月问,差别几乎没有。
机制很直白:大模型是按概率逐字采样生成文字的,同一个问题每跑一次,浮到最前面的那个名字就可能不一样。这个顺序反映的是语料里谁出现得多、加上采样的随机,而不是对「谁更适合你」做过评估。换句话说,底下根本没有一份稳定的排名在被读出来。
二阶效应:当商家开始花钱去抢「被第一个念到」的位置,他们追的其实是一个七成是噪声的目标;而把首选当权威的人,等于把「比较」这一步,交给了一个做不了比较的程序。
最强的反方:也许变的只是措辞,被点到的那一批品牌是稳定的。这对「让它列个候选清单」成立,但这项研究量的正是第一名——而人们是照着第一名行动的。
对你意味着什么:问 AI「最好的 X 是什么」,把它的回答当成一张要去核对的候选清单,不是裁决;拿你真正在意的两三个标准,自己再比一遍。什么会推翻这个判断:如果有人用同样的设计复跑,第一名在明显多数的成对回答里都保持不变,这条判断就要修正。
Built on the WAKENEX concept:所用的 WAKENEX 概念: Match the feedback to the question反馈分工:不要用一种指标替代其他问题的答案
Sources来源
Add WAKENEX as a preferred source on Google在 Google 把 WAKENEX 设为首选来源


