WAKENEX觉醒联盟
Signal觉醒信号Note短评

ChatGPT's top pick is more a coin toss than a ranking.问 ChatGPT「哪个最好」,它的首选更像一次抽签,不是一份排名。

In a bright, sunlit turquoise-and-white plaza, a hand drops a plain blank card into the slot of a big glossy white answer machine; from its two output trays, two completely different gold trophies are both planted on the single gold top step of one small winners' podium, so two different champions come out of the same question. Bold white text reads: Ask twice, a different winner. A line of Chinese below reads: 问两遍,冠军就换了.
In a bright, sunlit turquoise-and-white plaza, a hand drops a plain blank card into the slot of a big glossy white answer machine; from its two output trays, two completely different gold trophies are both planted on the single gold top step of one small winners' podium, so two different champions come out of the same question. Bold white text reads: Ask twice, a different winner. A line of Chinese below reads: 问两遍,冠军就换了.明亮的青绿与白色广场上,一只手把一张空白卡片投进一台白色大机器的投入口;机器的两个出口各推出一座完全不同的金色奖杯,两座奖杯同时被放在同一个金色领奖台的最高一级上——同一个问题,冒出两个不同的冠军。图上大字写着:Ask twice, a different winner.,下面一行中文:问两遍,冠军就换了。Image: WAKENEX图片:WAKENEX

In a study published on October 27, the marketing firm BusySeed asked the same question twice and compared 172,624 pairs of answers across 172 categories. With the model version, the web-search setting and the wording all held constant, the brand it placed first matched only 25.4% of the time — three times in four, a different brand came out on top. Asking a week apart or a month apart barely changed that.

The mechanism is plain: a large model writes by sampling tokens probabilistically, so each time you ask, a different name can float to the front. That order reflects how often a brand appears in the training text plus sampling noise, not any judgment about which fits you. There is no stable ranking underneath being read out.

Second-order: as companies start paying to be the brand named first, they are chasing a target that is three-quarters noise; and anyone who treats the top answer as authoritative has handed the comparison step to a process that cannot compare.

The strongest counter: maybe only the wording moves and the set of brands named is stable. That holds for generating a shortlist, but this study measured the first pick, and people act on the first pick.

What it means for you: when you ask an AI for "the best X," treat the answer as a shortlist to check, not a verdict, and compare it yourself against the two or three criteria you actually care about. This judgment needs revising if a replication with the same design shows the top brand holding steady in a clear majority of paired answers.

营销公司 BusySeed 在 10 月 27 日公布的研究里,把同一个问题问两遍、比对了 172,624 对回答,覆盖 172 个品类:模型版本、联网设置、措辞都不变,可排在第一位的品牌只有 25.4% 两次一致——也就是说,四次里有三次,被它推到最前面的是另一个品牌。隔一周问和隔一个月问,差别几乎没有。

机制很直白:大模型是按概率逐字采样生成文字的,同一个问题每跑一次,浮到最前面的那个名字就可能不一样。这个顺序反映的是语料里谁出现得多、加上采样的随机,而不是对「谁更适合你」做过评估。换句话说,底下根本没有一份稳定的排名在被读出来。

二阶效应:当商家开始花钱去抢「被第一个念到」的位置,他们追的其实是一个七成是噪声的目标;而把首选当权威的人,等于把「比较」这一步,交给了一个做不了比较的程序。

最强的反方:也许变的只是措辞,被点到的那一批品牌是稳定的。这对「让它列个候选清单」成立,但这项研究量的正是第一名——而人们是照着第一名行动的。

对你意味着什么:问 AI「最好的 X 是什么」,把它的回答当成一张要去核对的候选清单,不是裁决;拿你真正在意的两三个标准,自己再比一遍。什么会推翻这个判断:如果有人用同样的设计复跑,第一名在明显多数的成对回答里都保持不变,这条判断就要修正。

Sources来源

  1. Source: bnonews.com/index.php/2026/10/ask-chatgpt-the-same-question-twice-a…来源:bnonews.com/index.php/2026/10/ask-chatgpt-the-same-question-twice-a…bnonews.com ↗