The Average Went Up. One Real Skill Went Down on Purpose.平均分涨了，一项真本事却被刻意调低了。

On October 23, a Chinese lab's model jumped 11.5 points on a widely watched "intelligence index" and landed fourth in the world, a hair behind the strongest Western systems. The headline wrote itself: the gap is closing. But a single average can be lifted by spending compute to tune for the test, and in the very same run, this model was made deliberately worse at one real job. That combination is the whole lesson. The score is real, but it is not a verdict on the model. It bundles genuine gains, a deliberate cut, and a lot of compute spent tuning for the test into one number — and then hides which is which.

The model is Kimi K3.1 Pro, from Moonshot AI. On Artificial Analysis's composite Intelligence Index it scored 55.1, up from 43.6 for July's Kimi K3, ranking fourth overall behind Claude Opus 5.5 (around 58), Claude Sonnet 5.5 (56) and Muse Wonder Full (56). Weights are promised open by November 4. If that holds, it would be the highest-scoring open-weights model available. You can see why the story became "China's open model is closing in on the frontier." It is a clean, rankable, one-number story, and one-number stories travel.

The reasonable conclusion, stated fairly

Before taking it apart, give the default read its due. Artificial Analysis is an independent evaluator, not the vendor. The Index is public, the methodology is disclosed, and the jump is large: eleven and a half points is not noise. A reader who glances at the leaderboard and concludes "Kimi got a lot better and is now near the top" is not being lazy. They are doing exactly what the leaderboard is built to let them do — compare models on one axis without reading three technical reports. The problem is not that the number is fake. The problem is what a single number can and cannot carry.

What an "intelligence index" actually is

An "intelligence index" is not a measurement of intelligence. It is an average. A lab's model is run against a basket of separate tests — coding in a terminal, office-document reasoning, factual recall, agentic task completion, and so on — each produces a score, and the index is roughly the mean of those scores rolled into one figure. That design is a feature: it smooths out any single test's quirks and gives you a tier at a glance. But an average has a cost built into it. It throws away the shape of the distribution. Two models with the same index can be strong and weak at completely different things, and the number will never tell you which.

Now add the second fact, the one the headline skips. Kimi K3.1 Pro is not a new model trained from scratch. It is July's K3, the same base, put through much heavier post-training. Post-training is the stage after the expensive pre-training run: reinforcement learning and preference tuning that shape how the model answers rather than what it fundamentally knows. And here is the mechanism that matters: post-training compute raises scores on precisely the behaviors you point it at. If the behaviors you point it at are the ones the benchmarks reward, the benchmark scores go up. That is not cheating, and it is not empty: the same process can also make a model genuinely better at real work. It is what the tool is for. But it means a large jump on a composite, on a base model you already shipped three months ago, is strong evidence that a lab spent compute optimizing toward the test distribution. It is not, by itself, proof that the whole model got broadly smarter — only that the measured behaviors did.

So the same event supports two very different readings. "The model got much better" and "the lab got much better at the test" both predict a higher index. The index alone cannot separate them. You need a second piece of evidence that only one of those stories can explain.

The tell is in the lab's own disclosure

Moonshot handed us that second piece of evidence itself, and to its credit did not hide it. In the same post-training run that lifted the index 11.5 points, the company deliberately removed cybersecurity-related material from the training data. On its internal exploit-development test, a checkpoint before the removal solved 90.5% of tasks; the shipped model solves 81.0%. Moonshot's stated reason is a safety one: it does not want an openly downloadable model to "push the absolute frontier of offensive cyber capabilities," and says the shipped model stays behind the proprietary systems that have been public for at least three months. Independent coverage confirms the cut was deliberate and that cyber performance now trails the top Western models by design.

Hold those two numbers next to each other. One number — the composite — went up 11.5 points. Another number — a specific, real capability — was moved down almost ten points, on purpose, in the same run. That is the tell. In a single run, the average and a named skill moved in opposite directions: the composite up, the exploit score down. An average that can rise while a real capability is deliberately pushed down is not a proxy for that capability — or for any single one. It is a blend, and a blend the lab gets to compose. That is the cleanest proof that "index rank" and "can do a given job" are different measurements, not two names for the same thing.

And notice who controls both ends. The lab decides the training mix — what to add, what to strip out. The lab also decides which single number it puts in the press release. Buyers, meanwhile, rank models by that one number. When the party being measured controls both the inputs to the test and the framing of the result, the test drifts toward measuring their effort and their priorities, not your task.

So what does the number actually tell you

Here is where honesty cuts both ways, because the contrarian reading can be overstated. Several of Kimi's gains are real, independently measured, and broad. Artificial Analysis reports its terminal-based coding score as the highest of any model it has tested. On a demanding office-document reasoning benchmark it beats both Claude Opus 5.5 and the strongest GPT system. Its hallucination rate fell by roughly a fifth. For a developer doing agentic coding or document-heavy work, this is a genuinely stronger tool than K3, and at a third lower price. The composite is not pure theatre.

But that is exactly the point, not a rebuttal to it. The index told you Kimi rose to fourth. It did not tell you that the rise is concentrated in coding and office tasks, that factual reliability improved, and that one safety-relevant skill was cut. You only learned the shape by reading underneath the number. The composite gave you a tier on a shared test. It could not tell you whether the model will do your task, and it actively hid a domain the vendor weakened. A reader who stopped at "fourth, near the frontier" walked away with a true ranking and a false picture.

This is the durable rule, and it is not about Kimi or about China. Any aggregate score — a model leaderboard, a credit score, a single company KPI, a school's average test result — is a mean that rewards whoever optimizes for it, and conceals the distribution underneath. The more valuable the rank becomes, the harder the measured party works to climb it, and the wider the gap grows between "ranks high" and "does the specific thing you need." A composite is a map drawn at low resolution. It is useful for seeing the terrain. It is useless for finding your own house.

What to do when the weights open

The practical move follows directly. When Kimi's weights go open on November 4, do not pick it, or pass on it, based on its index rank. Run it on your own work — your codebase, your documents, your actual edge cases — and read the component scores, not the average. If your task is agentic coding, the high terminal-bench number is informative and you should weight it. If your task touches anything security-adjacent, the composite is not just uninformative, it points the wrong way, because the one domain the vendor cut is the one you care about.

Generalize it past this model. When a tool's headline is a jump on one aggregate score, treat the score as a question, not an answer. Ask three things. What is this number an average of? Which component actually maps to the job I need done? And who chose both the inputs and the framing — do they have a reason to move some parts and not others? Those questions cost you ten minutes. Anchoring on the rank can cost you a quarter of building on the wrong foundation.

What would change this read

A judgment is only worth writing if you can say what would overturn it. Here it is. When the open weights ship on November 4, independent evaluators — Artificial Analysis's per-benchmark pages, third-party suites run by people with no stake in the launch — will be able to test the model directly. If those re-runs show Kimi K3.1 Pro improving roughly in step with its 11.5-point jump across domains that the index does not specifically weight, with the deliberate cybersecurity cut standing out as the only regression, then the composite did track broad capability here, and the "an average hides a cut" reading weakens to "an average mostly tracked, with one disclosed exception." The check-by date is January 31, 2027.

Conversely, if independent testing finds the gains clustered tightly in exactly the behaviors the index rewards, with flat or declining performance in domains it ignores, the reading holds and hardens. Either way, the test is the same one you should apply to the model itself: do not trust the average. Go read the parts.