~ / guides / benchmark contamination

Benchmark contamination: what the Artificial Analysis v5 drops reveal

On 27 October 2026 Artificial Analysis published Intelligence Index v5. Sixty percent of its weight now comes from private, held-answer tests that no vendor can train against. Several models that led the public boards fell hard, and the size of each fall tracked how loud the model’s launch claims had been.

What v5 changed

Every removed public test was replaced by something a vendor cannot prepare for.

60% private weight

The majority of the headline score now comes from test sets whose questions and answers are held back by the benchmark owner.

source: Artificial Analysis

FrontierCode 2.0

132 private engineering tasks graded on whether the repository maintainer actually merged the model’s pull request.

new in v5

Terminal-Bench-Science

70 research-engineering tasks in a terminal. Most open-weight models score under 10%.

new in v5

Retired and hardened

GDPval-AA v2.1, SciCode and AA-LCR v1.1 were removed; HLE-Diamond, AA-Omniscience v1.1 and CritPt v2 hard were upgraded.

four categories, equal weight

The drops, ranked by size

Score on the old public-heavy index versus score on v5. A bar this long is not noise: it is the distance between the number a vendor marketed and the number a private test produced.

Xiaomi MiMo-V2.6-Pro−16.3
Z.ai GLM-5.3−13
Moonshot Kimi K3.1 Pro−9

Bar length is proportional to the point drop. MiMo-V2.6-Pro: 46.32 claimed on v4.3 to 30.0 measured on v5. GLM-5.3: 45 to 32. Kimi K3.1 Pro: 55 reported on v4.x to 46 on v5. See the caveat before treating any of this as proof of a general collapse.

Launch hype versus measured reality

The same three models, in the order of their drop, with what their makers said at release.

ModelThe launch claimOld indexv5Drop
MiMo-V2.6-Pro (Xiaomi)September launch page reported a 46.32 v4.3 score and marketed the model as the strongest open-source option46.32 (vendor-reported)30.0−16.3
GLM-5.3 (Z.ai)Led the open-weight conversation on v4.3 with a score of 454532−13
Kimi K3.1 Pro (Moonshot)Shipped 24 October at a reported 55 on the v4.x scale, sold as open weights55 (reported)46−9

Xiaomi carries two extra marks: on 7 October Vals AI announced an independent audit of the evaluation environment behind Xiaomi’s self-reported numbers, and the company is separately inside the “Claude probe” distillation investigation. Moonshot’s K3.1 Pro was labelled open-weight on 24 October; as of 27 October the weight set had not been published.

Where v5 actually landed everyone

Best setting per model, as published by Artificial Analysis.

Modelv5 scoreNote
Claude Opus 5.5 (max)58index leader
GPT-6 Astra (max)57
GPT-6.1 Sol (max)55
Gemini 4 Argon (high)50developer access 13 October
Kimi K3.1 Pro46best open-weight-labelled score; weights unpublished on 27 October
Claude Haiku 5.5 (max)36the small closed model
DeepSeek V4.1 Flash 1017 (max)32best open-weight model with published weights
GLM-5.3 (max)32tied with DeepSeek

The two best released open-weight models sit 26 points behind the leader and below Claude’s small Haiku tier. On v4.3 the open-weight gap to the leader was 3 points; on v5 it is 12.

The private tests are where the field splits

These are the sub-results the old public boards never showed.

TestBest closedBest open
FrontierCode 2.0 (132 private tasks, merged-PR grading)Opus 5.5: 46% (Astra, Terra and Sonnet 5.5 at 42%)DeepSeek V4.1 Flash: 28%
Terminal-Bench-Science (70 tasks)GPT-6 Astra: 63% (Opus 59%, Sol 58%)Kimi: 34% (DeepSeek 13%, Haiku 20% for scale; most open-weight models under 10%)
HLE-DiamondOpus 5.5: 68% (Astra 64%, Fable 5.1 60%)Kimi: 37%
CritPt v2 hardGPT-6 Astra: 72% (Sol 69%, Opus 61%)Kimi: 45%

The honest caveat

Read this before sharing the chart.

Artificial Analysis’s own warning: the widened gap “reflects this particular redesigned benchmark rather than proof Chinese open models broadly fell behind.” v4 and v5 are different instruments: 60% of the weight moved, tests were added and removed, and the scoring of effort settings changed. Part of every drop above is the ruler being re-cut, not the model being smaller. What the pattern still shows is narrower and harder to wave away: the models that led the public v4 boards with the loudest launch marketing lost the most points the moment the majority of the score moved behind held answers. That is consistent with training toward the visible test, and it is exactly why a private split matters.

The commercial loop this exposes

One release cycle, four steps, repeated by every lab.

1. Train on the visible test

Public benchmarks are downloadable. A lab that optimises post-training against them raises its public-board score without raising general capability. The board becomes an exam the model has sat before.

exam-style pretraining

2. Launch on the number

Marketing leads with the benchmark score: “strongest open source”, “frontier-class”. The number is real on the test it was taken on.

press amplifies

3. The private test re-measures

Held-answer sets cannot be prepared for. Scores fall, and the fall is largest where the claim was loudest.

reality check

4. Next launch promises the fix

The cycle restarts with a new model and a new leaderboard claim. Meanwhile the weights labelled “open” on day one sometimes stay unpublished, as with Kimi K3.1 Pro.

repeat

The same v5 release also priced the frontier honestly: Opus 5.5 costs $10.04 per completed task, Astra $6.24, Sol $1.27; the top ten span 13x in cost per task ($0.96 for GPT-6.1 Terra to $13.04 for Claude Fable 5.1). Median output tokens per task rose from 35k to 59k, and effort settings now move scores more than model choice did a year ago: Opus 5.5 runs 35 on low effort and 58 on max.

What to do with this as a buyer

Three rules that survive every benchmark relaunch.

Sources

All figures above as published by the named source.

See the self-claim badges on the leaderboard How AgentRanks scores News log