Benchmark contamination: what the Artificial Analysis v5 drops reveal
On 27 October 2026 Artificial Analysis published Intelligence Index v5. Sixty percent of its weight now comes from private, held-answer tests that no vendor can train against. Several models that led the public boards fell hard, and the size of each fall tracked how loud the model’s launch claims had been.
What v5 changed
Every removed public test was replaced by something a vendor cannot prepare for.
60% private weight
The majority of the headline score now comes from test sets whose questions and answers are held back by the benchmark owner.
FrontierCode 2.0
132 private engineering tasks graded on whether the repository maintainer actually merged the model’s pull request.
Terminal-Bench-Science
70 research-engineering tasks in a terminal. Most open-weight models score under 10%.
Retired and hardened
GDPval-AA v2.1, SciCode and AA-LCR v1.1 were removed; HLE-Diamond, AA-Omniscience v1.1 and CritPt v2 hard were upgraded.
The drops, ranked by size
Score on the old public-heavy index versus score on v5. A bar this long is not noise: it is the distance between the number a vendor marketed and the number a private test produced.
Launch hype versus measured reality
The same three models, in the order of their drop, with what their makers said at release.
| Model | The launch claim | Old index | v5 | Drop |
|---|---|---|---|---|
| MiMo-V2.6-Pro (Xiaomi) | September launch page reported a 46.32 v4.3 score and marketed the model as the strongest open-source option | 46.32 (vendor-reported) | 30.0 | −16.3 |
| GLM-5.3 (Z.ai) | Led the open-weight conversation on v4.3 with a score of 45 | 45 | 32 | −13 |
| Kimi K3.1 Pro (Moonshot) | Shipped 24 October at a reported 55 on the v4.x scale, sold as open weights | 55 (reported) | 46 | −9 |
Xiaomi carries two extra marks: on 7 October Vals AI announced an independent audit of the evaluation environment behind Xiaomi’s self-reported numbers, and the company is separately inside the “Claude probe” distillation investigation. Moonshot’s K3.1 Pro was labelled open-weight on 24 October; as of 27 October the weight set had not been published.
Where v5 actually landed everyone
Best setting per model, as published by Artificial Analysis.
| Model | v5 score | Note |
|---|---|---|
| Claude Opus 5.5 (max) | 58 | index leader |
| GPT-6 Astra (max) | 57 | |
| GPT-6.1 Sol (max) | 55 | |
| Gemini 4 Argon (high) | 50 | developer access 13 October |
| Kimi K3.1 Pro | 46 | best open-weight-labelled score; weights unpublished on 27 October |
| Claude Haiku 5.5 (max) | 36 | the small closed model |
| DeepSeek V4.1 Flash 1017 (max) | 32 | best open-weight model with published weights |
| GLM-5.3 (max) | 32 | tied with DeepSeek |
The two best released open-weight models sit 26 points behind the leader and below Claude’s small Haiku tier. On v4.3 the open-weight gap to the leader was 3 points; on v5 it is 12.
The private tests are where the field splits
These are the sub-results the old public boards never showed.
| Test | Best closed | Best open |
|---|---|---|
| FrontierCode 2.0 (132 private tasks, merged-PR grading) | Opus 5.5: 46% (Astra, Terra and Sonnet 5.5 at 42%) | DeepSeek V4.1 Flash: 28% |
| Terminal-Bench-Science (70 tasks) | GPT-6 Astra: 63% (Opus 59%, Sol 58%) | Kimi: 34% (DeepSeek 13%, Haiku 20% for scale; most open-weight models under 10%) |
| HLE-Diamond | Opus 5.5: 68% (Astra 64%, Fable 5.1 60%) | Kimi: 37% |
| CritPt v2 hard | GPT-6 Astra: 72% (Sol 69%, Opus 61%) | Kimi: 45% |
The honest caveat
Read this before sharing the chart.
The commercial loop this exposes
One release cycle, four steps, repeated by every lab.
1. Train on the visible test
Public benchmarks are downloadable. A lab that optimises post-training against them raises its public-board score without raising general capability. The board becomes an exam the model has sat before.
2. Launch on the number
Marketing leads with the benchmark score: “strongest open source”, “frontier-class”. The number is real on the test it was taken on.
3. The private test re-measures
Held-answer sets cannot be prepared for. Scores fall, and the fall is largest where the claim was loudest.
4. Next launch promises the fix
The cycle restarts with a new model and a new leaderboard claim. Meanwhile the weights labelled “open” on day one sometimes stay unpublished, as with Kimi K3.1 Pro.
The same v5 release also priced the frontier honestly: Opus 5.5 costs $10.04 per completed task, Astra $6.24, Sol $1.27; the top ten span 13x in cost per task ($0.96 for GPT-6.1 Terra to $13.04 for Claude Fable 5.1). Median output tokens per task rose from 35k to 59k, and effort settings now move scores more than model choice did a year ago: Opus 5.5 runs 35 on low effort and 58 on max.
What to do with this as a buyer
Three rules that survive every benchmark relaunch.
- Treat a launch benchmark number as marketing. It is the maker’s figure on the maker’s best setting until an independent run says otherwise. AgentRanks badges every such row self claim on the leaderboard.
- Wait for the cross-check. A score that survives a second harness (the DeepSeek V4.1 Flash case: 74.2 on the maker’s card, 65.6 under Codex) is a measurement. One that only appears in its maker’s table is a demo.
- Watch the weights, not the licence page. “Open source” announced with no download link is a press release. Mistral Large 4 and Kimi K3.1 Pro both shipped API-first behind an open-weight promise.
Sources
All figures above as published by the named source.
- Artificial Analysis Intelligence Index v5 — methodology, private weighting, sub-scores, published 27 October 2026. Independent.
- MiMo-V2.6-Pro model page — current v5 score of 30.0. Independent.
- Xiaomi MiMo launch page — the 46.32 v4.3 claim and the strongest-open-source marketing. Official.
- Moonshot K3.1 Pro announcement — 24 October release, open-weight promise. Official.
- AA Progress Index v1 — published alongside v5.
See the self-claim badges on the leaderboard How AgentRanks scores News log