Summary
According to a post on X by @ArtificialAnlys, the Artificial Analysis Intelligence Index v5 now uses private test sets or held-out answers for 60% of its benchmarks. The update adds Cognition's FrontierCode 2.0 and Terminal-Bench-Science, replaces Humanity's Last Exam with HLE-Diamond, upgrades AA-Omniscience and CritPt v2, and drops GDPval-AA. Artificial Analysis says open-weight models are falling further behind than it previously measured. Separately, bookwormengr disputed a related analysis by arguing about Kimi 3.1 scores, though the snippet is cut off before the full claim.
AI-assisted summary based on the listed source.
What happened
60% of the v5 index now uses private test sets or held-out answers: it adds Cognition's FrontierCode 2.0 and Terminal-Bench-Science, replaces Humanity's Last Exam with HLE-Diamond, upgrades AA-Omniscience and CritPt v2, and drops GDPval-AA. Artificial Analysis says open-weight models are falling further behind...
Why it matters
Developers and businesses comparing open-weight and closed models will need to weigh how much of the index now relies on private tests that outside parties cannot inspect. Investors and builders tracking the gap between open and proprietary models should watch whether future index updates keep showing open-weight models trailing. The disputed Kimi 3.1 scores suggest the new rankings may still be contested.
Signal Intelligence
Signal Strength 91%
Technical label WATCH
Public Interest 32
Category OPEN SOURCE
Reader Depth TECHNICAL
Signal Strength reflects source quality, relevance, freshness and evidence. Public Interest helps organize discovery; it is not proof of truth.
Public Interest components
Recognizable Entity Score 0
Practical Impact Score 26
Novelty Interest Score 94
Consequence Score 34
Curiosity Score 0
Shareability Score 30
Why this is here
VQV surfaced this signal because it is recent, relevant to Open Source LLMs, connected to Scobleizer News Pulse (X).
