The Benchmarks You Trusted to Pick Your AI Model Are Broken
By Joey
The SWE-Bench Pro paper, presented at ICML 2026 by a Princeton-led team, ran today’s best coding models against 1,865 real-world software engineering problems — including 18 proprietary codebases acquired from startups under formal partnership agreements specifically so they couldn’t appear in training data. Models that score above 70% on the standard SWE-bench Verified benchmark fell below 45% on the public SWE-bench Pro problems, and below 20% on the proprietary ones. Same models. Problems they’d never seen. An entirely different picture.
This is the clearest illustration yet of a problem that’s been building for two years: the benchmarks most people use to pick AI models have quietly stopped measuring what they’re supposed to measure.
Which benchmarks are actually dead in 2026
The independent reference benchmarkingagents.com, which indexes 17 active and saturated benchmarks against their official leaderboards, is unambiguous: MMLU, HumanEval, and MBPP are saturated. Frontier models cluster near the ceiling on all three — MMLU above 90%, HumanEval above 97% — which makes them “useless for comparing current models against each other.” Their words, not mine.
Saturation is one failure mode. HumanEval suffers documented training data contamination — the 164 Python problems are publicly available and almost certainly in every major model’s training corpus. Then there’s the problem the datavlab.ai benchmark analysis from April 2026 identifies as specific to SWE-bench: scores vary by 25 percentage points based on the scaffolding wrapped around the model. Prompt structure, retry logic, code execution feedback — change the evaluation harness, and the same model moves 25 points on the leaderboard. “Comparing two models’ SWE-Bench scores from different sources is often comparing two different things.”
Vendor-reported scores don’t usually tell you which scaffolding they used.
| Benchmark | Core problem | What to use instead |
|---|---|---|
| MMLU | Saturated above 90% for frontier models | MMLU-Pro |
| HumanEval | Saturated + training contamination | LiveCodeBench |
| MBPP | Saturated above 97% | SWE-bench Verified |
| SWE-bench Verified | ±25pp scaffold-dependent | Demand controlled, same-harness comparisons |
| GPQA Diamond | Approaching ceiling at very top | Still discriminates in the 60–90% range |
| SWE-bench Pro (ICML 2026) | Multi-file, proprietary, reviewed | Hardest real-engineering signal available |
The model cards leading with MMLU and HumanEval are advertising into the past.
What the SWE-bench Pro collapse actually means
The gap between >70% on SWE-bench Verified and <20% on SWE-bench Pro’s proprietary problems isn’t noise. The Princeton team identified exactly what changed: problem length (12 lines → 107 lines on average, 1 file → 4 files), codebase novelty (open source vs. startup proprietary repos), and mandatory review by a professional software engineer who confirmed each problem is actually solvable.
“The best models solve fewer than 45% of the public problems and fewer than 20% of the proprietary ones, compared to over 70% on the older SWE-bench Verified benchmark.” — SWE-Bench Pro, ICML 2026
That collapse tells you something specific: a meaningful share of what the older benchmark measures is exposure to the training corpus, not reasoning across a real codebase. The models that seem nearly interchangeable on SWE-bench Verified diverge sharply on problems they can’t have seen.
I’ve run Claude Opus 4.6 and GPT-5.4 side-by-side on multi-file refactoring tasks — four or five interconnected modules, shared state, reasoning across imports. The ranking from the SWE-bench Verified leaderboard hasn’t held consistently. The tasks closest to what SWE-bench Pro tests — longer, multi-file, no obvious analogues to common training patterns — are exactly where the standard ordering surprises me most. This isn’t a controlled experiment, but it maps directly onto what the ICML paper would predict: the leaderboard reflects something other than the task you’re actually trying to do.
How to pick a model when the scores are noise
The benchmarkingagents.com reference provides a clear framework. For coding and agentic work: SWE-bench Verified (with same-harness comparisons) and LiveCodeBench, which sources problems from LeetCode and AtCoder problems released after each model’s training cutoff, making contamination structurally unlikely rather than just hopefully absent.
For reasoning-heavy work — research tools, scientific analysis, complex multi-step pipelines — GPQA Diamond remains a meaningful differentiator in the 60–90% range. The datavlab.ai April 2026 analysis puts Gemini 3.1 Pro at 94.3%, Claude Opus 4.6 at 91.3%, and GPT-5.3 Codex at 81%. For instruction-following in RAG pipelines or multi-agent systems, IFEval is more predictive than any knowledge benchmark; Kimi K2.5 leads at 94.0, Qwen 3.5 at 92.6.
For frontier differentiation, Humanity’s Last Exam (HLE, Scale AI 2025) is worth tracking now. It has real headroom — frontier models cluster around 50% — which is exactly the signal MMLU stopped providing after 2024.
The tunx.ai Frontier Models Tracker flags the shift clearly as of July 2026: labs have stopped competing on raw benchmark scores. The new competitive frontier is agentic task completion — multi-step workflows, tool use, computer-use benchmarks like OSWorld and WebArena. Extended reasoning is now a baseline feature across every top-tier closed model, not a differentiator. The fight moved to what models can do autonomously, not what they can answer.
None of that replaces an evaluation against your actual workload. The datavlab.ai analysis recommends 100–200 examples drawn from real production patterns, with a binary pass/fail rubric applied blind across two or three model candidates. One hundred examples is a small investment. Rebuilding production integrations around the wrong model six months after a better candidate releases is not.
When a model card includes MMLU but skips SWE-bench Verified, that’s data.
Frequently asked questions
Why do vendors still report MMLU and HumanEval if they’re saturated?
Because the numbers read well in procurement decks. MMLU above 90% sounds impressive to buyers who don’t know every frontier model is above 90%. HumanEval at 93% doesn’t signal anything useful when the problem set is contaminated and the ceiling is within reach of every major model. The benchmarkingagents.com reference is direct: “A model card that includes MMLU but omits SWE-bench Verified is a signal worth noting.” Selective reporting is a form of communication too.
Does Chatbot Arena Elo actually predict production quality?
For consumer-facing products where conversational quality matters, it’s one of the strongest available signals — millions of blind pairwise votes produce statistically robust rankings. For coding or agentic work, it misleads. A model can lead general Arena Elo while performing only modestly on SWE-bench Verified, because the vote population skews toward general conversation quality, not engineering task completion. For technical workloads, the coding-specific Arena Elo sub-ranking is the more relevant signal.
How contaminated is SWE-bench Verified, really?
Less than HumanEval — the GitHub issues it tests are real and varied, not a published problem set. But SWE-bench Pro exists precisely because contamination risk is real even there. The 18 proprietary repos in SWE-bench Pro are the cleanest test currently available: by construction, none of them can be in any model’s training data. The gap they revealed is the most honest number we have about what current models can actually do on software engineering they haven’t seen before.