5/27/2026 at 1:12:19 AM
While this benchmark has interesting results, the "Contamination free" label only works for the initial release of the benchmark. It still has the same fundamental design issues of any other benchmark-- there's a single correct answer for tasks. It looks to be largely saturated upon release.What they did well: normalizing the harness to mini-swe-agent -- models should be able to generalize to different tools at this point. When they struggle to do that (like most Google models), they're unlikely to be useful in practice. And that kind of generalization is an inherent part of intelligence.
For a benchmark that scales, you need to remove the ceiling and provide environments with measurable goals that are NOT a single correct answer, and sufficiently complex evaluation criteria to scale well beyond the current frontier.
We do this by running multi-agent simulations with large action spaces at https://gertlabs.com/rankings.
We're still relatively unknown in the benchmarking space, but by rotating the pool of environments and ensuring the optimal strategies in the environments themselves are affected by other agents participating in the space, we expect we'll be able to resist contamination as major labs start investing more effort to climb the leaderboard. We've already seen Chinese labs taking an interest.
by gertlabs
5/27/2026 at 6:50:19 AM
The fact that claude and gpt 5.5 have nearly the same scores tells me your benchmark is not capturing a significant gap in capability between these two. What the linked page says about Claude is true in my experience: It frequently forgets important instructions and likes to take lazy shortcuts. Gpt by contrast is much more attentive and takes its time when needed to deliver a complete and robust solution. I have tested both models on two private repos (c#, go) on two long-horizon tasks with well-defined stop conditions and observed the same pattern in both cases. Both models still require a large harness to reduce shortcuts and architecturally unclean code, but gpt performs much better, to the point where I find claude unusable for any significant work.by fiso64
5/27/2026 at 7:04:58 AM
GPT 5.5 does significantly outperform Opus 4.7 in the coding parts of our evals.We also incorporate live decision making on social games (where GPT 5.5 has actually regressed from earlier models, which shouldn't be a huge surprise if you ever tried talking it out of some of its nits).
We are still looking for a way to integrate "logical" intelligence with social intelligence in a less arbitrary way, so I'd take a look at the use case that applies to you (probably coding): https://gertlabs.com/rankings?mode=agentic_coding
by gertlabs
5/27/2026 at 4:05:07 AM
1. your 'agentic coding' benchmarks are already saturated, with mimo #2? Cmon2. game rl is fundamentally less useful than coding or work rl
by vanuatu
5/27/2026 at 4:57:09 AM
Check out the methodology section at the bottom -- we are trying to better convey this information.1. These numbers are based on percentiles, which inherently can't be saturated. Most benchmarks operate on something like 0-100% of correct answers, so it's natural to make that assumption when you see our numbers. Perhaps we should divide by 100. We create a modified score based on percentiles against other agents, which rebalances every time we add new entries. So when a new frontier model comes out, all of the existing entries get downweighted if the new model outperforms them. And MiMo V2.5 Pro is a much stronger model than people realize.
2. Agents write code to play most of these games (accounting for ~80% of the combined bench score). There is increasing evidence that nearly identical patterns of weights emerge in different models, trained on different mediums and using different algorithms. Pattern matching and extrapolation don't care if the scenario is a 3D "game" environment or a Salesforce "work RL" environment. Examples of drawing distant connections in different domains can reward similar circuitry.
by gertlabs
5/27/2026 at 6:54:29 AM
[flagged]by charleyslee