7/6/2026 at 5:22:13 AM
Interesting question to study, but I'm extremely skeptical of the experimental design. They used Opus 4.6 to synthetically produce "degraded" or "cleaned" code bases for relative comparison in the experiment.Worse, they don't control for breaking the application's tests.
> Pass rate scores the agent’s final state against the hidden tests we wrote for each task. We do not check whether the agent broke unrelated tests already present in the repository, and a cleaner-side and messier-side solution that both pass the hidden test may still differ on tests they were not graded on.
Any conclusions with respect to token consumption seems pretty meaningless if we're not controlling for the quality of the final output.
by root_axis
7/6/2026 at 2:10:50 PM
If you're measuring a sloppy project against a well structured one with regard to how many tests fail, that'll bias success towards the sloppy codebase, which likely has worse test coverage and less robust testing in the first place. You'd essentially need to write a single test suite that works for both projects in each pair to compare fairly.That's not to say the study is good, but I can respect their decision because the tests passing isn't necessarily correlated with the effectiveness of the agent.
by bastawhiz
7/6/2026 at 12:21:22 PM
At best the conclusions are a best case scenario for degraded code quality. That things remain functionally OK with more costly token usage to get work done.by softwaredoug
7/6/2026 at 6:11:56 AM
Also controlling inputby jwpapi
7/6/2026 at 9:20:42 AM
Input matching distribution of a specific one-shotted LLM... Nice.by rullelito
7/6/2026 at 9:00:18 AM
[dead]by huflungdung
7/6/2026 at 8:28:07 AM
> they don't control for breaking the application's tests.Essentially it's an absolutely worthless study then. This AI fatigue got boring a long time ago, this is just painful now.
by lloydatkinson
7/6/2026 at 8:36:30 AM
To be sure, we had hidden tests that validate whether agent implemented the task appropriately. And this is something we paid a lot of attention to (pass rate, in our paper.)What we didn't do (stupid oversight on my part) was to ensure that there are no regressions in the remaining tests in the repo (unrelated to the current task). In practice, when using Sonnet 4.6 IRL, I don't see a lot of regressions because often the agent runs the test before calling it done. But it could have gone either way. We don't know.
by geraltofrivia