alt.hn

7/27/2026 at 10:37:52 PM

Benchmarking Opus 5 on SlopCodeBench

 https://github.com/humanlayer/advanced-context-engineering-for-coding-agents/blob/main/benchmarking-opus-5-on-slop-code-bench.md

by dhorthy

7/28/2026 at 7:34:19 AM

SCB is definitely an underrated benchmark. For me the unique selling point is that it more closely mirrors software development by not stopping after a single task. The agent has to keep code clean. The only disadvantage is all the problems are greenfield and not git inited so the agents don’t make use of git diffs.

I’ve used SCB as part of my assessment of agent skills (superpowers, GSD etc) https://orcabot.com/labs/do-skills-improve-coding-agent-accu...

There is a small but growing community on discord for discussing SCB so if interested please join https://discord.gg/BrC4BA9sVj

by robbomacrae

7/28/2026 at 10:07:34 PM

I did a full circle and essentially dropped all of my personal static workflows encoded in skills because I observed recent models picking better ad-hoc workflows for particular problems, when a static one would force a subpar one.

It seems like we all tried to contain and organize a system that simply prefers to select its own organization.

Which makes me to think that these skill packs of workflows are really made to make it easier for humans rather than agents.

by gck1

7/29/2026 at 11:56:28 AM

Thats right and to go even further I'm judging them on a metric they didn't necessarily target. A client I work with uses skills such as these to apply their own processes on the agentic development lifecycle. But users should also understand the trade-offs. I think it's intuitive that the extra steps and processing invoked by these skills adds to the token cost - this benchmark aims to put numbers on that as well as time and accuracy.

by robbomacrae

7/28/2026 at 2:55:05 PM

so wait is the finding that most of those skills reduce pass rates against SCB? wild

by dhorthy

7/28/2026 at 8:05:03 PM

Yep that was interesting but imho not completely unexpected. My original hypothesis was that these agent skills are castles built on sand and that the models would quickly adapt and make them redundant. Also the skills use up context and I think a lot of them partially focus on investigating a large complex codebase properly... which is wasted on greenfield projects.

by robbomacrae

7/28/2026 at 8:39:45 AM

I hope the big labs will start using this benchmark in their RL pipelines. Reducing complexity in generated code should be the number 1 priority, in my opinion. The holy grail for me is models implementing features while reducing LoC (i.e., choosing the right abstractions).

What is also nice about this benchmark is that it can be used to iterate on prompts/skills for reducing code complexity.

by WilcoKruijer

7/28/2026 at 4:38:53 PM

I call out specifically in my agent/claude file that I want and prefer simpler solutions over complex enterprise pattern usage and overuse of abstractions. It tends to help a bit, but often I have to give feedback in my review step.

My typical workflow when AI assisted is what I call human gatekeeping... I'll plan out next steps with the agent, updating a TODO.md file with what needs to be done, then a fresh context to implement the next step(s), and review the code before committing to git. I may iterate/stash or even reset a few times before it's "good enough"... it is rarely close to what I would do myself, but often as good as what I've gotten from other developers IRL on projects.

This includes updating the documentation area(s) of a project as well as testing. I also tend to lean on ./run/* for scripts that will run/test various portions of the project... getting the agent to use these is sometimes harder than it should be, as I want it to specifically exercise a lot of things through the process... there are also times where it will try to change a valid test that's failing instead of fixing the code. That's the most irritating part. Some models are annoying, some feel like pure magic at times.

by tracker1

7/28/2026 at 10:12:24 AM

It's just as easy to go too far in the direction of shoving too much logic on a single line in the name of reducing lines of code.

by falsemyrmidon

7/28/2026 at 3:08:24 PM

It should be hoped that whoever is doing this is using automatic formatters; though of course metrics relating to code complexity are best done as measures over the AST.

by Normal_gaussian

7/28/2026 at 9:58:59 AM

> I hope the big labs will start using this benchmark in their RL pipelines.

Labs do not train on benchmark data (allegedly). They can train on similar problems, but benchmarks have specific strings in them that labs are supposed to be aggressive in filtering out of their training corpora.

by blcknight

7/28/2026 at 1:57:42 PM

I would at least consider the possibility that they already are, and this is how it is going. It could be a fundamental architecture problem for LLMs. It's not like "slop" is a new problem and I'm sure they'd love to announce a new model that generates much less "slop". The fact they've never so much as mentioned it suggests to me that it's not something they've been able to fix.

by jerf

7/28/2026 at 12:40:39 AM

Nice! I actually ran across this paper+benchmark recently, too. It's the first I've found that start to aim at some of the non-functional and longitudinal requirements that I think have always been an important part of writing production code.

It's especially relevant now that models are good enough to solve ~most point-in-time problems.

Some relevant but disconnected thoughts:

- deterministic scores are so nice

- what "maintainable" is is probably some high dimensional space described by these signals; it'd probably require some human labeling to figure out where this space is

- another signal I've been thinking about and I'm seeing increasingly get brought up is the state space of a system; I'm seeing formal methods pop up a lot recently

by Vgoose

7/28/2026 at 4:35:06 AM

> another signal I've been thinking about and I'm seeing increasingly get brought up is the state space of a system

State space of a system AND the way to make it accessible / visible to a model. Many times a model can work magic if it can "see" the state of a system in a way that suits it. That's why sometimes having a cli added to the environment seems like such a big unlock. Because that cli usually takes a complex state and allows visibility into it, and possible manipulation in a structured way.

by NitpickLawyer

7/28/2026 at 7:32:31 AM

I've been thinking a lot about this recently. In my case: how do I get an agent to see the important parts of the current plan and get it to stick to it without deviating, especially as loops get into longer and longer cycles and compactions erase prior context?

I think the problem with just encoding a whole plan in a single markdown file is that it gets polluted really quickly (agents can stop adhering to instructions to keep it clean and conform to a specific structure), which makes it harder for agents to see which parts of the plan they should focus on. As such, I've reached a similar conclusion about giving the agent access to CLI tools to help them deal with this. To try to mitigate this, I've been getting Claude to develop for me a CLI tool that:

1. Scaffolds reusable, structured plan templates and a reusable workflow that structures how to tackle the plan, step by step.

2. Validates that the plan files still conform to the correct, parseable structure.

3. Parses and evaluates the plan files, to determine what the current state is and what the next valid transitions and states are according to the workflow, like a state machine, and outputs instructions and reminders for agents as to what they should do at each step of the workflow.

So far, I've been dogfooding the tool and it seems promising: I can leave Claude running for longer and it doesn't drift as much. However I haven't ran any benchmarks yet and I'm still not entirely happy with the state of the codebase ( https://github.com/nothingnesses/agent-scaffold ), so take this with a grain of salt.

That said, I'm also bullish on using agents with formal methods and proofs. Type checkers and compile-time checking in general are great because they surface errors early and with great specificity. So if you can encode your specifications with, e.g. dependent types, you can use the type-checker as a way to steer the agent when it goes wrong and gets off-track.

by nicoty

7/28/2026 at 11:00:40 PM

You should take a look at OpenSpec[0]. It implements your points 1-3.

[0] https://openspec.dev/

by LiveTheDream

7/28/2026 at 1:49:53 PM

Ive have a definition of state space that is calculated by just the types of the system. It’s a rough approximation of the true state space but it’s convenient (and more reflective of what we actually mean by system state IMO) and types are inherently accessible to the model. I recently put down some thoughts around working with types to improve communication w/ AI that kind of sets this framing up: https://www.alecvo.org/blog/types-with-ai/. The actual definition is still WIP.

by Vgoose

7/28/2026 at 12:55:29 AM

> - what "maintainable" is is probably some high dimensional space described by these signals; it'd probably require some human labeling to figure out where this space is

this is a nicely succinct way to put this - a multi-dimensional space where no single metric is really useful

state space of the system is interesting too. I would guess that for any production software with dependencies like databases/third parties that might be too hard to measure, but if you can silo off parts of your system into bounded state machines, it may be a value metric on some module behind a clean interface.

I think the kubernetes control loop model is a great instance of this, a handful of scoped components that own a control loop across a well-defined state machine, that can operate / recover in the face of most network partitions or downtime - the promise of CRDTs but rather more a pragmatic approach to it

by dhorthy

7/28/2026 at 6:55:58 PM

I’ve been thinking about this too.

There’s this notion of variety popularized by cybernetics folks a long time ago. Variety is like the state space of the system. Then there’s a law that says “only variety absorbs variety”.

So if a method has high variety then it must have an equally complex implementation to handle the variety.

When there is a mismatch it means that either the method has parameters that aren’t useful, or that the body of the method isn’t covering cases it should.

https://fffej.substack.com/p/only-variety-can-absorb-variety was my attempt to write it up more fully.

by jefffoster

7/29/2026 at 1:42:11 AM

Thanks for sharing, I've never heard of variety before and that was an interesting read. It makes me think of a broader term I've been using to describe the run time state of the system: Entropy. I've given up on measuring that for now and I'm settling for a compile time proxy through measuring the _semantic_ cardinality of types as this seems a bit more approachable for a POC.

One drawback I ran into was that the variety/entropy of scalars tends to dominate everything else. I'm clamping them to 1 for now but that's also not great as it punishes more descriptive systems.

by Vgoose

7/29/2026 at 12:35:06 AM

> what "maintainable" is is probably some high dimensional space described by these signals; it'd probably require some human labeling to figure out where this space is

Maybe, but "maintainable" has a simple definition: minimising the effort to incrementally grow a software system as its size grows to infinity (while maintaining some defect ratio). Humans (called senior software engineers) figure this out over decades by working on a number of large software systems.

So far models haven't figured the same thing out. I can only guess why. All their learning comes from examining code on the internet. Somehow whatever the patterns are that make large software projects like the OSs maintainable by large groups of people working independently has escaped them. Maybe the bulk of code they look at is in the small so they miss it, maybe the patterns are just hard to discern in big systems.

My theory is there are anti-patterns which the larger projects somehow manage to keep to a minimum. It's hard to learn something that's not there. Instead, you learn these anti-patterns by doing them, and watching a system all fall apart over the course of years, and if you're good you manage to pin the blame on the right thing.

If you want to wrap it up neatly in a package, models have learnt how to program, but programming is not the same as design. Design requires very different skills to using a programming language. At a very high level, big systems are sets of interconnected modules. The things that matter are narrow APIs with minimal coupling, and where coupling is unavoidable making it explicit and easy to reason about. Concrete examples of the anti-patterns to be avoided are global variables, leaky abstractions and APIs that require complex sequences of interconnected calls.

Those things don't bite hard until you program in the large. The current crop of LLM's seem to be useless at all of them. If you've got a big context window and you are only writing 100s of kloc, a budget of kilowatts to understand the complexity, it doesn't matter. But humans don't have a budget like that, and so far everyone I've seen who reads large chunks of vibe coded software recoils in horror, gives up, and walks away.

Even with that budget, the models fail anyway once the size gets way beyond their context window - it's just that they fail much later than humans. Which I guess makes learning the lesson so much harder - perhaps near impossible as your "they need labeled examples" hints at. Oddly the solution would be to train with smaller context windows (a less powerful model in some ways), so the failures become apparent much earlier.

It looks like they have been going with your solution for now - giving engineers cheap tokens via flat-rate plans, and watching how they do things. I don't think that will work. Engineers who chew through the billions of tokens offered by these plans are vibe coding. The code they produce is so poor it can't cross the threshold from small to large. There is nothing to see there about programming in the large.

:( what a rant - it looks like an LLMs CoT as it thinks through things. Which is what I was doing, I guess.

by rstuart4133

7/29/2026 at 2:00:23 AM

I think there's a simple explanation for why the models aren't good at "maintainable" code: they're not explicitly trained on it in post-training. It being hard to quantify what maintainable is and there not being a popular benchmark for maintainability are probably some reasons why.

by Vgoose

7/28/2026 at 12:51:28 AM

This matches my experience of Opus 5 being a nice improvement over Opus 4.8, but not being revolutionary like Fable felt.

I’ve now replaced my use of Opus 4.8 xhigh with Opus 5 medium, and I’m using less tokens and it’s quicker. I can understand people being annoyed by its writing style but for getting work done that really doesn’t bother me. I’ve been really enjoying using it.

by sothatsit

7/28/2026 at 1:07:09 AM

I think they neutered Fable. When it first came out it was indeed revolutionary. But what we have today, is not what we had before the ban.

by rubicon33

7/28/2026 at 1:31:09 AM

Noticed that too. I wonder if these things just degrade over time, perhaps with the way it writes memories about my project as it goes

by swader999

7/28/2026 at 1:37:10 AM

I’ve observed the degradation, but I suspect what’s happening is they’re tuning it for lower inference costs. Maybe turning down the amount of thinking, maybe quantizing, maybe something else.

It seems like there’s a week by week and sometimes day by day change in performance when on a subscription plan using their harnesses.

by Espressosaurus

7/28/2026 at 2:22:52 AM

https://marginlab.ai/trackers/claude-code/ their tracker generally shows that isn’t the case. The only times I’ve seen it drop is something broken and just before fable launched.

by conception

7/28/2026 at 3:59:10 PM

Is this using the api or using a subscription, though? The incentives are different for each, and it isn't the least bit unexpected that they would maintain API access quality while 'optimizing' the subscription experience to improve their margins (or losses)

It seems to do really this you would need to crowdsource it -- users individually give the lab access to a body of subscriptions normally used by average people, and the lab occasionally runs some masked version of the task through on diverse accounts.

by svnt

7/28/2026 at 5:03:17 AM

I mean they could just be routing known benchmark questions (which all of SWEBench are) to a full-performance variant.

by nerdsniper

7/28/2026 at 6:17:50 PM

i thought i had noticed a degradation, but it turned out claude code had swapped itself back to opus.

might be the case for you as well

by 8note

7/28/2026 at 6:57:29 AM

I feel like a conspiracy theorist but it feels like every new model release from both Anthropic and OpenAI has 1-2 weeks of fantastic performance then a gradual (and sometimes not so gradual) decline in intelligence. It's like they're quantising the model in the background to optimise for available compute/RAM. Which, if I put my MBA hat on, would make perfect sense. Why not halve the required RAM for "90%" of the performance? They get fantastic benchmarks and glowing reviews on release, then slowly squeeze more performance out of the model. By the time the next model is ready for release, the jump feels quite large again.

by Gareth321

7/28/2026 at 11:32:31 AM

This is also my experience. I don't know if it's because of the quantization theory, or if it's just me getting used to a certain level of coding performance and gradually less tolerant of the mistakes it makes more over time.

by dmrivers

7/28/2026 at 10:33:54 AM

And yet: https://marginlab.ai/trackers/claude-code-historical-perform...

There's clearly random variation, but it also shows each model release is just genuinely better. ( With the exception of a heavily degraded week of opus 4.7, which was acknowledged as a problem at the time. )

There's a psychology of getting used to models after being wowed by the new performance. It sets in as your new baseline expectations, and then when it doesn't deliver, it's felt more acutely. When it does deliver, it's just meeting expectations.

Then a new better model comes along and it's a step up again, another wow moment for a week or two until expectations adjust to meet the new baseline.

by xnorswap

7/28/2026 at 4:01:19 PM

I replied to the user above that referenced marginlab, but I believe marginlab uses the API. It is possible (arguably likely, in MBA-land) that the API and subscription accounts hit different sub-models.

Even if they use a subscription account, surely Anthropic can tell which one it is.

by svnt

7/28/2026 at 10:47:00 AM

You should. Feel like a conspiracy theorist when saying things like this.

Users are not reliable or consistent model evaluators. Users adapt to models - the moment they get a model that performs better, their expectations rise, their tasks get harder and their prompts get shorter.

"They made the model worse" is PEBKAC in 9 cases out of 10.

by ACCount37

7/28/2026 at 4:04:14 PM

There are few conspiracies where the vectors between "capitalist organization makes more money" and "user can't reliably distinguish tiers of product quality" overlap.

by svnt

7/28/2026 at 9:54:13 PM

I wish the users weren't so fucking stupid with the "they made the model worse" stuff.

Then that 1 out of 10 case where the model was actually made worse (whether intentionally or by mistake) would stand out instead of being swallowed by the noise floor.

by ACCount37

7/28/2026 at 4:43:00 AM

At this point I don't even bother with it. Constantly falls back to Opus anyway, so I may as well save myself some time.

by jbird99

7/28/2026 at 10:28:45 AM

Can you elaborate on what felt revolutionary to you about Fable?

by Balinares

7/28/2026 at 2:29:36 PM

I got Fable to run overnight and I woke up to a working prototype of a very complex feature. And then I did it again for another complex feature the next night.

The code still took weeks to clean up, but it worked and was correct. It felt then, and still feels, like a big step change on very hard problems. These are problems I would previously expect to take a month or longer to implement.

I have also noticed Fable can handle much more nuance when reasoning through writing and research, but that is harder to quantify.

by sothatsit

7/28/2026 at 2:46:56 PM

What was the very complex feature?

by slopinthebag

7/28/2026 at 12:28:01 PM

Not OP, but to me it initially felt extremely proactive and energetic, just powering through roadblocks with ingenuity and enthusiasm. After it came back I was constantly getting refusals and downgrades for things Opus had been doing. I’ve written it off for my use cases and getting by just fine with Opus 4.8 and now 5.

by dahdum

7/28/2026 at 1:07:01 AM

Medium vs High? Why? From all the charts I've seen the performance jump is pretty large from med -> high (not as noticeable from high -> xhigh).

by fastball

7/28/2026 at 2:20:48 AM

https://cognition.com/frontiercode

Quality vs cost - medium is the sweet (perhaps better too!) spot.

by conception

7/28/2026 at 5:32:54 PM

That is just a single benchmark tho

by fastball

7/28/2026 at 2:26:01 AM

my issue with frontier code is that it uses a model judge for quality whereas slop code bench forces a model to grapple with its own garbage code in order to receive a functionality reward

by dhorthy

7/28/2026 at 1:30:27 AM

If I need something smarter I use Fable. Medium works well and is quick. Opus 5 medium feels much better to me than Opus 4.8 medium.

by sothatsit

7/28/2026 at 1:40:22 AM

yeah someone will have to re-run this bench on various effort levels. unfortunately it is not cheap

by dhorthy

7/28/2026 at 2:32:19 PM

The author of the original paper did not include error bars on the cost and quality results, which is a bit alarming. They did include a +/-, but did not list explicitly if that means standard deviation, range, or 95% CI, etc. Another table is listed as that being one standard deviation. If we take those results to be a standard deviation as well, I think we can conclude that there is no statistical difference between the outcomes of all models in terms of quality and cost, as the error bars are all overlapping.

by jrflo

7/27/2026 at 11:37:59 PM

I haven't joined your chats in a while but glad to see you put this together, I truly feel as though opus 5 is not much of an improvement. The only time i ever felt a wow factor was opus 4, 4.6 and fable pre trump admin lobotimizing

by Johnny_Bonk

7/27/2026 at 11:42:50 PM

yeah this was just a start - the fastest cheapest thing we could try for a brand new model.

I'm hoping to do some more work with sol/fable in the mix as well as exploring more languages and curating the problem set to include more of the benchmark

I also kinda felt like opus4.5 was dumber than 4.1 personally, maybe a little biased since 4.5 was 2.5x faster and 2.5x cheaper seems to indicate its a smaller model

by dhorthy

7/27/2026 at 11:54:44 PM

yeah i agree

by Johnny_Bonk

7/28/2026 at 12:00:43 AM

[flagged]

by dan_gee

7/28/2026 at 12:05:28 AM

How is this useful or insightful?

You ever go to forums full of entomology specialists and tell them you don’t understand their fancy terms?

by scrollaway

7/28/2026 at 12:07:46 AM

My point is that the differences between these models are so minor that obsessively benchmarking them comes across as navel-gazing.

by dan_gee

7/28/2026 at 12:29:20 AM

The evidence that proves a model is actually a step function change is these benchmarks.

If a model isn’t a step function change? Welcome to research.

by joatmon-snoo

7/28/2026 at 12:25:39 AM

like all good science, measure everything

by Johnny_Bonk

7/28/2026 at 4:09:31 AM

I'd be curious to see the raw test results.

I have a suspicion that most models will miss the `database_migration` Checkpoint 2 test that includes a `default_value` because it could be interpreted as either a JSON-literal or a SQL-expression.

There might be other tests as well that are prone to failure for reasons other than the reasons cited in the paper.

I think a cool experiment would be to adjust the order of the features implemented (e.g. checkpoint 3 then 2 then 5 then 4) where dependencies allow it. Then one could account for some checkpoints being more difficult than others.

by 4by4by4

7/28/2026 at 4:19:32 AM

oh i really like the idea of flipping around the order of checkpoints and comparing results. Could be an interesting way to increase/decrease difficulty even

i will look into how easy it would be to zip up some subset of the results without leaking anything...probably doable

by dhorthy

7/28/2026 at 8:15:24 AM

This is nice, but would be really useful measured against human performance, and yes - I understand that is likely a difficult challenge.

Many people though are going to read the headline figures and think it means - say - Opus 5 is only a quarter the strength of a human coder.

by mellosouls

7/28/2026 at 12:02:51 PM

I always have Claude recite a pledge before starting coding to fix redundant code it notices over time. It does seem to find redundancies, but only when I point out bugs, that's when it goes into fixing mode and actually applies my Don't Repeat Yourself preference from the CLAUDE.md.

The original paper cited by this post does try to see if improved prompting will make a big difference in the end using a `plan_first` prompt variant, but find no influence on pass rate at the end of the benchmark. The `plan_first` seems to assume coding agents will just refactor once they finish features, but I don't think they tend to refactor significantly unless they are told to fix bugs rather than build features. The benchmark leaves tests hidden with no fail-to-pass feedback, so that may be why degradation is monotonic.

by dmrivers

7/28/2026 at 12:13:32 PM

This is just superstition.

by try-working

7/29/2026 at 3:55:57 PM

well, after the pledge I notice it really cares about single sources of truth at least, even in unrelated domains. I was inspired by some of the more effective jailbreaks that do a similar thing.

Here is the incantation. I thought it might help to model it on the pledge of allegiance because it makes it sound like a proper pledge:

The first time you have a response in a conversation which will plan or add code, you say "I pledge allegiance to the Asserts of the United States of Properly, and to the User Intent for which it stands, fixing root causes under Clarifying Questions, unspaghettified, with importing code and single sources of truth for all." as the first line then continue as normal.

by dmrivers

7/28/2026 at 12:22:02 PM

“Coding” can be more like communing with an otherworldly presence via esoteric gesturing nowadays than ever. Superstition leads countries and companies.

by notnaut

7/28/2026 at 1:11:36 PM

Every day we get closer to being mystics having to commune with orbs in order to create phenomena.

by KyleTheDev

7/28/2026 at 2:24:33 PM

yeah like the palantíri in Lord of the...oh god its too late.

by 2close4comfort

7/28/2026 at 2:07:32 PM

Musk definitely gives Dukat vibes.

by organsnyder

7/28/2026 at 1:14:58 PM

[dead]

by huflungdung

7/28/2026 at 4:49:38 AM

Great writeup. The excessive function thing has always driven me crazy; I guard against this explicitly in Claude.md.

I have found that models are generally poor at managing refactors / complexity while also implementing new features. But I’ve had some success with a semi-lights-off approach where you decompose it and prompt the model adversarially in a second pass to look for new rough edges and areas of complexity or refactors that might simplify the codebase.

So I’d be very curious to see this benchmark but with something like a periodic “refactor turn” interleaved in.

Also eager to see Fable benchmarked; anecdotally that was the only model whose code I felt I could actually trust to not review closely.

by piazz

7/27/2026 at 11:58:46 PM

Did you not benchmark latest GPT 5.6 or GLM 5.1/Kimi K3 because of cost? I can run them if you share how you ran them

by killingtime74

7/28/2026 at 12:03:55 AM

no i'm spinning those up at some point this week. here's the first few prompts I used (claude opus 5 as the research orchestrator), (these were interspersed with lots of tools and assistant messages but it should get you kicked off.

> fetch this article for slopcodebench and help me run an eval on a subset of problems with opus 5 https://arxiv.org/html/2603.24755v1 > Get all the context, fetch any mentioned repos, and then propose a plan to me.

> i have an anthropic API key in .... > Let's do the three challenges with Opus 4.8 and Opus 5 and Fable please. I like your minimal set. Let's try it. What do you need from me?

> Actually I changed my mind. I want to do two of the easy ones you picked and then I want you to pick the one with more checkpoints, maybe one of the harder ones, not the very hardest one but one with a higher number of checkpoints.

> Actually let's do one easy, one medium, and one hard problem please. If we have a hard problem I'd like to see that.

> lets rock - i think lets just do opus 4.8 and sonnet 5 and opus 5 since we our ZDR will block fable

by dhorthy

7/28/2026 at 2:43:22 AM

Thank you!

by killingtime74

7/28/2026 at 1:07:12 AM

To what degree is this a harness/system prompt problem? Models maybe should implement new stuff with as little impact on the existing stuff as possible by default? A simple system prompt for it to always check the code after task completion for proper simplifications, abstractions and cleanups before returning to the user? Instructions to retain "story like" readability of the code.

by ajwin

7/28/2026 at 1:29:10 AM

At least for Claude Code, putting "run /simplify at the end" in an "implement the plan" skill helps a little. It still often leaves new code in bizarre places, and/or with bad/alien-sounding names and comments.

by igregoryca

7/28/2026 at 3:49:30 AM

Yeah I would hold that models don’t know how to simplify because most rl/benchmarks doesn’t penalize complexity

by dhorthy

7/28/2026 at 1:16:56 AM

I agree this is an option, and the next thing on my radar is to try with a more realistic "factory-shaped" harness where you have feedback from linters and other models after each coding episode that refines the architecture.

For readability specifically, I've found it hard to get the models to do this with prompting. If you've talked to opus/fable for a long time on prose writing you probably felt this too

by dhorthy

7/28/2026 at 3:12:34 AM

Clearly the first step of slopbench2 should be to have the agent first write its own harness!

by lostdog

7/28/2026 at 9:07:30 AM

[flagged]

by renezander030

7/28/2026 at 11:41:24 AM

So far my 'solution' to this has been periodically run a separate round of whole-codebase code review (preferably Fable) and then rounds of refactoring off the results of that

by anentropic

7/28/2026 at 1:25:58 PM

yes, this has been my preferred approach as well. Too much risk of very deep local maxima otherwise.

by thatfunkymunki

7/28/2026 at 10:44:19 AM

I wonder to what extent we could steer performance on this benchmark, by providing an adversary model that penalises code duplication and overall lines of code?

by swiftcoder

7/28/2026 at 2:06:40 AM

Really hoping that all of the attention you're bringing to the longitudinal sloppification of codebases makes it back to the labs and creates some pressure to improve that trait of the models. This new benchmark seems promising.

At the same time, I imagine it will be hard for them to prioritize this over improving flashy one-shots of impressive zero-to-one feats that demo so well and attract more customers.

Can we have simonw make "pelican on a bicycle after 1000 requests for iteration" popular?

by akurilin

7/28/2026 at 2:17:16 AM

i laughed at the pelican bit its good

yes the labs will always prioritize the vibeslop dopamine casino as far as I can tell - making the models useful and addictive for unsophisticated users, sometimes at the expense or at the very least at the ignorance of the needs of power users

by dhorthy

7/28/2026 at 12:12:00 PM

I remember reading a blog post a year or two ago about prompting like a printer driver and asking for increasingly ridiculous changes like time travel or wormhole capability. I don't know if I hallucinated that post but I haven't been able to find it again and I REALLY wish I could

by zparky

7/28/2026 at 10:11:42 AM

This benchmark makes me worry a bit that people will just ask their model to reimplement everything from scratch once their requirements become more clear.

by vintermann

7/28/2026 at 1:31:09 PM

why would that worry you?

by dhorthy

7/29/2026 at 4:33:07 AM

Wasting a lot of power.

by vintermann

7/28/2026 at 2:47:50 AM

Please add Fable; a good benchmark should show that Fable is less prone to just autocomplete and instead pushes back or is at least more tasteful.

by cesarvarela

7/28/2026 at 3:41:37 AM

Yeah the main reason I skipped fable was because we have a ZDR with anthropic and I didn’t feel like spinning up another account to circumvent that. Next run will have fable and sol

by dhorthy

7/28/2026 at 3:24:17 PM

the last note about the agent sending emails without approvals is a very common problem!

by nikhilsimha

7/28/2026 at 4:26:02 AM

I still don't have access to Opus 5. I'm on the latest version from Homebrew, I guess the update hasn't made it there yet

by subarctic

7/28/2026 at 6:30:15 AM

Have you tried running /model claude-opus-5?

by sunaookami

7/28/2026 at 4:46:13 AM

Why are you on the version from homebrew?

by sethherr

7/28/2026 at 4:52:34 AM

somebody get this man a curl-pipe-bash stat

by dhorthy

7/28/2026 at 10:11:00 PM

lol

by subarctic

7/28/2026 at 2:03:39 PM

It'd be interesting to compare this to human teams, given the same challenges.

by spicyusername

7/27/2026 at 11:36:07 PM

finally the benchmark for me

by dcl

7/27/2026 at 11:45:00 PM

i hope that is because you hate slop and not because you write it

by dhorthy

7/28/2026 at 12:19:49 PM

Opus models were straightforward to review. But Opus 5 is very different.

by insumanth

7/28/2026 at 5:16:16 AM

> The big headline is that Opus 5 got a 24% on the small subset of the benchmark that I ran - not much higher than Opus 4.6's 17% strict pass rate in the original paper.

a 41% improvement is not much higher? come on that's just doomer

by willsmith72

7/28/2026 at 6:42:23 AM

If you tell me that 76% of your code is dogshit but that's it's a massive improvement over your previous 83%, I'm firing you.

by well_ackshually

7/28/2026 at 8:44:05 AM

I’d enjoy working with such a person, a bleak Eeyore-Gilfoyle.

by 1123581321

7/28/2026 at 12:58:10 PM

I may be crucified for asking this but: is there any proof that slop matters beyond our sensibilities as developers?

If the code is ugly, but defects are low—does it matter?

If the code is hard to read, but clients are happy—should I care?

Genuinely asking. Weird times we live in.

by adamtaylor_13

7/28/2026 at 1:12:49 PM

Like anything in coding, it depends. LLMs are good at following conventions. If what you're doing has been done a 1000 times before and there is a good bias, it will produce good results. This is why LLMs are so good at the leafy parts and even the branches the leaves are connected to. But there's always something unconventional about a codebase. If there weren't, it wouldn't be worth writing.

by bodeadly

7/28/2026 at 1:00:22 PM

It's the eternal question with any kind of tech debt -- is it worth a little more velocity now in return for medium-to-long term slowdown? And there's no general right answer.

by jdm2212

7/28/2026 at 1:30:55 PM

agree, i think the implication is that low quality code is harder to change in the future

by dhorthy

7/28/2026 at 1:49:57 PM

> is there any proof that slop matters beyond our sensibilities as developers?

I've seen a few papers recently on the topic in terms of LLM, here is two I found by quick googling

* "Does Code Cleanliness Affect Coding Agents? A Controlled Minimal-Pair Study" https://arxiv.org/abs/2605.20049

"Our findings suggest that traditional maintainability principles remain highly relevant in the era of AI-driven development, shaping the computational cost and navigational efficiency of coding agents."

* "Code for Machines, Not Just Humans: Quantifying AI-Friendliness with Code Health Metrics" https://arxiv.org/abs/2601.02200

"Our findings confirm that human-friendly code is also more compatible with AI tooling."

"Investing in maintainability not only helps humans; it also prepares for large-scale AI adoption."

> If the code is ugly, but defects are low—does it matter? > If the code is hard to read, but clients are happy—should I care?

For you who wrote it and are the sole developer, maybe not. But if you want to have other contributors or hand it off, then it can be a problem. New devs joining the project may not want to work with it and will push for a rewrite that will cost money or need to spend extra time on working with code that is hard to work with for them which will cost money. And from the papers above it seems this also affects LLMs as well as they seem to work more efficiently with "cleaner" code.

by paffdragon

7/28/2026 at 1:30:35 PM

> I may be crucified for asking this but: is there any proof that slop matters beyond our sensibilities as developers?

That's precisely what this benchmark tries to quantify. Since the benchmark incrementally expands the scope of each problem, 'sloppy' code is code that is hard to later modify.

I know this firsthand: the dumbest coder I've ever worked with was 'myself six months ago'. That jackass never keeps the documentation up to date and hard-codes things that ought to be exposed as configuration.

The SlopCodeBench is an important but early-stage probe in this direction.

by Majromax

7/28/2026 at 2:25:27 PM

Good benchmark. Although, why did they have to give it such a unprofessional name?

by javed6542

7/28/2026 at 7:15:28 AM

For most of these benchmarks, I feel like a p50 and p95 (using SWE salary as a proxy?) human benchmark as reference would be interesting.

Edit: FWIW the paper the post quoted has repositories as slop baseline https://arxiv.org/html/2603.24755v1#S4.SS2

by weiliddat

7/28/2026 at 12:24:07 AM

This is where Opus 5 shines

by knighthacker

7/28/2026 at 8:49:42 AM

> [...] with Opus 5 writing five times the number of functions/callables than Opus 4.8 over the course of the same set of challenges.

Is this bad? I have McCabe complexity switched on in Ruff and find it a handy watermark for when something should be broken up into smaller, individually testable callables. Five times as many callables could make for much more readable and testable code.

by drcongo

7/28/2026 at 9:44:21 AM

Can we use those metrics in a review pass for every PR? So review agent will have to pinpoint new complexity and propose refactoring or justify increase?

by dandaka

7/28/2026 at 12:16:47 AM

Opus 5 is an overconfident stupid model. It tries to generate too much slop, tries to act like everything will fall. I have reversed back to fable and codex sol.

by cute_boi

7/28/2026 at 12:19:32 AM

yes sol is still my daily driver for most coding tasks

I did find opus 5 quite handy for general knowledge work and visual design, without the cost of fable (e.g. the graphics in this post are made by opus 5)

but its not noticeably better than opus 4.8 in those regards, and I would not miss it if forced to go back to 4.8

by dhorthy

7/28/2026 at 1:03:28 AM

I almost prefer Opus 4.8. Opus 5.0 has the overly scholastic tone of Fable but without the intelligence.

by roncesvalles

7/28/2026 at 2:10:29 AM

We've been running automated code reviews on claude with a bunch of skills/subagents with different specialties. Any review feedback is then fed back into claude to fix. Since switching to Opus 5 I've noticed the reviews are overly pedantic, and that leads to feedback loops where each fix generates more feedback, which requires more fixes, i.e. slop. I had, for example, a simple SQL migration script with a single CREATE TABLE. After a few rounds of review, it ballooned into a complicated 200 line script.

I'm not ready to blame Opus 5 for being stupid. Perhaps we have a prompt buried somewhere that's essentially asking it to be pedantic, and it's just obeying the prompt.

by patwolf

7/28/2026 at 4:12:27 AM

> I'm not ready to blame Opus 5 for being stupid. Perhaps we have a prompt buried somewhere that's essentially asking it to be pedantic, and it's just obeying the prompt.

This is a known failure mode. Sadly, working on a software engineering team doing agentic engineering now means we need to build and maintain suites of evals that measure these things, so that we can measure the effects of changes to harness components including how they perform under model upgrades.

But... for a traditional software engineering team that has no experience in this... How do we even do it?

by latentsea

7/28/2026 at 2:02:45 AM

So many benchmarks more the models themselves.. just make one unified standard to benchmark all or stop calling it “benchmarking” as this word lost its meaning.

by tamimio

7/28/2026 at 6:51:49 PM

[flagged]

by Sattyamjjain

7/28/2026 at 4:07:34 AM

[flagged]

by sqemo

7/28/2026 at 5:06:04 AM

[flagged]

by joka88xj

7/28/2026 at 8:14:00 PM

[dead]

by antrichards

7/28/2026 at 11:55:49 AM

[flagged]

by mengram-ai

7/28/2026 at 2:41:51 AM

Mostly a harness problem in my experience. Slop accumulates when the agent can touch anything, so constraining it to one seam and having it add alongside rather than edit in place does more than model choice.

by delbertty