8/4/2026 at 3:24:32 PM
I’ve been thinking about how we practically implement this at an organizational layer for large codebases. There’s clearly alpha to be had in optimizing AGENTS.md / skills / tools / … to improve performance, quality, and cost efficiency of an agent. The problem is defining what quality means, and providing a way for agents to optimize the harness using that lever.The first step I see towards this is building a generic, reliable, and accurate *fitness function* for codebases - turning PRs into gradeable tasks that an agent can solve, and improve on.
I’m pretty curious how others are optimizing the coding agent harness now, as this has been a huge pain point for my company (we’re pretty much relying solely on vibes).
by bisonbear
8/4/2026 at 5:49:47 PM
I literally hold a "retro" with my agent after a session.> Let's conclude with a retro. Did you run into any issues during this session that you think could be improved? Any failed tool calls, confusing docs/prompts, or tricky wording that took you effort to figure out, etc? Any final thoughts that you want to raise? Anything minor you didn't mention? Help make this codebase easier for the next agent to work in.
It has ideas for coding changes, spots unmentioned small bugs, suggests invariants (I have a high-level list of invariants to keep), principles to adopt, lint rules, tooling tweaks (I've built SO MUCH agent-tooling; so many useful bash scripts), skill-file updates, follow up work, all kinds of stuff.
Listen to your agent's whinging.
by AlexErrant
8/4/2026 at 10:53:46 PM
Agreed. After an implementation session, I often use:"did you notice any inconsistencies or opportunities to clarify/consolidate parameters or functionality while you were working?"
Another strategy that has worked well for my MCP server is noticing a sequence of failed tool-calls, then returning a response that contains a request for the agent to provide feedback via a skill.
by karl_gluck
8/4/2026 at 10:39:57 PM
Yes, I build into my gnite skill for closing conversationsby dr_dshiv
8/4/2026 at 10:50:29 PM
I have Opus grade each of my repos, and each subsystem of them, and do it every weekend.Since the version of opus has changed over time, so have the evaluations. I was going to constrain it, but since they edit the same files, I can alter those (which it uses as summaries and starting points).
by sroussey
8/4/2026 at 5:24:44 PM
Ive been building my own ai voice agent harness from scratch for close to a year now and following good software architecture practices is a good start. So those rules have to be coded in agents.md somewhere also really helps to have a "vision" section or "spirit of the project" section that describes what the end goal vaguely looks like and things I care about in achieving for the project. This prevents agent from being brittle and "single minded" about its work. But yeah vibes most of the time is how I've also been doing it, but I did find one very important thing that has really sped up my work. So I figure I'd share it here. And that advise is to ignore front end design at all costs until the very end and you are ready to launch. UI related woes kill any type of inertia and are responsible for most of the issues. So I decided abandon all UI stuff until the very end and just focus on iterative refinement, cutting, and other back-end related work and its been great ever since. Ideas can be had, tested, validated/invalidates and you ' get stuck on the optimization thought loop. Recently I have started to consider how i can fully automate the development process as the capabilities are there but designing the instructions on how to do this well and how to handle niche cases without getting me involved takes careful planning in writing out the guide so that will be interesting to see once i get there.by nowittyusername
8/4/2026 at 5:03:45 PM
Just working on my own stuff so YMMV, but from a cost reduction standpoint, the two most outsized ROI wins came from making sure the agent runs all terminal commands in quiet and using a codebase index mcp to speed up understanding and reduce full file reads [0]. I've also found the latter also dramatically improved the quality of output.by jtr1
8/4/2026 at 9:28:45 PM
Trying something like this in a reusable way at https://github.com/mateffy/gesetzIt only tries to solve the „quality“ aspect of that equation tho, at least for the code output.
But I think for actual evals, some heuristics about a coding agent session are needed. But since the input tasks are always different, it’s hard to make any KPI actually comparable.
Did the agent needing 3 rounds of feedback on a big task perform worse than the one that needed none, but only worked on a small change?
by capevace
8/4/2026 at 5:14:40 PM
maybe a bit counter-intuitive but:I found that removing
- system prompt
- skills
- agents.md
- mcps
gives better results than having 'more' of them
(e.g. look at these traces to see more vs less in action:)
https://smolenv.com/t/nested-template-includes-60636/
not saying the right context does not help
(it definitely does!, but it's not trivial to provide the right context)
by tosh
8/4/2026 at 5:55:10 PM
This is the right direction (Anthropic removing 80% of the system prompt https://claude.com/blog/the-new-rules-of-context-engineering...)However what I wonder how we make these decisions on what to remove. Certainly some subset of the skills / AGENTS.md / prompt / tools do actually help agent performance, the problem is in identifying those, throwing away the rest, and validating that this didn't actually hurt the agent performance
by bisonbear
8/4/2026 at 5:18:55 PM
'better results' in terms of what though? A benchmark, or code that I would actually click "approve" on in a pull request scenario?by radlad
8/4/2026 at 5:21:35 PM
apologies, I should have clarified the 'better' claim - same task result (passed)
- finished faster
- fewer tokens, less cost
- fewer requests for inference
- fewer tool calls
- less peak RAMby tosh
8/4/2026 at 5:57:39 PM
Yes, my question is directed around how you are determining "task result." Is this measuring whether the code works, or whether it is maintainable?As with human-powered coding, we read code far more frequently than we write it. It's worth spending a little extra (time|tokens) during authorship to make future maintenance feasible.
My CLAUDE.md, memories, and skills are all about either (a) adherence to project standards and guidelines, (b) product decisions which impact future code, and (c) instructions on how to prototype and work in my environment.
Removing these instructions would mean more turns with the AI to get the desired result.
by radlad
8/4/2026 at 10:29:27 PM
You don't have to remove them - you can move things that are not important to every prompt to aseparate small docs that are referenced in agents.md with ,"when needed, reference these files:" and list them in form * relpath - content summary" or similar.. When you want those instructions to be followed, mentiont them in your prompt. "Test this following procedures on docs/test.md". (I've found I don't have to do that extra instruction in al cases depending on model)
by GrinningFool
8/4/2026 at 6:00:49 PM
in this case there was a hidden grader that checked if the implementation was correct (because that was the easiest thing to check), all 3 agents cleared this hurdle in all 9 runsI agree, next it makes sense to try more open ended tasks + have humans (and/or multiple models) grade the runs and their results
by tosh
8/4/2026 at 4:07:28 PM
You've just described what the AI industry refers to as "evals", a collection of which forms a particular benchmark. I definitely encourage you to define your own evals, because the public benchmarks are often either saturated (largely solved and only going to see small improvements going forward) or seemingly not predictive of real world performance. I could also go further and speculate that they are in the training data, although credible benchmarks avoid this, I'm just not sure how successful they are.by sulam
8/4/2026 at 4:42:22 PM
Thanks for the comment - I’ve actually been working on something to generate evals from private repos, as I 100% agree that public benchmarks are either contaminated (as OpenAI found with SWE-bench Verified), not high quality (as OpenAI found with SWE-bench Pro), or in the case of “good” benchmarks (Cognition’s Frontier Code comes to mind), non-representative of a specific repo’s work.Sharing some challenges / learnings from the journey so far:
* Tests alone are non-representative: an agent can pass tests but write the code in a way that’s subjectively worse than another test-passing result
* However, tests are still important and the best form of deterministic evaluation we have
* Creating executable environments to run the tests is challenging, especially when considering that this should be doable for arbitrary repos
* Selecting tasks that discriminate is an art - they need to be challenging, but not too challenging, represent the variety of work that’s done in the repo, and contain tests that failed prior to the change
* LLMs / agents are very powerful at judging code - but doing so in a way that is calibrated, consistent, and representative of codebase standards requires careful rubric creation and grading
* I’ve shifted towards using agents (from “static” LLM calls) to generate the rubrics, and to later grade the rubrics. This is powerful but even more non-deterministic
by bisonbear
8/4/2026 at 6:08:45 PM
> The problem is defining what quality means, and providing a way for agents to optimize the harness using that lever.I've built and my team uses plannotator [1] for specs and code review. It stores all human annotated feedback, which we compile from time-to-time and iterate on coding standard skills for automated review (that seems to get better with the more feedback we provide).
You could write a script that does the same for your PR review comments, which it sounds like you maybe do with a bit more structure.
by ramoz
8/4/2026 at 4:06:23 PM
[flagged]by clamshelldev