alt.hn

2/16/2026 at 12:15:39 PM

Evaluating AGENTS.md: are they helpful for coding agents?

https://arxiv.org/abs/2602.11988

by mustaphah

2/16/2026 at 5:30:27 PM

My personal experience is that it’s worthwhile to put instructions, user-manual style, into the context. These are things like:

- How to build.

- How to run tests.

- How to work around the incredible crappiness of the codex-rs sandbox.

I also like to put in basic style-guide things like “the minimum Python version is 3.12.” Sadly I seem to also need “if you find yourself writing TypeVar, think again” because (unscientifically) it seems that putting the actual keyword that the agent should try not to use makes it more likely to remember the instructions.

by amluto

2/16/2026 at 6:28:24 PM

I also try to avoid negative instructions. No scientific proof, just a feeling the same as you, "do not delete the tmp file" can lead too often to deleting the tmp file.

by mlaretallack

2/16/2026 at 6:26:37 PM

I'd be interested to see results with Opus 4.6 or 4.5

Also, I bet the quality of these docs vary widely across both human and AI generated ones. Good Agents.md files should have progressive disclosure so only the items required by the task are pulled in (e.g. for DB schema related topics, see such and such a file).

Then there's the choice of pulling things into Agents.md vs skills which the article doesn't explore.

I do feel for the authors, since the article already feels old. The models and tooling around them are changing very quickly.

by pajtai

2/16/2026 at 7:01:44 PM

Progressive disclosure is good for reducing context usage but it also reduces the benefit of token caching. It might be a toss-up, given this research result.

by dpkirchner

2/16/2026 at 6:45:12 PM

Quite a surprising result: “across multiple coding agents and LLMs, we find that context files tend to reduce task success rates compared to providing no repository context, while also increasing inference cost by over 20%.”

by medler