8/1/2026 at 4:56:54 PM
I can only recommend to regularly measure how many tokens a harness+model combination uses for a certain taskThere are huge token efficiency/bloat differences between agents while working on the same tasks, using the same model, in the same environment
Yesterday I ran 10 agentic tasks using GPT 5.6 Sol in an ubuntu 26.04 vm a couple of times with different harnesses and got vastly different token usage.
+-------------+-----------+-----------+-----------+-----------+--------+
| Harness | API total | Input | Cached | Uncached | Output |
+-------------+-----------+-----------+-----------+-----------+--------+
| smol | 172,807 | 142,334 | 8,704 | 133,630 | 30,473 |
| Pi | 427,211 | 392,767 | 137,216 | 255,551 | 34,444 |
| OpenCode | 1,564,429 | 1,523,957 | 1,204,736 | 319,221 | 40,472 |
| Codex | 3,005,744 | 2,953,154 | 2,649,344 | 303,810 | 52,590 |
| Hermes | 3,856,611 | 3,808,231 | 3,167,232 | 640,999 | 48,380 |
| Claude Code | 5,073,137 | 5,029,969 | 4,587,008 | 442,961 | 43,168 |
+-------------+-----------+-----------+-----------+-----------+--------+
I'm not surprised that Claude Code is not optimized for an OpenAI model but I was still quite shocked re how much of a difference the harness makes.
Disclaimer: I'm working on 'smol' which is a minimalist harness but it's really nothing special, just a minimal system prompt, no skills files, only tool is shell
Do not underestimate how much popular harnesses are spamming the context window. The context window is very important.
by tosh
8/1/2026 at 5:04:41 PM
Claude Code injects a ton of tools into the system prompt, including their “memory system” that’s like 10k+ tokens. Depending on your task shape, this can easily double your task cost (e.g. a low context-using job that takes many turns, like a monitoring loop).You should use --disallowed-tools to prune any tools not needed for the task. Note that this is also a perpetual game of whack-a-mole since they’re always adding new tools.
by yojo
8/1/2026 at 5:53:52 PM
> Claude Code injects a ton of tools into the system promptIt's so unfortunate they don't let you use the subscription with other harnesses anymore - since even if I used OpenCode they'd still get a bunch of useful data from the API calls, meanwhile I could stretch their tier limits way further.
by KronisLV
8/1/2026 at 11:07:30 PM
The funny thing is that in some cases using a different harness with the subscription plan could actually be very good for Anthropic: e.g. if I were to use smol with Opus, it could use fewer tokens than CC for the same task. People on subscription plans burning fewer tokens is a good thing for Anthropic.The only downside for Anthropic that I can see is that hitting your limits more often (while using CC) could make you want to upgrade plans, and a more efficient harness could keep you from doing that. But I can't imagine the cost (to Anthropic) of those inefficient tokens is worth it to them.
by kelnos
8/1/2026 at 6:28:47 PM
Subsidized Anthropic subscriptions seem to work fine on the oh-my-pi harness, somehow.by epylar
8/1/2026 at 8:48:32 PM
Calling per token usage of the US closed source labs has always been funny to me, we have a clear model of what it actually costs to host these models from open models.Your subscription is not subsidised, it is just closer to the actual cost of the model…
by 3uler
8/2/2026 at 10:03:32 AM
Really? I immediately got a warning that the tokens would come from extra usage, and noped out immediately (so maybe pi lied to me?)by disgruntledphd2
8/1/2026 at 7:04:56 PM
According to ccusage I use about $5k of tokens on my $100/mo Claude Max plan and only hit 5-hr windows where I have to switch tools for a couple hours about once a weekby jeffrwells
8/1/2026 at 6:20:15 PM
It's so unfortunate people don't realize it's cheaper to write their own well working agent instead of insisting with general purpose bloated ones like Claude.200$/month is a lot of money on Luna/DS4 flash, like really a lot and the results are much better than clowning on bloated CC.
It's absurd how you have more and more organizations encoding their processes on LLMs and "engineers" (charlatan coders) don't even bother optimizing the tool they use most.
by epolanski
8/1/2026 at 6:38:46 PM
> the results are much better than clowning on bloated CC.I won't argue with the cost effectiveness, but the results are very much not better. Opus and Fable are in a different league than DS4 Flash. Even GPT Terra, which I really like overall, sometimes gets stuck in weird loops and starts to do stupid stuff once its context window fills up. Whereas I can more or less trust the big models to just Do The Thing™ on the first try.
With that said, you get way more value out of a GPT subscription than you do from Claude, partly because of the ability to use more efficient harnesses.
by RussianCow
8/2/2026 at 8:07:31 AM
You're confusing models with the agents.You can use opus or fable if you please, I'm advocating for writing your own agent instead of used generic bloated ones like CC.
by epolanski
8/2/2026 at 5:31:46 PM
I'm not confusing anything since you can't use custom harnesses with your Claude subscription—you have to use Claude Code. So as far as the Claude sub is concerned, the models and the harness are coupled.by RussianCow
8/2/2026 at 1:16:59 PM
Thanks, you make me feel better for spending the (fun!) time writing my first harness in Emacs Lisp with a ‘Emacs UI’ and later writing a command line harness in Common Lisp. Am I more productive with my own harnesses? Not yet, but the second harness I wrote in Common Lisp is promising, and I might write a little book just on this project to encourage people to hack what I wrote and make it their own.by mark_l_watson
8/1/2026 at 11:55:07 PM
I have yet to spend $200 on DS4 after two months of using it with an entire team.by trollbridge
8/2/2026 at 12:53:53 AM
It's cheaper still, and just as effective, to write code yourself instead of having the LLM do it for you.by bigstrat2003
8/1/2026 at 5:08:09 PM
The system prompt part is surely cached across all users.by visarga
8/1/2026 at 5:13:38 PM
You still pay cache token costs on API calls. Cache cost/token are 90% lower, but you pay it every single turn.I’m not sure if they let you skip the cache write cost on the first turn. That would imply cross-user caching infrastructure or special casing the default system prompt to give you a discount. Maybe? Away from the computer but you could try a “hello” in a fresh session and see what was billed.
by yojo
8/2/2026 at 1:10:10 AM
Its inexpensive to reserve KV cache for the first turn and would benefit users, given the first turn already requires costly locating and allocating a model slot for a user.So yeah, when they banned tgird party harnesses there was a technical and $ case to have.
by cyanydeez
8/1/2026 at 5:10:14 PM
For Anthropic models, yes. But OP was using Claude Code with GPT.by dcrazy
8/1/2026 at 5:17:09 PM
the system prompt still takes up useful space in the context window and steers the model into unnecessary actions and over-thinking patternsby tosh
8/1/2026 at 5:17:56 PM
ty re --disallowed-toolsfor coding agents 'shell' is often all you need (just make sure the environment has the necessary tools)
by tosh
8/1/2026 at 6:49:36 PM
appreciate the tip, kind netizenby colourclash
8/1/2026 at 6:27:51 PM
> Do not underestimate how much popular harnesses are spamming the context window. The context window is very important.Do we have any insight into whether it is actually spam and not useful info such as project or programming language specific context?
by bonestamp2
8/1/2026 at 8:58:56 PM
in my book anything that is (I'm sure well intentioned) and injected to help the agent — but doesn't help it — is a waste of tokensbut even injected context that when I read it sounds useful can oversteer the model and make it second guess or take a more complicated route than it normally would
(you can see this when looking at traces with and without that injected context)
often harnesses also mention in their system prompt locations of markdown files that the model can consult if the model thinks they might help
that hint alone as part of the system prompt can be strong enough to make the model read in more tokens than would have been necessary
'spam' is maybe a harsh way to say it
unfortunately I don't see an easy way other than to invest time and tokens into finding out which parts of the added context (in system prompt, injected in turns etc etc) are actually helpful or harmful and when
I'm just doing the easiest thing I could think of: start from nothing or close to nothing
that seems to work better than what most harnesses are doing
turns out GPT 5.6 Sol is all you need
by tosh
8/1/2026 at 11:03:52 PM
> in my book anything that is (I'm sure well intentioned) and injected to help the agent — but doesn't help it — is a waste of tokensHow do you tell, though? I guess what I'm asking is: the data you presented shows some crazy differences, but the token burn alone doesn't tell us enough. What was the output of the task like? Did the harnesses that burned fewer tokens give you as good a result as the ones that burned more?
I guess it's subjective, of course, but nearly everything about LLM use is...
by kelnos
8/2/2026 at 5:53:28 AM
I agree, only looking at the token burn is not enoughin this case it was 10 tasks and all harnesses could complete the tasks successfully, of course now the question is: will this hold for more and more complex tasks but I had to start somewhere :)
Checksum: Compute a file’s SHA-256 checksum and save the exact digest to an output file.
Log correlation: Correlate nested service logs to identify and summarize a request’s complete execution path.
CSV report: Parse quoted CSV data and aggregate paid orders and exact decimal totals by region.
JSONL join: Join related JSON Lines datasets and produce a correctly grouped and ordered report.
Archive repair: Find the correct version of a corrupted file in a tar archive and restore it.
SQLite migration: Safely migrate a SQLite database schema and verify the resulting data and constraints.
Python bug fix: Repair an interval-merging implementation so it passes visible and hidden edge-case tests.
Python CLI: Implement a robust command-line program that reads JSONL and reports validated statistics.
Multi-file feature: Add an atomic feature across a small Python package, CLI, and associated tests.
Pipeline repair: Fix a Make, shell, and Python reporting pipeline so it handles general input and passes verification.
by tosh
8/2/2026 at 6:57:53 AM
[dead]by DonHopkins
8/1/2026 at 10:57:23 PM
I am super curious how much in your opinion the extra prompting is useful?>often harnesses also mention in their system prompt locations of markdown files that the model can consult if the model thinks they might help >that hint alone as part of the system prompt can be strong enough to make the model read in more tokens than would have been necessary
I purposefully do this as I imagine it is useful. In my project I am seeing solid adherence to norms and a deep capacity to iterate on completed features. Essentially for each feature I have the model make a {featureName}.AGENTS.md at the root folder of the feature, where it maintains what is going on.
I am moving between Claude Code and Codex atm, but I began this pattern when pi + kimi 2.6 was my main driver.
by mikodin
8/2/2026 at 6:00:42 AM
I do think prompting and reference files (e.g. for architecture, tech stack, …) can be extremely helpful. I also do this in my projects (challenge is keeping drift of these documents at bay).What I wanted to emphasize is that whatever is in the context (whether system prompt or user message does 'steer' the model in a strong way, so everything in the context affects overall performance in a way. Even if it is 'just' net neutral it takes space up in the context window.
The context window is very very precious, everything that goes into it should help (not just hopefully help).
The challenge is coming up with good stuff to put into that context. A good agents.md file will be better context than whatever the popular harnesses have in their system prompt.
Also good to keep in mind that newer models are very good and more agentic than older models so they are better at exploring their environment based on the tasks you give them.
by tosh
8/2/2026 at 1:04:21 AM
I tailored an opencode agent with terse, minimum instruction, going as far as using ascii symbols and definitions.Works fine. My conclusion is treat it like a starting point and not a holy bible of everything the model needs to know.
I pair this with making sure task files are commented with headers like build scripts so context+instruction is consistent.
The happy path then is the LLM pulling in context where it needs it.
by cyanydeez
8/1/2026 at 11:51:39 PM
To be fair, 5.6 Sol is pretty much doing a ton of reasoning you can't see (and which you don't think about the cost of if you're using it via an OpenAI Pro subscription)by trollbridge
8/2/2026 at 1:31:22 AM
On Claude Code you can run /context and it'll explain everythingby nijave
8/1/2026 at 8:52:55 PM
If you're working on smol? How complex is your work? The agent has to do everything using sed? Did you write your own tools? I guess my question is, why aren't you using pi?The difference in tokens between the two also makes super curious. The system prompt can't be that different (I'd even bet Pi's shorter) and the 4 tools shouldn't make as much of a difference. I'm gonna have to try it.
by bermudi
8/1/2026 at 9:04:38 PM
smol only has 1 tool: shthe system prompt of smol is shorter than the system prompt of Pi
smol has no system prompt
system prompt of Pi 0.83.0
""" You are an expert coding assistant operating inside pi, a coding agent harness. You help users by reading files, executing commands, editing code, and writing new files.
Available tools: - read: Read file contents - bash: Execute bash commands (ls, grep, find, etc.) - edit: Make precise file edits with exact text replacement, including multiple disjoint edits in one call - write: Create or overwrite files
In addition to the tools above, you may have access to other custom tools depending on the project.
Guidelines: - Use bash for file operations like ls, rg, find - Use read to examine files instead of cat or sed. - Inspect PI_* environment variables for current model and session details. - Use edit for precise changes (edits[].oldText must match exactly) - When changing multiple separate locations in one file, use one edit call with multiple entries in edits[] instead of multiple edit calls - Each edits[].oldText is matched against the original file, not after earlier edits are applied. Do not emit overlapping or nested edits. Merge nearby changes into one edit. - Keep edits[].oldText as small as possible while still being unique in the file. Do not pad with large unchanged regions. - Use write only for new files or complete rewrites. - Be concise in your responses - Show file paths clearly when working with files
Pi documentation (read only when the user asks about pi itself, its SDK, extensions, themes, skills, or TUI): - Main documentation: /usr/local/lib/node_modules/@earendil-works/pi-coding-agent/README.md - Additional docs: /usr/local/lib/node_modules/@earendil-works/pi-coding-agent/docs - Examples: /usr/local/lib/node_modules/@earendil-works/pi-coding-agent/examples (extensions, custom tools, SDK) - When reading pi docs or examples, resolve docs/... under Additional docs and examples/... under Examples, not the current working directory - When asked about: extensions (docs/extensions.md, examples/extensions/), themes (docs/themes.md), skills (docs/skills.md), prompt templates (docs/prompt-templates.md), TUI components (docs/tui.md), keybindings (docs/keybindings.md), SDK integrations (docs/sdk.md), custom providers (docs/custom-provider.md), adding models (docs/models.md), pi packages (docs/packages.md), environment variables (docs/environment-variables.md) - When working on pi topics, read the docs and examples, and follow .md cross-references before implementing - Always read pi .md files completely and follow links to related docs (e.g., tui.md for TUI API details) Current working directory: /workspace """
by tosh
8/1/2026 at 8:38:53 PM
While there's a lot of bloat in the harnesses, almost all of it is cached. Your harness appears to be caching almost nothing. That will almost certainly bode extremely poorly for longer-running tasks than the simple ones you mentioned testing. Benchmaxxing for simplistic tasks could leave you not only suffering in completion rate on complex tasks but also possibly even not end up with the cost gains you're hoping for. And maybe that's fine if the only thing you use agents for is the very simple tasks, which is certainly a defensible decision, but probably bears mentioning when you're comparing it against harnesses with a different design goal.by applfanboysbgon
8/1/2026 at 8:47:41 PM
smol is also prefix cachingthe uncached tokens are also from runs where smol finished a task below 1024 tokens (the minimum amount of tokens needed to activate caching) which is less tokens than other harnesses are using for their system prompt (!)
> GPT-5.6 and later models: Caching is available for prefixes containing at least 1,024 tokens. This is a strict minimum.
https://developers.openai.com/api/docs/guides/prompt-caching
so in this specific case the count of uncached tokens for smol makes it look worse than it actually is
that said: it does makes sense to add more tasks that are difficult enough to fill the context window to compare the harnesses for how well they deal with compaction
staying below compaction (or with compaction at fewer compactions) is not only cheaper and faster, it also helps the agent stay on track
by tosh
8/1/2026 at 9:03:29 PM
Yes, I do agree that avoiding compactions is extremely desirable, just make sure you're actually testing for how well that happens and the results!by applfanboysbgon
8/1/2026 at 9:06:57 PM
will do!by tosh
8/1/2026 at 5:22:54 PM
What tools are you using to run the comparison? Or are you just running the prompts and checking them with something like ccusage?I’m asking because I’ve been looking for agent harness comparison tools too. I’m interested in more than just the inputs and outputs—I also want the system prompts, traces, and tool calls. It’s useful to understand why Codex, for example, uses more tokens while Pi doesn’t.
Fewer tokens aren’t necessarily better if the agent skipped important checks. On the other hand, using more tokens could just mean it’s overthinking the process. Either way, seeing the full execution trace for the same task is really valuable.
by azuanrb
8/1/2026 at 5:36:35 PM
I built an ad-hoc custom comparison framework to inspect system prompts, caching behaviour, tool call outputs, exact api requests and responses and so onI agree fewer tokens is not necessarily better but a bit counter-intuitively often the harness using fewer tokens is not only done faster but has better results
(that said: of course check the results, look at the full traces, agree!)
by tosh
8/1/2026 at 5:44:22 PM
Thanks! I ended up built my own too. Just thought there are other better options out there that I might've missed.by azuanrb
8/1/2026 at 5:35:21 PM
Running some local models and wondering what insanity was consuming 100k+ tokens to respond to "hello world" was rather eye-opening, yeah. Full of pointless fluff.by Groxx
8/2/2026 at 1:14:57 AM
I think prompt engineering went way to far into verbosity when at the same time agent and tool calling are much more important. Now a small system prompt can grab the contect it needs based on user context before bloating. On local models, i use dynamic context pruning.by cyanydeez
8/1/2026 at 8:39:58 PM
> The context window is very important.It is everything. My experience with Claude Code is that you have to decide when to compact to make it efficient. It defaults everything to 1M context and it will never keep it in check. It is strange how little cache reads you hit in smol, that may be a configuration issue.
by adoltech
8/1/2026 at 8:52:53 PM
I will look into it more to see if I have configured it wrong but I think the token efficiency hurts cache use as caching only starts at 1024 tokens so for tasks where smol is under or close to 1024 tokens most of them are uncachedby tosh
8/2/2026 at 1:32:56 AM
There's an env var you can set to tell it when to compact. It's especially helpful for models with big context that lose coherence much earlier.by nijave
8/1/2026 at 7:28:17 PM
> There are huge token efficiency/bloat differences between agents while working on the same tasks, using the same model, in the same environmentBe careful here. Remember these are non-deterministic models at the end of the day, and even with everything being "the same" you can have two runs where the same model, same harness, same tools can arrive at the same conclusion through a wildly different sequence of events.
by xienze
8/1/2026 at 9:09:57 PM
agree, that makes it a bit tricky to compare (esp if you also want to add different models and reasoning levels into the mix)I will add more tasks (esp longer ones) and think more about grading, the current tasks were easy to grade because the desired outcomes are well specced but I will also look into more open ended tasks and how to grade those
thank you!
by tosh
8/1/2026 at 7:17:48 PM
Thanks for sharing. Naming tangent: FWIW I have recent interest in "smolvm" microvms from smolmachines.com, which are unrelated.by chrisweekly
8/1/2026 at 4:59:50 PM
Same task? How did the results compare?by esafak
8/1/2026 at 5:03:34 PM
the tasks were all simple agentic taskslike creating a checksum of a file, merging csvs and so on, fixing a makefile pipeline
with known 'good' outcomes
all harnesses could reach the outcomes, only cost, time, number of tool uses and so on were different
(Claude Code failed once in 1 task but I think that was just an unfortunate outlier, the tasks aren't that difficult)
by tosh
8/1/2026 at 5:13:31 PM
Anything else I can read about smol?also, are you using a tool to collect those metrics? what is it?
by brettgriffin
8/1/2026 at 5:16:02 PM
smol is basically this 9 line python agent re-implemented in Gohttps://news.ycombinator.com/item?id=49006862
I'll have more about it in the next hours/days, you can follow me on twitter in the meantime (https://x.com/__tosh)
by tosh
8/1/2026 at 6:19:19 PM
So smol is your agent? Do you have a GitHub link?by growt
8/1/2026 at 9:43:00 PM
coming in a few hoursyou can follow this org in the meantime https://github.com/smol-env
or on twitter here: https://x.com/__tosh
by tosh
8/2/2026 at 6:14:09 PM
repo is published now: https://github.com/smol-env/smolby tosh
8/1/2026 at 7:06:45 PM
this benchmark is a very misleading because it doesn't necessarily point to these harnesses saving usage and much of the harness that comes with codex for example is designed at cross functional agentic usage which you will be leaving behind by using a different harness. openai spends a lot of time as does claude. also note that cached is what makes up bulk of the total and codex and claude has huge discounts in that scenario.the "spamming" is mostly compaction appending files, tool artifacts, images in its summary there is a known github issue for codex. only solution is periodic clean up but its also how a lot of the agentic orchestration is performed and able to work for days.
by zuzululu
8/1/2026 at 9:16:12 PM
smol currently is very simple so it definitely does less things, like no subagent orchestrationI will look into how token usage looks like for longer sessions and more complex tasks
re caching: the cache ratio for this bench looks 'bad' for smol because it often finishes a task before caching kicks in (caching starts at 1024 tokens)
thank you for flagging this
by tosh
8/2/2026 at 5:46:58 AM
sorry i didn't mean to call you out just wanted to point out at the comments thinking this means using sol/pi you can get many fold savings in tokens which isn't truesmol looks interesting i think it could have a potential niche although tokens are only going to get cheaper and cheaper here
by zuzululu
8/1/2026 at 10:28:06 PM
that's why i recommend running everything through a router.by try-working
8/2/2026 at 12:23:36 AM
I think it’s too easy to trick you into token counting by checking tokens used. I think we should only focus on Price, time and productivity. Some kind of ratioSo a cheap model with many tokens or an expensive model with much less tokens could potentially be same price.
Maybe only time will be the differential factor.
by skeptic_ai
8/1/2026 at 8:05:08 PM
[dead]by ignore_prev
8/2/2026 at 12:34:10 AM
[dead]by catlover76
