8/23/2026 at 10:26:16 AM
There's quite a few tangential features that must be implemented correctly or risk affecting the LLM output in significant ways.Parsing/encoding is one example: A couple of months ago I've debugged a reasoning loop bug in Step 3.7 Flash on llama.cpp that was caused by the parser capturing an extra `\n` as part of a reasoning block. It was something that only manifested at longer multi-turn agentic sessions, and the extra linefeed was steering the model into making reasoning self corrections that only got worse with longer sessions (more details about this issue: https://github.com/ggml-org/llama.cpp/issues/24181#issuecomm...)
No inference engine is perfect, but I feel that llama.cpp is the most reliable way to run language models locally.
by tarruda
8/23/2026 at 4:14:56 PM
Meanwhile, entirely too frequently I see unit tests that have atol / rtol values which permit errors of one part in 200 or even higher, where people bother with unit tests at all. At some point someone in the pytorch ecosystem used rtol=5e-03 in a unit test and crowds of people started passing that around unquestioned.With attention matrix sizes being what they are, that's high enough that you can literally zero out a row or two and still have the test suite pass. Guess how I know!
Check your numbers, folks.
by tsukikage
8/23/2026 at 2:14:32 PM
Debugging any LLM output when you also have done substantial harness engineering is a total pita and I wish there were better tools for it to isolate issues.I spent ages tracking down start appears to be an issue with the current Deepseek v4 flash 0731 version that would cause it to output giant walls of gibberish in Hermes with reasoning turned on.
by shostack
8/23/2026 at 2:06:10 PM
This is fascinating. I’m struggling to understand how that was causing such a large difference in the output. Is the “autoparser” vulnerable to injections somehow? How do you distinguish between user text, model text, and metadata, or is there ambiguity in the parsing?by catlifeonmars
8/23/2026 at 5:05:34 PM
My take, as someone who just read through the github issues conversation.The model was outputting reasoning traces that were supposed to lead to tool calls. So the model might do something like:
<think>
I should use a tool
</think>
... should make the tool call here
<think>
I should use a tool
</think>
by stillpointlab
8/23/2026 at 8:44:05 PM
> I’m struggling to understand how that was causing such a large difference in the output.It is incremental, the more a pattern appears in the context, the more likely it was to continue appearing in future turns.
So the model was likely trained to end a reasoning trace with a single linefeed and a `</think>`. It was also likely trained that two consecutive linefeeds sometimes produce an "Actually..." sequence.
So you can think of it as:
- the first time the reasoning trace was parsed, the two trailing linefeeds followed by </think> were added to the context by the template.
- next time the model was finishing a reasoning block, it added an extra linefeed instead of just closing directly with </think>. This slightly increases the chance that the next token will begin an "actually" sequence instead of closing.
- If it caused an actually, that was added to the context, further increasing the chance of a self correction at the end of the thinking block. The more self correction paragraphs are added, the higher the chance that following turns will have more.
- Eventually it can result in a state where it enters that loop forever (or at least for a very long time).
> Is the “autoparser” vulnerable to injections somehow?
The autoparser was (and still is) incorrectly parsing a trailing linefeed as part of a reasoning block. The were two ways to fix this, both of which must be implemented for the fix to be complete IMO:
- fix the autoparser definition to ensure remove surrounding whitespace is not returned as part of the text blocks
- trim leading/trailing whitespace in the encoding phase, so it fixes bugs or even "injections" where the client deliberately adds the whitespace to trigger problems.
For this specific issue, the maintainer later fixed by trimming the extra linefeed before passing to the template.
> How do you distinguish between user text, model text, and metadata, or is there ambiguity in the parsing?
That is model specific. Ultimately, a token stream is being produced and parsed by the inference engine, and each model uses different tokens/formats. The goal of the autoparser engine was to simplify the creation of parsers for new models by inferring the delimiter tokens from the chat template.
The llama.cpp API server returns pre-parsed data, so clients don't need to do any parsing to know what is a thinking block, a text block or a tool call.
by tarruda