8/19/2026 at 11:12:13 PM
Watch the video carefully. DFlash2's tool call fails on python syntax.Usually models in this class nail things like that 1 shot, which the other side did.
I don't know the cause. It may be nothing. But I'd like to see the model doing something where its path is a bit more constrained, to help out rule out such oddities.
by ilc
8/20/2026 at 12:31:26 AM
DFlash is lossless so this would be a bug in the implementation if it is indeed a regression against the target model.by zackangelo
8/20/2026 at 12:45:59 AM
Only if you do greedy sampling. With probabilisitic sampling (categorical sampling), you will end up with different trajectory just “mathematically equivalent”.by liuliu
8/20/2026 at 1:33:26 AM
Can you explain a little bit more please?by stymaar
8/20/2026 at 4:36:57 AM
Not OP, but you can influence how deterministic your LLM behaves using the temperature setting. The neural network doesn't directly output tokens, but logits which are then converted to probabilities and then a token is chosen at random, unless the temperature is 0 (i.e. greedy, we just always pick the most probable token without any randomness). All speculative decoding methods have to "commit" to a token though even when they don't know the actual logits of the full size NN yet. The question then is (and I don't know the answer): how do the common inference engines behave when the speculation landed on the most probable token, but the random choice still doesn't land on it? You can imagine that in the interest of performance as long as we stay reasonably inside the probability we just go ahead with the speculation. Not sure if thats implemented like that though.Edit: I just looked up the math, and actually the idea of speculative decoding is done in a clever way that fully preserves the probability distribution while still maximizing the acceptance rate of draft tokens. So I would have to disagree with OP and say that no, non-greedy sampling doesn't influence the trajectories.
by stkdump
8/20/2026 at 6:54:44 AM
Yes, it doesn’t impact the probability distribution due to verifier. However, remember how you use PRNG and effectively due to the drafter is sampled from a different distribution initially, a separate rejection sampling won’t be able to recover what the “old PRNG” would choose in a “without drafter” case. Hence in my original post, it is about different trajectories you will end up with, not the correctness of each stochastic sampling.by liuliu
8/20/2026 at 7:49:14 AM
If you assume that the RNG generates true randomness, then the two are identical. Only if you care about the determinism of the RNG (for example you want to use identical seeds and get the exact same generation between the two) it makes a real difference.by stkdump
8/20/2026 at 8:00:31 AM
Correct. I am trying to explain why even it is "exact", the generated text is different from the with / without DFlash2 runs, and potentially why the DFlash2 run will contain the invalid Python syntax.by liuliu
8/20/2026 at 9:40:28 AM
If the underlying probability distributions are the same, then DFlash can lead to an invalid Python Syntax iif the autoregressive process could have generated one if the random sampling picked a different token.If a model can output a “wrong” sequence with a certain probability p, then Dflash can also output the wrong sequence with the same probability. They wouldn't necessarily produce the same output from the same seed, but speculative decoding shouldn't be able to produce anything that the autoregressive model couldn't also produce when using a different seed.
Or am I misunderstanding something?
by stymaar
8/20/2026 at 6:29:38 PM
I agree. But I think the DFlash2 case is just that 1/1000 invalid syntax failure case from sampling rather than a bug.by liuliu
8/20/2026 at 9:40:31 AM
It will contain the same or different syntax with or without it. Also multiple runs without DFlash2 will contain the same or different syntax. And multiple runs with DFlash2 will have the same or different syntax with the same probability. DFlash2 literally has no influence (unless buggy). The difference is purely caused by the randomness.by stkdump
8/20/2026 at 9:39:02 PM
[dead]by zhijianliu