8/17/2026 at 6:24:02 PM
As a comparison, Qwen3.6 27B scores 38, which was the highest in its small model category (4B–40B).Qwen3.8 27B beats all medium models (40B–150B). It has the same score as DeepSeek V4 Flash 0731, which ranks #5 in large model category (> 150B).
Sources:
- https://artificialanalysis.ai/models/open-source/small
by beltsazar
8/17/2026 at 6:32:40 PM
Simon Willison's post about this gives a good context on why exactly this is happening. While it doesn't mention this in the Artificial Analysis page, this is likely with Max reasoning, which has extremely long reasoning traces:https://simonwillison.net/2026/Aug/16/qwen-38-27b/
It seems like the token usage is 2.3x GPT Luna Max and almost 2x Kimi K3!
I'm curious if they can make up for this with insanely high tokens-per-second especially when served from hosted providers, though, given how tiny it is (37B!)
by phsource
8/17/2026 at 6:48:46 PM
Qwen models are slower in tokens/s, compared to similarly sized gemma4 and others, and they use more tokens per task, in part thanks to that xhigh default.On the other hand, there are some of us who are stuck with hardware that has plenty of compute, but limited (V)RAM. The new 27B is just perfect for that.
by kees99
8/17/2026 at 7:00:13 PM
> Qwen models are slower in tokens/s, compared to similarly sized gemma4 and othersNo? Gemma 31B and Qwen 27B are about the same speed. Gemma 26B-A4B and Qwen 35B-A3B are about the same speed.
by petu
8/17/2026 at 7:18:47 PM
What configuration are you using? On both vllm and llama-cpp, I get significantly higher speeds from gemma4 than qwen3.6 (with their respective speculative decoding methods).Output TPS in vllm for instance:
- Gemma4 26B-A4B: 200-300TPS
- Qwen3.6 35B-A3B: 120-180TPS
- Gemma4 31B: 80-120TPS
- Qwen3.6 27B: 60-80TPS
This is for a first request on a dual 5090 setup, with their respective speculative decoding methods.
by trouve_search
8/17/2026 at 8:13:20 PM
Single 3090 under llama.cpp: | model | size | test | t/s |
| ------------------- | ------- | ------ | ---- |
| gemma4 31B Q4_0 | 16.1 GB | pp2048 | 1248 |
| gemma4 31B Q4_0 | 16.1 GB | tg512 | 40 |
| qwen35 27B Q4_K | 15.9 GB | pp2048 | 1248 |
| qwen35 27B Q4_K | 15.9 GB | tg512 | 39 |
| gemma4 26B.A4B Q4_0 | 13.3 GB | pp2048 | 4304 |
| gemma4 26B.A4B Q4_0 | 13.3 GB | tg512 | 160 |
| qwen35 35B.A3B Q3_K | 15.7 GB | pp2048 | 3329 |
| qwen35 35B.A3B Q3_K | 15.7 GB | tg512 | 144 |
You're benchmarking drafter acceptance rate, then. Which is real life values, yes, but attributing worse drafter performance to the other 95% of the model being inherently slower.
by petu
8/18/2026 at 3:24:08 PM
I think it's a vllm vs llama_cpp performance thing, will pay more into it.One note I had between the two is that gemma has a much higher prefix cache hit rate in general.
by trouve_search
8/18/2026 at 12:15:42 AM
Dual 4090, getting 85-113 t/s depending on task (draft seems to speed up quite a lot, disproportionately more for content like svg etc): ./llama.cpp/llama-server \
-hf unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL \
--webui-mcp-proxy \
--no-mmproj \
--parallel 1 \
--kv-unified \
--flash-attn on \
--fit off \
--split-mode tensor \
-ngl 999 \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
-ub 256 \
--no-context-shift \
--host 0.0.0.0 \
--tools all \
--jinja \
--ctx-size 262144 \
--spec-type draft-mtp \
--spec-draft-n-max 3 \
--reasoning on \
--chat-template-kwargs '{"reasoning_effort":"medium"}' \
--reasoning-preserve \
--temp 1.0 \
--top-p 0.95 \
--top-k 20 \
--min-p 0.0 \
--presence-penalty 0.0 \
--repeat-penalty 1.0
IMHO draft model support on dense models is great alternative to MoE on GPUs (high bandwidth, less memory) – more intelligence, speed somewhere mid way there which is usually sufficient.
by mirekrusin
8/18/2026 at 3:33:13 PM
thanks for posting your setup! I think it's smart to set the reasoning effort default to something saner in the base config.Here's a VLLM command for 3.6 (I'll update to 3.8 today) to test out:
```
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
vllm serve Qwen/Qwen3.6-27B-FP8 \
--dtype auto \
--kv-cache-dtype fp8 \
--enable-chunked-prefill \
--enable-prefix-caching \
--trust-remote-code \
--enable-auto-tool-choice \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder \
--speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":3}' \
--default-chat-template-kwargs '{
"enable_thinking": true,
"reasoning_effort":"medium"
}' \
--tensor-parallel-size 2 \
--max-model-len 250000 \
--gpu-memory-utilization 0.9 \
--max-num-batched 12000 \
--max-num-seqs 24
I took the liberty of adding your reasoning effort chat template to my setup. You can play around with the last few parameters. In generall VLLM will be better in higher concurrency scenarios, so if you only use it for a personal vibe coding assistant and less as a general home model for task execution llama.cpp may be better.
by trouve_search
8/18/2026 at 6:52:21 PM
yes, you know, personal use doesn't necessary mean no concurrency.it's good to play with harness setup where you fan out multiple concurrent branches that share non trivial amount of prefix then reduce their output/summary back into main agent.
ie. instead of serially reading further skills/relevant source files for planning/thinking, you can branch and read them in parallel reusing prefix / or use to to approach request from different angles in parallel - to map-reduce result onto main context of what's actually relevant. branching subagents has benefits of not polluting main context, shared prefix prefill is close to free on a cache hit and with concurrent decoding/continuous batching you can utilize gpu well to get good speedups.
ie. what's relevant is number of active concurrent sequences (and their shape, ie. shared prefix), not so much number of users.
i'm not sure with llama.cpp vs vllm regarding concurrency – llama server has multiple server slots, continuous/dynamic batching enabled by default, prompt caching (also on by default), ram prompt cache, context checkpoints, unified kv buffer across sequences etc. so shouldn't be bad, i guess would be good to actually benchmark. personally i'm happy with llama.cpp.
by mirekrusin
8/17/2026 at 10:28:25 PM
Have you tried running it on a single 5090? Dual 5090 require https://github.com/aikitoria/open-gpu-kernel-modules for higher perf. Are you using TP?by xfalcox
8/18/2026 at 3:26:11 PM
Yes, I mentioned the setup, but on vllm you can only use TP with speculative decoding or pipeline parallelism without, so there's tradeoff to both.I gave general numbers of what I'm getting above, the performance ratios seemed similar regardless of setup (eg. getting a AWQ-in4 quant on a single GPU vs PP without speculative decoding vs TP with speculative decoding).
Overall single GPU is fastest, and TP+speculative decoding is still faster than PP, but for fp8 models you need dual GPUs whether you want it or not.
by trouve_search
8/17/2026 at 7:04:29 PM
There's no Qwen3.8-35B-A3B though.by stymaar
8/17/2026 at 8:47:20 PM
I benched Qwen 3.6 35B-A3B against Qwen 3.8 27B with the same parameters, thinking set to low. Despite 35B having 9x fewer active parameters, it benched only 2.34x slower. The 35B got only 50% more agentic tasks done per hour.by hadlock
8/17/2026 at 10:57:59 PM
i feel like Simon omitted an important part of how Qwen's "reasoning" levels work. they are just one sentence additions/omissions to the system promptxhigh -> "Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer."
medium -> no mention of effort (sentence omitted)
low -> "Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration."
in my testing this doesn't seem to produce exactly deterministic thinking levels, because it's just a system prompt nudge. i had instances where medium thought longer than xhigh
by thousand_nights
8/17/2026 at 11:11:43 PM
The models are post-trained on these prompt additions so they’re more structural than thinking of them as “system prompts” suggests. (All LLMs ever see is tokens going in, so even the concept of a system prompt is just formatting they’ve seen in post-training.)You can also apply fixed token budgets for the reasoning blocks, though it will decrease quality in some cases.
by anon373839
8/18/2026 at 1:05:35 AM
Why not invent a few magic token values for reasoning level instead? It would be like 4 out of a vocabulary of 200k and save like 30 tokens in every promptby nixon_why69
8/18/2026 at 1:00:56 AM
yes of course, I understand that. but I feel like it would've been nice to include in the article because the main point of it is the effort and overthinkingby thousand_nights
8/17/2026 at 6:37:32 PM
I'm running 3.8 27B locally, and the results from the past few days have been excellent. I find raw speed is less of an issue when you can trust the model more to reach the right result.by skohan
8/17/2026 at 7:08:00 PM
> insanely high tokens-per-second especially when served from hosted providers, though, given how tiny it is (37B!)It's a dense model so it will use all of its parameters per token. 37B active parameters isn't tiny at all, it's almost what Deepseek R1 had, and it's 2/3 of what Kimi k3 uses, so it's not going to be “insanely high” tps: it's going to be three times slower than Deepseek Flash (Prefil speed is going to be quite high though, but not token generation).
by stymaar
8/17/2026 at 8:34:45 PM
Its 27B not 37B and having just 27B in total and 3T and like 30B active of those is still totally different. A 120B with 5B active is still much slower than a proper 5B. Just like the new Ling 3.0 Tiny with 8B and 1B active only gets around 120tk/s compared to 250tk/s which a real 1B one gets on my hardware.by Azantys
8/17/2026 at 9:39:48 PM
Thanks for mentioning Ling 3 Tiny. This model has completely bypassed me and seems promising for how small it is.by jakswa
8/17/2026 at 9:21:15 PM
On the other hand, it used about the same tokens as GLM 5.2 and got 1 point lower score.The fact that we have a GLM 5.2-class model that can run on two 3090's comfortably at Q8 is absolutely insane. It wasn't long ago that GLM 5.2 was considered amazing for open weight models.
by 2001zhaozhao
8/17/2026 at 6:36:27 PM
A ThinkingCap variant of Qwen 3.8 27b would be extremely interesting.https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B
And then a Bonsai ternary on top of that model.
by ArvidSu
8/17/2026 at 7:04:32 PM
Re: bonsai - unsloth's quants have Q2 (UD-IQ2) variants, which are more or less same in size....or did Prism do something special with their "bonsai" releases? I didn't notice anything like QAT being mentioned.
by kees99
8/17/2026 at 7:12:30 PM
There is some special sauce that they have. It’s not just a simple quant of another release. Or so they imply. I don’t have any insight into how it works or what the Bonsai special sauce is.by drob518
8/20/2026 at 12:03:01 AM
I suggest nobody repeats claims of "special sauce" if such "special sauce" is not understood.by DoctorOetker
8/17/2026 at 7:06:58 PM
It’s still going to chew up context quickly. Surely, some of the added tokens are helping the model, but does it require as many as it generates? What happens on long, multi step tasks as it pushes old tokens out of context? I’m not sure we know the answers to those.by drob518
8/18/2026 at 7:28:16 AM
Damn, beating Kimi k3 is crazy. It produces already a ton of tokens.But I think the next step will be even more thinking on smaller models. Maybe fine-tuned and we get really crazy stuff
by Bombthecat
8/18/2026 at 3:50:01 PM
It doesn't beat k3, it doesn't even beat deepseekv4Flash-0731 let alone glm5.2by segmondy
8/17/2026 at 6:58:31 PM
It's fun to see "test time" scaling work out so well, maybe the best example of all.by kzrdude
8/17/2026 at 10:24:39 PM
This only makes me understand how flawed AAII is. This Qwen model is nowhere close to the other models in that score range.by bermudi