8/24/2026 at 4:05:37 PM
I did a similar thing running Q6_K model and Q8_0 DFlash2 (draft=7) quants: DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-NM-DAU-NEO-MAX-MTP-GGUF
incoai/Qwen3.8-27B-DFlash2-GGUF
by karmakaze
8/24/2026 at 5:23:50 PM
wow i need to check the 32GB cards. how did you handle the KV cache. Oh i see it was the Q6 model. Did you have to cut down on context window?by LuD1161
8/24/2026 at 5:30:26 PM
I used Q8 kv cache and using 64K context but can go a bit higher. The DFlash2 model takes a few gigs and using Q6_K (rather than a Q6_K_M/Q6_K_XL that unsloth publishes) saves some more. Also using Vulkan that has less VRAM overhead, but ROCm has faster prefill. Overall I'm quite pleased with it, but need to give it more agentic work. I also run an Qwen3.5-9B (another DavidAU finetune) for subagent editing/etc as well as an document embedding model (on different GPU). Using MTP reclaims the DFlash2 model memory and only a bit slower so a viable path for more context.by karmakaze