Re:Jamesob's guide to running SOTA LLMs locally
by turova on Friday July 03, 2026 @12:53PM
(#48777139)
For qwen3.6-27b you can also run the q4 variant with full ~250K context on one 3090. It's fast enough to not be frustrating so the speed gains with 2x 3090s wouldn't be worth it to me. Running a q6 on 2x 3090s at half the speed with a smaller context is an option, but you're really not going to compete with SOTA models there anyway so unless you already have 2x 3090s, I would say 1 is the best investment given current prices. It's good enough to do a lot, especially with a well-configured harness.
[ Parent
| on HN ]
Re:Jamesob's guide to running SOTA LLMs locally
by nabakin on Friday July 03, 2026 @03:52PM
(#48779207)
Are you running qwen3.6-27b on one 3090 with your KV cache at q4? Ime there is significant long-context recall accuracy degradation at that precision. I prefer putting the KV cache at q8 and working with the 120k context
Re:Jamesob's guide to running SOTA LLMs locally
by Der_Einzige on Friday July 03, 2026 @04:48PM
(#48779785)
Use modern samplers and you don’t need to limit yourself to 8bit at half the context window. I could push it down to 1.58 bits and get decently good output easily by simply not using the garbage default top_p and top_k that vendors continue to wrongly recommend.
Re:Jamesob's guide to running SOTA LLMs locally
by anon373839 on Friday July 03, 2026 @05:39PM
(#48780321)
Where do you find optimal samplers and sampler settings for these models? Very interested in this as I, too, use Q8.