News for hackers. Stuff that matters.

A comment on Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s. Parent comment

Re:Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX ... by geye1234 on Sunday October 04, 2026 @05:41PM (#49958189)
I'm running Pennyroyal's Docker image (on Podman) which uses sglang. I have a single RTX 6000 Blackwell and 128GB RAM. I turned off disk caching. I'm running with a ~500K context, but have been limiting it to 256K in the client (pi).

It always detects its spelling mistakes, btw, but it worried me. It may turn 'rm -rf ' into 'rm -rf /' one day.

Almost certainly the problem is my config, not the image.

[ Parent | on HN ]
Re:Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX ... by xiconfjs on Monday October 05, 2026 @02:11AM (#49961231)
Did you check your GPU for memory errors/defects?
[ Parent | on HN ]
Re:Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX ... by gewetensleegte on Monday October 05, 2026 @06:42AM (#49963137)
> It may turn 'rm -rf ' into 'rm -rf /' one day.

but surely you're running it in a container?

[ Parent | on HN ]
Re:Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX ... by phacker007 on Monday October 05, 2026 @09:48AM (#49964874)
are you using MTP with concurrent requests? I think there could be an issue with KV cache I had with llama.cpp that randomly threw in characters from other thread's KV cache...
[ Parent | on HN ]