The Strix Halo and DGX Spark are pretty danged slow, relatively speaking. I don't recall exact numbers, but with MoE models in this size ballpark (Laguna S 2.1), I seem to recall I was seeing about 20-25 t/s with a big context, which is close to usable. Qwen 3.8 27B crawls on this hardware, though, at 10-16 t/s, definitely not comfortable for interactive use. (Though this makes it seem like you can cook pretty good with a 4-bit ROCmFP4 quantization: https://github.com/julianmb/q38rocm the model does get notably dumber below six bits.)

A model similar in size to Laguna S 2.1, but with only 6B active parameters, should be a notable amount faster, so I would imagine 25-30 t/s would be a reasonable guess for where Qwen 3.8 Flash Next will land.

DFlash2 might improve all these numbers. It wasn't available last I was testing new models on the Strix Halo; I've only used MTP (which doesn't generally improve MoE models, but I believe DFlash2 can).

Given software improvements, I'm hopeful an MoE in this size range will be the sweet spot that pushes past 40 t/s and is also smart enough for real work. Qwen 3.8 27B is finally a self-hostable model that's smart enough, but it thinks so hard it still isn't really useful for agentic interactive use.

On the DGX I get 44.5 tokens per second (NVFP4). With 8 concurrent it's 241 t/s total.

I am using the PrismaAQUA

standard 9.7 t/s

+ Dflash2 30 t/s

+ torch-compile 37 t/s

c8 = 177 t/s

What model? Also, I don't know what "the PrismaAQUA" means, ddg thinks it's a CPAP machine, which seems unlikely to help with inference performance.

Also, 4-bit has measurable intelligence loss. Sometimes worth it, but, at this size models are barely smart enough at 8 or 6.

Qwen3.8-27B-PrismaAQUA-5.5bit-vllm

The output quality is higher. It's held at full precision (not quantized).

You can only use up to 90gb for the GPU, so it doesn't fit

No. In Linux, you can ignore the BIOS (well, set it to 512MB) and set GTT to allow almost the entirety of memory for GPU. If you don't run a desktop GUI (and why would you), you don't need memory for much of anything else. Just a few GB for the OS and llama.cpp.

These are roughly the settings I use: https://github.com/kyuz0/amd-strix-halo-toolboxes#kernel-par...