Actually Qwen3.8-flash-next comes very close to a "nerfed frontier model" the way I host it locally at 4.75bpw with int8 kv. It is pretty much equivalent to opus 4.6 up to 4.8 on coding. What it lacks is some factual knowledge - this is to be expected, it's a far smaller model. But if you run the kind of coding tasks where you have to spec them in high detail anyway (large very custom environment) this model is very usable.

I run it on a 6*rtx3090 consumer PC (no threadripper etc sadly) with very constrained pcie topology (4 cards sit on 2 lanes basically) and i still get 7k prefill and 50t/s single user or 200t/s 7 users. All at full 262k contexts of course and with 96GB of ram kv cache that means I can easily have 10+ Claude code sessions open with 1-7 generating at any time.

I've been the person that said, before opus 4.6 came out that this AI stuff is absolute crap for programming. Then it suddenly wasn't and I was using it many hours each day.

Local AI was the same. It was all crap. Models like GLM5.3 Flash that are many times larger than that qwen, lots of people are oooing and ahhing over I found way too inferior. Previous DeepSeek V4.1 Flash 0731 was atrocious. They benchmaxed it to hell, but at least it was small.

The current model of same name but release 1017 and double the size is currently the best open source model IMO. It even beats my self hosted Qwen3.8-flash-next.

I've tried hosting that DeepSeek model, but even adding another rtx3090 for a total of 7 cards and my 192gb of ram bought when it cost under $1000 I get only about 25t/s decode and few hundred prefill. That is far too slow for me to use.

But, I believe once this pyramid scheme of circular financing finally bursts there will be a glut of hardware on the market where people like me will be able to buy equipment to run 750B models like we now run 120B.

Just curious, did you put all these cards in a server or desktop case ? and what kind of motherboard ?

Also, how much power does that consume at full tilt ?

I've been very impressed by it and recommended it to everyone. 32gb ram, 12gb 3080ti, and I run 3.8 flash next coder (half the experts ripped out, leaving in coding) int8 kv 128k context at 35-55tk/s and I agree with the opus comparison; I personally say 4.7 is what it seems like the most.

Truly insane times

I read stuff like this and just think that if these open models work well enough financially you'd still be better off using them via OpenRouter or some other hoster.

For Deepseek V4 Flash (not 4.1), you can use two 128GB unified memory boxes.

That's around 1% of the price of 8x B200.

you can run Qwen3.8-27b with a single GPU.