Flash Next is really great, but you still need pretty decent hardware. A DGX Spark, Strix Halo, or Mac with a lot of memory. You can use things like Strata to squeeze it into a system with only 32GB of VRAM by offloading a lot of it to system memory and still get something that "works", but the quality falls off a cliff too. At that point you get better results out of Qwen 3.8 27b in int4.
Never go below a four bit quant if you want good results. A worse quant of a bigger model is often easily beaten by the small model at decent quality. A lot of the people who think local AI suck think that because they don't understand that low quants == dumbass model. You aren't just compressing data, you're reducing the amount of compute it uses which effectively lowers it's IQ.
If you want to run something actually useful for cheap Qwen 3.8 27b on a 32GB GPU in int4 should be passable. It's more like Sonnet for a year ago. To match last years Opus you really need at least 128Gb.
Personally I have two Sparks connected with ConnectX 7. Now I can run GLM 5.3 locally at 30-40 tps and it's honestly been great. It really cut my dependence on cloud providers and I can run an abliterated version that never refuses my Cyber related tasks.
Folks like to think of 'weights' as just storage, but they aren't (except with stuff like engrams). Those digits of precision encode actual computation, not just storage. There is always a cost, it's lossy compression after all. It's like MP3 vs AAC. If you can't hear the difference it just means you haven't encountered it YET.
That's not to say that you ever will. The cost could be in weird far out edge cases, but whatever it once encoded... It's gone now and it ain't coming back.