Depending on what you are doing you may find all the models are written for CUDA though. I'm playing with gaussian splatting and it's like that. Codex is doing a good job porting stuff to Metal, but it would be easier & faster if I had Nvidia hardware to just run the models natively.

Not everything with AI is LLMs running on llama.cpp.

https://zluda.org/ seems to work at near-native performance from what I can find on Google.

Is that the legit site? Clicking the "testimonials" directed me to a link where Opera (with which malware is commonly bundled) auto-downloaded.

Maybe you have some malware or they have some targeted thing running, clicking the testimonials does nothing at all for me. I'll request to remove it just in case.

We've delinkified it above.

That's the thing, my understanding is that these Nvidia ones are supposed to have very impressive performance.

They're severely hampered by memory bandwidth

Yes, but in practice it’s more interesting than that

The massively lopsided compute makes speculative execution techniques (MTP/DFlash etc) way more effective, which makes modern MoE models quite usable

I’m running Qwen 3.8 Flash Next with Nota AI’s conservative NVFP4 quant at 30tk/s lowest decode, usually more around 45-55 for coding, and with prompt processing at a solid 2000tk/s

And that’s without very optimised kernels or anything, mostly standard vLLM