News for hackers. Stuff that matters.
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update by whatifitoldyou on Saturday November 14, 2026 @09:29AM (#50497284)
I could not find one that's why I built my own. It only covers python and C++ but at least it is not benchmaxed. I specifically tried to focus on models that are runnable in consumer GPUs https://sevangelatos.github.io/spiruxbench-results/
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update by plipt on Saturday November 14, 2026 @12:43PM (#50499018)
Thanks for sharing

I see you already have average tokens per task but one aspect of model comparisons I find helpful to see is tokens per second or a general sense of time till completion.

Obviously a MOE is going to be faster than a similarly sized dense model, but getting a wrong answer with a smaller / faster model in a third of the time can allow for a second try, sometimes before the smarter / slower model can finish its first response. I feel like that is a useful metric to compare models on.

Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update by whatifitoldyou on Saturday November 14, 2026 @01:58PM (#50499743)
I will add an average time per task collumn when I get the chance. I feel tokens per second is a very fluid metric though as it varies wildly by hardware, inference engine and configuration as well as MTP config. If you can fit these models in VRAM (aka you have at least 24gigs of VRAM) you should be able to get at least 50/60 tps from the slower dense models and easily more than 100 tps from the MOE models. But it depends on lots of factors.