Re:Unsloth GLM-5.2 – How to Run Locally
by phamilton on Monday June 22, 2026 @08:19PM
(#48638445)
Generation is basically just memory bandwidth math.
Each token has to read all the active weights. I think that's around 40B parameters active. At a 4-bit quant that's 20GB. With 100GB/s (replace with whatever your bandwidth is) and you get 5 tokens per second.
Re:Unsloth GLM-5.2 – How to Run Locally
by phamilton on Tuesday June 23, 2026 @10:19AM
(#48645382)
MTP on a MoE is hit or miss. If you're bottlenecked on memory, MTP can increase the number of active experts (like any batch processing would), which can eat away gains from it.
Each token has to read all the active weights. I think that's around 40B parameters active. At a 4-bit quant that's 20GB. With 100GB/s (replace with whatever your bandwidth is) and you get 5 tokens per second.