Smollm3: Smol, multilingual, long-context reasoner LLM

Re:Smollm3: Smol, multilingual, long-context reasoner LLM by wizee on Wednesday July 09, 2025 @10:30AM (#44510494)

You should use flash attention with KV cache quantization. I routinely use Qwen 3 14B with the full 128k context and it fits in under 24 GB VRAM. On my Pixel 8, I've successfully used Qwen 3 4B with 8K context (again with flash attention and KV cache quantization).

[ Parent | on HN ]

Re:Smollm3: Smol, multilingual, long-context reasoner LLM by sigmoid10 on Thursday July 10, 2025 @04:43AM (#44518755)

>On my Pixel 8, I've successfully used Qwen 3 4B

How many tokens/s? I can't imagine that this would run in any practical way.

[ Parent | on HN ]