Re:Smollm3: Smol, multilingual, long-context reasoner LLM
by tripplyons on Tuesday July 08, 2025 @09:57PM
(#44505665)
Have you tried setting device="mps" to use Metal? It should be faster than PyTorch's "cpu" device on Mac.
Re:Smollm3: Smol, multilingual, long-context reasoner LLM
by reach-vb on Wednesday July 09, 2025 @05:11AM
(#44507822)
Hey Simon, VB from Hugging Face here and the person who added the model to MLX and llama.cpp (with Son). The PR hasn't yet landed on llama.cpp, hence it doesn't work OTB on llama.cpp installed via brew (similarly doesn't work with ollama since they need to bump their llama.cpp runtime)
The easiest would be to install llama.cpp from source: https://github.com/ggml-org/llama.cpp
If you want to avoid it, I added SmolLM3 to MLX-LM as well:
You can run it via `mlx_lm.chat --model "mlx-community/SmolLM3-3B-bf16"`
(requires the latest mlx-lm to be installed)
here's the MLX-lm PR if you're interested: https://github.com/ml-explore/mlx-lm/pull/272
similarly, llama.cpp here: https://github.com/ggml-org/llama.cpp/pull/14581
Let me know if you face any issues!
Re:Smollm3: Smol, multilingual, long-context reasoner LLM
by kosolam on Wednesday July 09, 2025 @07:33AM
(#44508731)
Could you please enlighten me regarding all these engines, I’m using lamacpp and ollama. Should I also try mlx, onnx, vllm, etc. I’m not quite sure whats the difference between all these. I’m running on CPU and sometimes GPU
Re:Smollm3: Smol, multilingual, long-context reasoner LLM
by pzo on Wednesday July 09, 2025 @11:00AM
(#44510843)
Ollama is a wrapper around llama.cpp thei using ggml format. Onnx is different ml model format and onnxruntime developer by microsoft. Mlx is ml framework from Apple. If you want the fastest speed on MacOS most likely stick with mlx
Re:Smollm3: Smol, multilingual, long-context reasoner LLM
by knowaveragejoe on Thursday July 10, 2025 @07:25PM
(#44526834)
> similarly doesn't work with ollama since they need to bump their llama.cpp runtime
Just curious, how frequently does that happen?
(llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'smollm3')
I've managed to run it using Python and transformers with PyTorch in device="cpu" mode but unsurprisingly that's really slow - it took 35s to respond to "say hi"!
Anyone had success with this on a Mac yet? I really want to get this running with tool calling, ideally via an OpenAI-compatible serving layer like llama-server.