Re:Llama.cpp supports Vulkan. why doesn't Ollama?
by zozbot234 on Friday January 31, 2025 @09:03AM
(#42887763)
Given that the 671B model is reportedly MoE-based, it definitely could be powering the web interface and API. MoE slashes the per-inference compute cost - and when serving the model for multiple users you only have to host a single copy of the model params in memory, so the bulk doesn't hurt you as much.
[ Parent
| on HN ]