Re:Ollama now supports AMD graphics cards
by airocker on Friday March 15, 2024 @07:28PM
(#39721758)
yes, not sure you can do better than that. You cannot still have one instance of LLM in (GPU) memory answer two queries at one time.
[ Parent
| on HN ]
Re:Ollama now supports AMD graphics cards
by eclectic29 on Friday March 15, 2024 @07:50PM
(#39721907)
Of course, you can support concurrent requests. But Ollama doesn't support it and it's not meant for this purpose and that's perfectly ok. That's not the point though. For fast/perf scenarios, you're better off with vllm.