News for hackers. Stuff that matters.
Re:Ollama now supports AMD graphics cards by Zambyte on Friday March 15, 2024 @07:14PM (#39721665)
That is still one model per instance of Ollama, right?
Re:Ollama now supports AMD graphics cards by airocker on Friday March 15, 2024 @07:28PM (#39721758)
yes, not sure you can do better than that. You cannot still have one instance of LLM in (GPU) memory answer two queries at one time.
Re:Ollama now supports AMD graphics cards by eclectic29 on Friday March 15, 2024 @07:50PM (#39721907)
Of course, you can support concurrent requests. But Ollama doesn't support it and it's not meant for this purpose and that's perfectly ok. That's not the point though. For fast/perf scenarios, you're better off with vllm.
Re:Ollama now supports AMD graphics cards by airocker on Friday March 15, 2024 @07:53PM (#39721937)
Thanks! This is great to know.