News for hackers. Stuff that matters.

A comment on Show HN: Ollama for Linux – Run LLMs on Linux with GPU Acceleration. Parent comment

Re:Show HN: Ollama for Linux – Run LLMs on Linux with GPU Ac... by Patrick_Devine on Tuesday September 26, 2023 @01:44PM (#37663016)
We don't current compile in CLBlast or ROCm support but if there's a lot of demand for this, we'll definitely add it in the future. One concern is not wanting to bloat out the binary size too much (CUDA is already huge!) but given how big the LLM models are anyway, maybe it's not a huge concern.
[ Parent | on HN ]
Re:Show HN: Ollama for Linux – Run LLMs on Linux with GPU Ac... by globuous on Tuesday September 26, 2023 @02:15PM (#37663541)
AMD support would be amazing <3

I get the boot concern, and the maintenance concern (!!!), but as you say, these models are already quite huge anyway :)

[ Parent | on HN ]
Re:Show HN: Ollama for Linux – Run LLMs on Linux with GPU Ac... by capableweb on Tuesday September 26, 2023 @01:57PM (#37663262)
Offer two builds :) One AMD and one NVIDIA.
[ Parent | on HN ]
Re:Show HN: Ollama for Linux – Run LLMs on Linux with GPU Ac... by Patrick_Devine on Tuesday September 26, 2023 @02:11PM (#37663473)
Possibly, but a core guiding principle for us is to keep everything as simple as possible. We're a small project, so if we add too many features/permutations it really makes it hard to keep everything working!
[ Parent | on HN ]
Re:Show HN: Ollama for Linux – Run LLMs on Linux with GPU Ac... by nebster on Tuesday September 26, 2023 @03:52PM (#37664960)
Also, some of us weird folk have both an AMD and NVIDIA GPU in one machine...

Actually, I guess that is more common now that AMD CPUs have a GPU built in?

[ Parent | on HN ]
Re:Show HN: Ollama for Linux – Run LLMs on Linux with GPU Ac... by zamalek on Tuesday September 26, 2023 @05:58PM (#37666331)
You could download both binaries for that scenario.
[ Parent | on HN ]
Re:Show HN: Ollama for Linux – Run LLMs on Linux with GPU Ac... by zamalek on Tuesday September 26, 2023 @06:00PM (#37666341)
> permutations

I don't believe that AMD/NVIDIA is a low entropy bit so far as configuration permutations go. Although NVIDIA is far more widespread, AMD has significant market share. The Darwin bit you already facilitate for is probably lower entropy.

[ Parent | on HN ]