News for hackers. Stuff that matters.
Re:Ollama now supports AMD graphics cards by jameshart on Friday March 15, 2024 @08:03PM (#39721994)
Sure.

Just be aware that there’s a lot of expressive difference between building on top of an HTTP API vs on top of a direct interface to the token sampler and model state.

Re:Ollama now supports AMD graphics cards by verdverm on Friday March 15, 2024 @08:12PM (#39722040)
I'm aware, I don't need that amount of sophistication yet.

Python seems to be the way to go deeper though. Is there a good reason I should be aware of to pick llama.cpp over python?

Re:Ollama now supports AMD graphics cards by jameshart on Friday March 15, 2024 @08:36PM (#39722187)
Python’s as good a choice as any for the application layer. You’re either going to be using PyTorch or llama-cpp-python to get the CUDA stuff working - both rely on native compiled C/C++ code to access GPUs and manage memory at the scale needed for LLMs. I’m not actually up to speed on the current state of the game there but my understanding is that llama.cpp’s less generic approach has allowed it to focus on specifically optimizing performance of llama-style LLMs.
Re:Ollama now supports AMD graphics cards by verdverm on Friday March 15, 2024 @08:51PM (#39722281)
I've seen more of the model fiddling, like logits restrictions and layer dropping, implemented in python, which is why I ask

Most of AI has centralized around Python, I see more of my code moving that way, like how I'm using LlamaIndex as my primary interface now, which supports ollama and many more model loaders / APIs