News for hackers. Stuff that matters.
Re:Ollama now supports AMD graphics cards by jameshart on Friday March 15, 2024 @08:36PM (#39722187)
Python’s as good a choice as any for the application layer. You’re either going to be using PyTorch or llama-cpp-python to get the CUDA stuff working - both rely on native compiled C/C++ code to access GPUs and manage memory at the scale needed for LLMs. I’m not actually up to speed on the current state of the game there but my understanding is that llama.cpp’s less generic approach has allowed it to focus on specifically optimizing performance of llama-style LLMs.
Re:Ollama now supports AMD graphics cards by verdverm on Friday March 15, 2024 @08:51PM (#39722281)
I've seen more of the model fiddling, like logits restrictions and layer dropping, implemented in python, which is why I ask

Most of AI has centralized around Python, I see more of my code moving that way, like how I'm using LlamaIndex as my primary interface now, which supports ollama and many more model loaders / APIs