News for hackers. Stuff that matters.
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by fsiefken on Thursday August 15, 2024 @06:15PM (#41261028)
It's good, but I'm already paying for GPT4o and Sonnet. How much memory does this need? If Alex Cheema (Exo Labs, Oxford) https://x.com/ac_crypto/status/1815969489990869369 could run Llama 3.1 405 Model on 2 macbooks, does this mean this can run on one macbook?
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by TradingPlaces on Thursday August 15, 2024 @07:22PM (#41261559)
I did not have great luck with 405b on an M1 Max Mac Studio with 64 GB of RAM via olama. I think memory was this issue. 70b runs with acceptable speed
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by Lerc on Thursday August 15, 2024 @08:09PM (#41261841)
64GB of ram is 550b bits of ram. That doesn't give you a lot of options on how many bits you want to keep per model parameter.
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by rabi_molar on Friday August 16, 2024 @07:32AM (#41265365)
Exo looks cool, but I wonder how long it will take until it supports llama.cpp. Definitely some spare machines I'd like to use this way. I also wonder to what extent other forms of native apple metal (iPhones, iPads) could be used in clusters for additional sources of aggregate cluster memory.