Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model
by fsiefken on Thursday August 15, 2024 @06:15PM
(#41261028)
It's good, but I'm already paying for GPT4o and Sonnet.
How much memory does this need? If Alex Cheema (Exo Labs, Oxford) https://x.com/ac_crypto/status/1815969489990869369 could run Llama 3.1 405 Model on 2 macbooks, does this mean this can run on one macbook?
[ Parent
| on HN ]
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model
by TradingPlaces on Thursday August 15, 2024 @07:22PM
(#41261559)
I did not have great luck with 405b on an M1 Max Mac Studio with 64 GB of RAM via olama. I think memory was this issue. 70b runs with acceptable speed
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model
by Lerc on Thursday August 15, 2024 @08:09PM
(#41261841)
64GB of ram is 550b bits of ram. That doesn't give you a lot of options on how many bits you want to keep per model parameter.
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model
by rabi_molar on Friday August 16, 2024 @07:32AM
(#41265365)
Exo looks cool, but I wonder how long it will take until it supports llama.cpp. Definitely some spare machines I'd like to use this way. I also wonder to what extent other forms of native apple metal (iPhones, iPads) could be used in clusters for additional sources of aggregate cluster memory.