News for hackers. Stuff that matters.
Re:A comprehensive guide to running Llama 2 locally by Patrick_Devine on Tuesday July 25, 2023 @03:16PM (#36867871)
Ollama works with Windows and Linux as well too, but doesn't (yet) have GPU support for those platforms. You have to compile it yourself (it's a simple `go build .`), but should work fine (albeit slow). The benefit is you can still pull the llama2 model really easily (with `ollama pull llama2`) and even use it with other runners.

DISCLAIMER: I'm one of the developers behind Ollama.

Re:A comprehensive guide to running Llama 2 locally by mschuster91 on Tuesday July 25, 2023 @04:23PM (#36868811)
> DISCLAIMER: I'm one of the developers behind Ollama.

I got a feature suggestion - would it be possible to have the ollama CLI automatically start up the GUI/daemon if it's not running? There's only so much stuff one can keep in a Macbook Air's auto start.

Re:A comprehensive guide to running Llama 2 locally by jmorgan on Tuesday July 25, 2023 @04:38PM (#36869062)
Good suggestion! This is definitely on the radar, so that running `ollama` will start the server when its needed (instead of erroring!): https://github.com/jmorganca/ollama/issues/47
Re:A comprehensive guide to running Llama 2 locally by DennisP on Tuesday July 25, 2023 @05:32PM (#36869870)
I've been wondering, is the M2's neural engine usable for this?
Re:A comprehensive guide to running Llama 2 locally by npsomaratna on Tuesday July 25, 2023 @10:26PM (#36872404)
I think you'd need to offload the model into CoreML to do so, right? My understanding is that none of the current popular inference frameworks do this (not yet, at least).
Re:A comprehensive guide to running Llama 2 locally by DennisP on Wednesday July 26, 2023 @10:14AM (#36878071)
Re:A comprehensive guide to running Llama 2 locally by sanex on Tuesday July 25, 2023 @08:08PM (#36871441)
Matt Williams talk today at that conference was a great intro to Ollama.