News for hackers. Stuff that matters.
Re:Llamafile – The easiest way to run LLMs locally on your Mac by superkuh on Monday December 04, 2023 @04:42PM (#38523524)
It only takes away choice if you use the demo files with the models baked in. There are versions of this under the Releases->Assets that are only the actual llama.cpp OS portable binaries that you pass the model file path to as normal.

Compiling llama.cpp is relatively easy. Compiling llama.cpp for GPU support is a bit harder. I think it's nice this OS portable binaries of llama.cpp applications like main, server, and llava exist. Too bad there's no opencl ones. The only problem was baking in the models. Downloading applications off the internet is not that weird. After all, it's the recommended way to install Rust, etc.

Re:Llamafile – The easiest way to run LLMs locally on your Mac by wokwokwok on Monday December 04, 2023 @06:07PM (#38524507)
> Compiling llama.cpp is relatively easy. Compiling llama.cpp for GPU support is a bit harder.

It is not.

For Mac using M1 as per this specific post, you run “make”.

If you have an older Intel Mac and have to run using cpu, you run “make”.

> Downloading applications off the internet is not that weird. After all, it's the recommended way to install Rust, etc.

Downloading applications from trusted sources is not that weird.

Eg. Rust, from the rust organisation.

Downloading and running user uploaded binaries is a security nightmare.

Re:Llamafile – The easiest way to run LLMs locally on your Mac by paolop on Monday December 04, 2023 @06:41PM (#38524867)
While in general I agree with your security concerns, here the links are from very trusted sources (Mozilla Internet Ecosystem and Mozilla's innovation group) and the user is well known (present on X too with a large following).

Re: "simplicity", sure for you and I it's simple to compile llama.cpp, but it's like asking a regular user to compile their applications themselves. It's not that simple for them, and should not be required if we want to make AI and OSS AI in particular more mainstream.

Re:Llamafile – The easiest way to run LLMs locally on your Mac by wokwokwok on Monday December 04, 2023 @07:14PM (#38525246)
The command to run is:

Open terminal

curl -LO https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/ma...

chmod 755 …

./…

Open localhost:8080 in browser

To make this accessible to a broader cohort you would package it into an app and put it somewhere with provenance, eg. A well known GitHub account or App Store.

The solution, as shown, doesn’t solve either of the problems you’ve said are problems it attempts to solve.

It is a bad solution.

Re:Llamafile – The easiest way to run LLMs locally on your Mac by paolop on Monday December 04, 2023 @07:52PM (#38525612)
Totally agreed it's not yet ideal - absolutely. But I feel we are expanding the pie of users with this step, which is just an intermediate step. Do you want to work on that packaging ;-)?