News for hackers. Stuff that matters.
Re:Llamafile lets you distribute and run LLMs with a single ... by jxy on Wednesday November 29, 2023 @10:08PM (#38468994)
so next time llama.cpp releases an update, other people update their favorite backend, you redownload a 4.26 GB file. Epic.

EDIT: oh, wait. Actually people usually have a handful to a few dozen of the these models lying around. When they update their backend, you just redownload every single model again.

EDIT 2: right, you can release a program that automatically patches and updates the downloaded model+executables. Such an invention.

Re:Llamafile lets you distribute and run LLMs with a single ... by simonw on Wednesday November 29, 2023 @10:52PM (#38469369)
Each llamafile is a .zip, so if you want to extract the weights out of it you can extract the gguf file directly.

    unzip -l llamafile-server-0.1-llava-v1.5-7b-q4 | grep llava-v1
    Archive:  llamafile-server-0.1-llava-v1.5-7b-q4
    4081004224  11-15-2023 22:13   llava-v1.5-7b-Q4_K.gguf
    177415936  11-15-2023 22:13   llava-v1.5-7b-mmproj-Q4_0.gguf
Re:Llamafile lets you distribute and run LLMs with a single ... by aphroz on Wednesday November 29, 2023 @11:11PM (#38469520)
Compared to modern bandwidth usage that's not such a big size anymore. Everyday millions of people download 100gb video games, watch 4k video podcasts, etc.
Re:Llamafile lets you distribute and run LLMs with a single ... by simonw on Wednesday November 29, 2023 @11:23PM (#38469614)
You can even run a full LLM in your browser these days - try https://webllm.mlc.ai/ in Chrome, it can load up a Llama-2-7b chat model (~4000MB, took my connection just under 3 minutes) and you can start chatting with it.
Re:Llamafile lets you distribute and run LLMs with a single ... by Zuiii on Wednesday November 29, 2023 @10:27PM (#38469165)
That's why I always download the original version and quantize myself. With enough swap, you can do it with a modest amount for ram. I never had to download a model twice.

But yes, unless there is a way to patch it, bundling the model with the executable like this is going to be more wasteful.

Re:Llamafile lets you distribute and run LLMs with a single ... by column on Thursday November 30, 2023 @02:48AM (#38470834)
This is for convenience. You can also download a 4.45Mb executable (llamafile-server-0.1) and pass any GGUF model as an argument.

> llamafile-server-0.1 -m llama-2-13b.Q8_0.gguf

https://github.com/Mozilla-Ocho/llamafile/releases/tag/0.1

Re:Llamafile lets you distribute and run LLMs with a single ... by taneq on Thursday November 30, 2023 @03:47AM (#38471242)
Spoken like someone who hasn’t spent hours trying to get LocalAI to build and run, only to find out that while it’s “OpenAI API compatible!0” it doesn’t support streaming so the Mattermost OpenAI plugin doesn’t work. I finally gave up and went back to ooba (which also didn’t work with the MM plugin… hmm.) Next time I’ll just hack something on the side of llama.cpp
Re:Llamafile lets you distribute and run LLMs with a single ... by _kidlike on Thursday November 30, 2023 @04:04AM (#38471342)
salty much?

You know, most people don't have 24+GB GPUs sitting around to train these models. So in my book this is a huge step forward. Personally, this is the first time i am able to run an LLM on my computer, and it's purely thanks to this.