News for hackers. Stuff that matters.

A comment on Llamafile is the new best way to run a LLM on your own computer.

Re:Llamafile is the new best way to run a LLM on your own co... by superkuh on Friday December 01, 2023 @02:18PM (#38490942)
It's not the best way. It's a really cool and technically interesting way. But embedding the model with the executable is terrible for anything beyond a demo. The best way would be just running llama.cpp (what llamafile uses) and loading external models. I get that compiling is too difficult for some people and llamafiles are great for them. Maybe even the best for them.

But it's far from best for people that actually want to explore LLM and play.

[ Parent | on HN ]
Re:Llamafile is the new best way to run a LLM on your own co... by brandall10 on Friday December 01, 2023 @03:16PM (#38491711)
I'd argue LM Studio is the best way by far.

- Download and launch the app

- Use their interface to huggingface models, download the one you want

- Go to the chat window and select load

- Chat away w/ a ChatGPT like interface that includes markdown processing and easy copying of code blocks, etc

Outside of the time to download a model, it's about 30 seconds of work to get up and running

[ Parent | on HN ]
Re:Llamafile is the new best way to run a LLM on your own co... by breckenedge on Friday December 01, 2023 @03:02PM (#38491520)
Per the article:

> You can also download a much smaller llamafile binary from their releases, which can then execute any model that has been compiled to GGUF format

[ Parent | on HN ]
Re:Llamafile is the new best way to run a LLM on your own co... by dragonwriter on Friday December 01, 2023 @07:17PM (#38494429)
Or you can download LM Studio or oobabooga and execute any model compiled to GGUF format, and also models in a wide variety of formats that aren't GGUF.
[ Parent | on HN ]
Re:Llamafile is the new best way to run a LLM on your own co... by rodrigobellusci on Friday December 01, 2023 @03:07PM (#38491577)
From what I've read llamafile can also load other models if you pass a flag with a path to them. There's also LM Studio for anyone who'd like to play with a ChatGPT-like GUI.
[ Parent | on HN ]