News for hackers. Stuff that matters.
Re:Llamafile lets you distribute and run LLMs with a single ... by Luc on Wednesday November 29, 2023 @02:44PM (#38464251)
This is pretty darn crazy. One file runs on 6 operating systems, with GPU support.
Re:Llamafile lets you distribute and run LLMs with a single ... by tfinch on Wednesday November 29, 2023 @02:51PM (#38464340)
yeah the section on how the GPU support works is wild!
Re:Llamafile lets you distribute and run LLMs with a single ... by amelius on Wednesday November 29, 2023 @04:18PM (#38465463)
Why don't package managers do stuff like this?
Re:Llamafile lets you distribute and run LLMs with a single ... by thelastparadise on Wednesday November 29, 2023 @04:01PM (#38465261)
So if you share a binary with a friend you'd have to have them install cuda toolkit too?

Seems like a dealbreaker for the whole idea.

Re:Llamafile lets you distribute and run LLMs with a single ... by brucethemoose2 on Wednesday November 29, 2023 @04:10PM (#38465372)
> On Windows, that usually means you need to open up the MSVC x64 native command prompt and run llamafile there, for the first invocation, so it can build a DLL with native GPU support. After that, $CUDA_PATH/bin still usually needs to be on the $PATH so the GGML DLL can find its other CUDA dependencies.

Yeah, I think the setup lost most users there.

A separate model/app approach (like Koboldcpp) seems way easier TBH.

Also, GPU support is assumed to be CUDA or Metal.

Re:Llamafile lets you distribute and run LLMs with a single ... by jart on Wednesday November 29, 2023 @04:43PM (#38465727)
Author here. llamafile will work on stock Windows installs using CPU inference. No CUDA or MSVC or DLLs are required! The dev tools are only required to be installed, right now, if you want get faster GPU performance.
Re:Llamafile lets you distribute and run LLMs with a single ... by vsnf on Thursday November 30, 2023 @02:36AM (#38470777)
My attempt to run it with the my VS 2022 dev console and a newly downloaded CUDA installation ended in flames as the compilation stopped with "error limit reached", followed by it defaulting to a CPU run.

It does run on the CPU though, so at least that's pretty cool.

Re:Llamafile lets you distribute and run LLMs with a single ... by jart on Thursday November 30, 2023 @02:53AM (#38470862)
I've received a lot of good advice today on how we can potentially improve our Nvidia story so that nvcc doesn't need to be installed. With a little bit of luck, you'll have releases soon that get your GPU support working.
Re:Llamafile lets you distribute and run LLMs with a single ... by abareplace on Thursday November 30, 2023 @07:18AM (#38472590)
The CPU usage is around 30% when idle (not handling any HTTP requests) under Windows, so you won't want to keep this app running in background. Otherwise, it's a nice try.
Re:Llamafile lets you distribute and run LLMs with a single ... by fragmede on Wednesday November 29, 2023 @04:17PM (#38465453)
I'm sure doing better by windows users is on the roadmap, exec then reexec to get into the right runtime, but it's a good first step towards making things easy.
Re:Llamafile lets you distribute and run LLMs with a single ... by quickthrower2 on Wednesday November 29, 2023 @04:48PM (#38465775)
Like a docker for LLMs
Re:Llamafile lets you distribute and run LLMs with a single ... by verdverm on Wednesday November 29, 2023 @09:37PM (#38468714)
I don't see why you cannot use a container for LLMs, that's how we've shipping and deploying runnable models for years
Re:Llamafile lets you distribute and run LLMs with a single ... by simonw on Wednesday November 29, 2023 @09:44PM (#38468784)
Being able to run a LLM without first installing and setting up Docker or similar feels like a big win to me.

Is there an easy way to run a Docker container on macOS such that it can access the GPU?

Re:Llamafile lets you distribute and run LLMs with a single ... by verdverm on Wednesday November 29, 2023 @10:06PM (#38468972)
Not sure, I use cloud VMs for ML stuff

We definitely prefer to use the same tech stack for dev and production, we already have docker (mostly migrated to nerdctl actually)

Can this project do production deploys to the cloud? Is it worth adding more tech to the stack for this use-case? I often wonder how much devops gets reimplemented in more specialized fields