News for hackers. Stuff that matters.
Re:Llamafile lets you distribute and run LLMs with a single ... by reisse on Thursday November 30, 2023 @04:22AM (#38471457)
What is the point of wrapping absolutely portable single-file program into a Docker container, honest question?

Looks like cargo cult for me.

Re:Llamafile lets you distribute and run LLMs with a single ... by finnjohnsen2 on Thursday November 30, 2023 @04:37AM (#38471541)
I see this as not polluting my OS (filesystem and processes) with bits and bobs I downloaded off the internet. The cargo cult is a clean, safe and warm space and I highly recommend it.
Re:Llamafile lets you distribute and run LLMs with a single ... by reisse on Thursday November 30, 2023 @07:36AM (#38472727)
I see you and other commenters don't quite understand my point. If you're wrapping model into a docker container, you don't need amalgamated single file version. It makes it harder to upgrade llamafile/model weights separately afterwards, it needs you to store separate llamafile binary for each container, etc, etc. Why not just build proper layered image with separate layer for llama.cpp and separate layer or volume for model?

Cargo cult is not in using Docker, but in using Docker to wrap something already wrapped into a comparable layer of abstraction.

Besides,

> not polluting my OS (filesystem and processes) with bits and bobs I downloaded off the internet

is purely self-deception. It's not like Docker images are not stored in some folder deep in the filesystem. If anything, it's harder to clean up after Docker than just doing rm -rf on a directory with llamafiles.

Re:Llamafile lets you distribute and run LLMs with a single ... by simonw on Thursday November 30, 2023 @08:08AM (#38473005)
If you want to use Docker then you can go ahead - I don't see anyone claiming that this brand new, highly experimental project should be used by everyone instead of Docker.

There are tradeoffs hers. For some people the tradeoff of a single executable file with everything in it compared to setting up a Docker system is worthwhile.

Re:Llamafile lets you distribute and run LLMs with a single ... by reisse on Thursday November 30, 2023 @08:11AM (#38473037)
Sure. I just question why people want to use both simultaneously.
Re:Llamafile lets you distribute and run LLMs with a single ... by xylophile on Thursday November 30, 2023 @04:46PM (#38479650)
Do you routinely download unsigned binaries of unprovable provenance and run them? Because if you do, you might eventually find reason to appreciate the additional isolation that namespaces et al give you very conveniently via Docker (or your favorite alternative).
Re:Llamafile lets you distribute and run LLMs with a single ... by badrequest on Thursday November 30, 2023 @02:18PM (#38477846)
`docker system prune --force --all --volumes` and they're gone.
Re:Llamafile lets you distribute and run LLMs with a single ... by topato on Saturday December 02, 2023 @04:35AM (#38497358)
Ah, Ive been meaning to distro hop, from Arch,t4 GG Huover to Vanilla OS or BlendOS, for the past few weeks; I can't
Re:Llamafile lets you distribute and run LLMs with a single ... by tluyben2 on Thursday November 30, 2023 @05:21AM (#38471825)
Personally I wanted to see if this llama can generate the required docker files to run itself just from pasting in the instructions from the my parent commenter and asking for docker/docker-compose. It did and it works.
Re:Llamafile lets you distribute and run LLMs with a single ... by reisse on Thursday November 30, 2023 @07:39AM (#38472756)
Well that's a good reason I didn't think about, thank you!
Re:Llamafile lets you distribute and run LLMs with a single ... by oefrha on Thursday November 30, 2023 @04:41AM (#38471555)
Wrapping it in Docker makes it harder to access your filesystem without some dedicated jailbreak.
Re:Llamafile lets you distribute and run LLMs with a single ... by rudasn on Thursday November 30, 2023 @11:13AM (#38475266)
Volumes maybe? I don't see the issue here.
Re:Llamafile lets you distribute and run LLMs with a single ... by oefrha on Thursday November 30, 2023 @12:21PM (#38476121)
Like the other comment said, not being able to access fs is a feature.
Re:Llamafile lets you distribute and run LLMs with a single ... by czbond on Thursday November 30, 2023 @12:09PM (#38475968)
security - i have no idea if an executable will not be a malicious actor to my system. Dockerizing it, and removing access to system files - is the reason.

I am not trusting that a tradition virus / etc scanner will find stuff hidden in executable models.

Re:Llamafile lets you distribute and run LLMs with a single ... by belter on Thursday November 30, 2023 @04:30AM (#38471495)
Deploy your models into a Kubernetes cluster and let them fight for resources to death? A modern Roman Gladiators circus with Models?
Re:Llamafile lets you distribute and run LLMs with a single ... by ukuina on Thursday November 30, 2023 @11:00AM (#38475088)
Give them write access to your control-plane and the first one to write syntactically correct IaC wins!
Re:Llamafile lets you distribute and run LLMs with a single ... by Bjartr on Thursday November 30, 2023 @12:21PM (#38476125)
More like a sequel to Core War... Which actually does sound pretty amusing now that I think about it. Call it Cloud War.

https://en.m.wikipedia.org/wiki/Core_War

Re:Llamafile lets you distribute and run LLMs with a single ... by devneelpatel on Thursday November 30, 2023 @05:02AM (#38471683)
Homelabbers like me have a docker swarm cluster / k8s cluster so this def helps!