Re:Llamafile lets you distribute and run LLMs with a single ...
by quickthrower2 on Wednesday November 29, 2023 @04:48PM
(#38465775)
Like a docker for LLMs
[ Parent
| on HN ]
Re:Llamafile lets you distribute and run LLMs with a single ...
by verdverm on Wednesday November 29, 2023 @09:37PM
(#38468714)
I don't see why you cannot use a container for LLMs, that's how we've shipping and deploying runnable models for years
Re:Llamafile lets you distribute and run LLMs with a single ...
by simonw on Wednesday November 29, 2023 @09:44PM
(#38468784)
Being able to run a LLM without first installing and setting up Docker or similar feels like a big win to me.
Is there an easy way to run a Docker container on macOS such that it can access the GPU?
Re:Llamafile lets you distribute and run LLMs with a single ...
by verdverm on Wednesday November 29, 2023 @10:06PM
(#38468972)
Not sure, I use cloud VMs for ML stuff
We definitely prefer to use the same tech stack for dev and production, we already have docker (mostly migrated to nerdctl actually)
Can this project do production deploys to the cloud? Is it worth adding more tech to the stack for this use-case? I often wonder how much devops gets reimplemented in more specialized fields