Re:Llamafile lets you distribute and run LLMs with a single ...
by Zuiii on Wednesday November 29, 2023 @10:27PM
(#38469165)
That's why I always download the original version and quantize myself. With enough swap, you can do it with a modest amount for ram. I never had to download a model twice.
But yes, unless there is a way to patch it, bundling the model with the executable like this is going to be more wasteful.
But yes, unless there is a way to patch it, bundling the model with the executable like this is going to be more wasteful.