At least Makeayo asks where my models are now. It's obnoxious that I have to use symlinks for comfy/automatic....
All they need to do is ask me where my stuff is on first run, and an area in the config to update that setting. Not so hard!
./llamafile-server-0.1 -m llama-2-13b.Q8_0.gguf
My experience is that the world of Python dependency management is a mess which sometimes works, and sometimes forces you to spend hours-to-days searching for obscure error messages and trying maybe-fixes posted in Github issues for some other package, just in case it helps. This sometimes extends further - e.g. with hours-to-days spent trying to install just-the-right-version-of-CUDA on Linux...
Anyway, the (somewhat annoying but understandable) solution that some developers take is to make their utility/app/whatever as self-contained as possible with a fresh install of everything from Python downwards inside a venv - which results in (for example) multiple copies of PyTorch spread around your HDD. This is great for less technical users who just need a minimal-difficulty install (as IME it works maybe 80-90% of the time), good for people who don't want to spend their time debugging incompatibilities between different library versions, but frustrating for the more technically-inclined user.
This is just another approach to the same problem, which presumably also presents an even-lower level of work for the maintainers, since it avoids Python installs and packages altogether?
What I suspect is happening is that people are cargo-culting zero-click installations. It seems rather fashionable right now.
"You can also also download just the llamafile software (without any weights included) from our releases page, or directly in your terminal or command prompt"
There is no cargo-culting going on. Some of us do legitimately appreciate it.
In this specific case there is an option for you that addresses this complaint, where the model isn't coupled with the app.
Is having everything normalized in your system that worth it? I would say having (some) duplicates in your system is mostly fine, better that having some spooky-action-at-a-distance break things when you don't expect.
I expect the future is something like Windows's WinSxS, NixOS's /nix/store, pnpm's .pnpm-store where that deduping isn't "online" but it still is somewhat automated and hidden from you.
Yes? It's right here, at the least.
And if that's the future, then the future sucks. We can teach people to be smarter, but no, instead our software has to bend over backwards to blow smoke up our ass because grandma.
I think you’re imposing your own very specific (and solvable!) gripe onto an unrelated project and it makes for pretty clumsy commentary.
But also: It might not be for a developer like you, but it is for a developer like me.
I enjoy writing software, but I don't particularly enjoy futzing with building things outside my day-to-day work, and on systems I don't write myself. If it was up to me everything would be one click.
Things like this are like accessibility: it benefits me even though I don't particularly need it.
I am utterly amazed and perplexed that answering ONE configuration question (which is all that is needed here) is too much. How lazy have we become?
One question. That is too much for folks?!
And I have no objection to the autodownloading. Just ask me where to put it first!
Well, that's on the MAS/TestFlight for not doing delta updates.
You could have multiple updates my using say iso time and doing a sort (so that ~/.freechat.run.20231127120000 would be overriden by ~/.freechat.run.20231129160000 without making the user delete anything)
._.
Pain.
> I'm kind of struggling with the use-cases for this one.
IMO cosmopolitan libc is a "really neat trick". And it deserves praise and it probably does have some real use cases. But it's not practical for most purposes. If we had a format like ELF that was so fat as to support as many architectures and OSs as desired, would we be using that? I have a feeling that we would not.
Then again -- after having used "zig cc" for a while, maybe it would be reasonable to have something like "one build" that produces a mega-fat binary.
And the microarch-specific dispatch is a nice touch.
...maybe I'm convincing myself of the alternative....
[x] Doubt.
That user was caught stealing code and banned from llama.cpp by its creator (your [2] citation) https://news.ycombinator.com/item?id=35411909
Maybe the same thing is happening here. Plagiarism of code.
They made a PR that was built on top of another PR. The authorship information was preserved in the git history, and there was no attempt at deception. They also supposedly collaborated with the author of the original PR (which was never denied by either of them). All of this is totally normal working practice.
Those allegations of "stealing" just stem from a GH user piling onto the drama from the breaking change by pointing out where the initials from the new file format come from (which wasn't called into question on the original PR).
They were also not banned for those stealing allegations. They, as well as the author from the reversal PR were banned, as the maintainer deemed the resulting "drama" from the breaking changes to be a distraction to the project goals. The maintainer accepted the PR, and the nature of the breaking changes was obviously stated, so that drama wasn't completely on jart.
See this post https://news.ycombinator.com/item?id=35418066
And of course, I can imagine where the whole cosmopolitan thing comes from,... even as manifest of sorts for the idea of systems-neutrality and potentially gender fluidity. But I really wonder whether GGUF actually needs this, since llama.cpp already compiles and runs pretty much everywhere.
Why introduce one more container? Who benefits from binary distribution of this sort?
Mass adoption comes from ease-of-use, which includes ease-of-deployment.
Most of the HN crowd is savvy enough to run HF models directly in their terminal, but there's a reason Ollama, LM Studio, and Faraday are so popular.
Until LLM runners are part of the base OS, methods like this are extremely helpful to reduce friction.
About your question in the third paragraph: This is totally orthogonal to GGUF, and a cursory reading of the README shows that it does uses GGUF. This is not about a new universal LLM format, this is about packing it in a universal executable that runs everywhere, using Cosmopolitan.
Some examples do pack the executable and GGUF weights together in a single file, but that's not dissimilar from an self-executing zip, the only difference is that this executable is not OS-specific, so you can use the same exact binary for macOS or Linux, for example.
Well, it simplifies things when you don't need to compile things.
Also, you literally can't download or compile the wrong binary by mistake, it's the same binary for all supported processor/OSes Cartesian product matrix.
> Why introduce one more container?
It makes stuff more convenient.
`application/zip` is also a ubiquitous standard. I doubt anyone is being "introduced to it".
I also appreciate the fact that tooling for handling `application/zip` is very widespread, so you don't need totally bespoke tooling to retrieve the models from inside a `llamafile`.
> Who benefits from binary distribution of this sort?
Anyone that doesn't have a compiler SDK on their computer.
I make a small macOS app [1] which runs llama.cpp with a SwiftUI front-end. For the first version of the app I was obsessed with the single download -> chat flow and making 0 network connections. I bundled a model with the app and you could just download, open, and start using it. Easy! But as soon as I wanted to release a UI update to my TestFlight beta testers, I was causing them to download another 3GB. All 3 users complained :). My first change after that was decoupling the default model download and the UI so that I can ship app updates that are about 5MB. It feels like someone using this tool is going to hit the same problem pretty quick when they want to get the latest llama.cpp updates (ggerganov SHIIIIPS [2]). Maybe there are cases where that doesn't matter, would love to hear where people think this could be useful.
[0]: https://justine.lol/cosmopolitan/
[1]: https://www.freechat.run
[2]: https://github.com/ggerganov/llama.cpp