News for hackers. Stuff that matters.
Re:Llamafile lets you distribute and run LLMs with a single ... by lol768 on Wednesday November 29, 2023 @06:03PM (#38466568)

    $ chmod +x llamafile-server-0.1-llava-v1.5-7b-q4
    $ ./llamafile-server-0.1-llava-v1.5-7b-q4 
    run-detectors: unable to find an interpreter for ./llamafile-server-0.1-llava-v1.5-7b-q4
Hmm. Did I do something wrong? (Ubuntu 22.04 / )

Installing the portable binfmt_misc gets me further, but still:

    $ ./llamafile-server-0.1-llava-v1.5-7b-q4 
    zsh: permission denied: ./llamafile-server-0.1-llava-v1.5-7b-q4

    $ sh -c ./llamafile-server-0.1-llava-v1.5-7b-q4
    sh: 1: ./llamafile-server-0.1-llava-v1.5-7b-q4: Permission denied
Re:Llamafile lets you distribute and run LLMs with a single ... by jart on Wednesday November 29, 2023 @08:27PM (#38468003)
You can solve the run-detectors issue with:

    sudo wget -O /usr/bin/ape https://cosmo.zip/pub/cosmos/bin/ape-$(uname -m).elf
    sudo sh -c "echo ':APE:M::MZqFpD::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register"
    sudo sh -c "echo ':APE-jart:M::jartsr::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register"
You can solve the zsh permission denied issue by either (1) upgrade to zsh 5.9+ (I upstreamed a fix for this bug in zsh two years ago) or (2) use the sh -c workaround you discovered. If that one doesn't work, then it likely needs to be chmod +x. If the execute bit is set, and your sh still isn't working, then please let me know, because I'm not aware of any sh that still doesn't support APE.

See the Gotchas section of the README https://github.com/mozilla-Ocho/llamafile#gotchas

Re:Llamafile lets you distribute and run LLMs with a single ... by stavros on Wednesday November 29, 2023 @08:35PM (#38468078)
That worked, thanks Justine! I use fish, so I didn't get a zsh error, but I had missed the Gotchas section (and the README), so this helps!
Re:Llamafile lets you distribute and run LLMs with a single ... by jart on Wednesday November 29, 2023 @08:40PM (#38468124)
Fish is another cool shell I got to help improve two years ago by upstreaming a patch for this. So long as you're using a recent version, you should be golden (provided binfmt_misc doesn't cause any issues). Let us know what you think of llamafile!
Re:Llamafile lets you distribute and run LLMs with a single ... by stavros on Sunday December 03, 2023 @05:07PM (#38511219)
Thank you, I really like it! It's a very clever way to get LLMs deployed, and with Cosmopolitan, I don't need to point people to different downloads for the same LLM. Excellent job.
Re:Llamafile lets you distribute and run LLMs with a single ... by jart on Thursday December 07, 2023 @01:33AM (#38553505)
Thanks!
Re:Llamafile lets you distribute and run LLMs with a single ... by pmarreck on Wednesday November 29, 2023 @08:53PM (#38468271)
Yet another jart tour-de-force. I knew I had to sponsor you on Github back when I read your magnificent technical breakdown of APE, lol.

(sorry for OT!)

Re:Llamafile lets you distribute and run LLMs with a single ... by jart on Thursday November 30, 2023 @08:32PM (#38481831)
You're awesome!
Re:Llamafile lets you distribute and run LLMs with a single ... by phh on Wednesday November 29, 2023 @06:26PM (#38466827)
Last thing you need is to chmod +x the interpreter: chmod +x /usr/bin/ape (it is indeed not in the README)
Re:Llamafile lets you distribute and run LLMs with a single ... by lol768 on Sunday December 03, 2023 @08:47AM (#38507029)
This was it, wget creates the file and it's non-executable (and I'd already double checked the actual llamafile was executable, but had missed this) - thanks!
Re:Llamafile lets you distribute and run LLMs with a single ... by stavros on Wednesday November 29, 2023 @08:27PM (#38467999)
I get the same error, and there's no `ape` file to make excecutable, hm.
Re:Llamafile lets you distribute and run LLMs with a single ... by jart on Wednesday November 29, 2023 @08:30PM (#38468036)
You can manually download the `ape` command from https://cosmo.zip/pub/cosmos/bin/ Please see the Gotchas section of the README for the copy/pastable commands you can run: https://github.com/mozilla-Ocho/llamafile#gotchas