News for hackers. Stuff that matters.

A comment on Llamafile lets you distribute and run LLMs with a single file. Parent comment

Re:Llamafile lets you distribute and run LLMs with a single ... by FragenAntworten on Wednesday November 29, 2023 @11:34PM (#38469683)
It doesn't seem to run on NixOS, though I'm new to Nix and may be missing something.

    $ ./llava-v1.5-7b-q4-server.llamafile --help
    ./llava-v1.5-7b-q4-server.llamafile: line 60: /bin/mkdir: No such file or directory
Regardless, this (and Cosmopolitan) are amazing work - thank you!
[ Parent | on HN ]
Re:Llamafile lets you distribute and run LLMs with a single ... by jart on Wednesday November 29, 2023 @11:43PM (#38469749)
The APE shell script needs to run /bin/mkdir in order to map the embedded ELF executable in memory. It should be possible for you to work around this on Linux by installing our binfmt_misc interpreter:

    sudo wget -O /usr/bin/ape https://cosmo.zip/pub/cosmos/bin/ape-$(uname -m).elf
    sudo sh -c "echo ':APE:M::MZqFpD::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register"
    sudo sh -c "echo ':APE-jart:M::jartsr::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register"
That way the only file you'll need to whitelist with Nix is /usr/bin/ape. You could also try just vendoring the 8kb ape executable in your Nix project, and simply executing `./ape ./llamafile`.
[ Parent | on HN ]
Re:Llamafile lets you distribute and run LLMs with a single ... by FragenAntworten on Wednesday November 29, 2023 @11:58PM (#38469844)
`./ape ./llamafile` worked immediately and without problems I can see - thank you!
[ Parent | on HN ]