Llama.cpp guide – Running LLMs locally on any hardware, from scratch

Re:Llama.cpp guide – Running LLMs locally on any hardware, f... by yjftsjthsd-h on Friday November 29, 2024 @03:32PM (#42276869)

You might also try https://github.com/Mozilla-Ocho/llamafile , which may have better CPU-only performance than ollama. It does require you to grab .gguf files yourself (unless you use one of their prebuilts in which case it comes with the binary!), but with that done it's really easy to use and has decent performance.

For reference, this is how I run it:

  $ cat ~/.config/systemd/user/[email protected]
  [Unit]
  Description=llamafile with arbitrary model
  After=network.target
  
  [Service]
  Type=simple
  WorkingDirectory=%h/llms/
  ExecStart=sh -c "%h/.local/bin/llamafile -m %h/llamafile-models/%i.gguf --server --host '::' --port 8081 --nobrowser --log-disable"
  
  [Install]
  WantedBy=default.target

And then

  systemctl --user start llamafile@whatevermodel

but you can just run that ExecStart command directly and it works.

[ Parent | on HN ]

Re:Llama.cpp guide – Running LLMs locally on any hardware, f... by SahAssar on Friday November 29, 2024 @05:02PM (#42277515)

Is that `--host` listening on non-local addresses? Might be good to default to local-only.

[ Parent | on HN ]

Re:Llama.cpp guide – Running LLMs locally on any hardware, f... by yjftsjthsd-h on Saturday November 30, 2024 @12:11AM (#42279658)

Good call out; in my context yes I do want it listening for use by other machines in its subnets and deliberately set that option (including using the IPv6 form), but most people are probably better off binding to loopback. Thanks

[ Parent | on HN ]

Re:Llama.cpp guide – Running LLMs locally on any hardware, f... by chatmasta on Friday November 29, 2024 @05:30PM (#42277697)

Be careful running this on work machines – it will get flagged by Crowdstrike Falcon and probably other EDR tools. In my case the first time I tried it, I just saw “Killed” and then got a DM from SecOps within two minutes.

[ Parent | on HN ]

Re:Llama.cpp guide – Running LLMs locally on any hardware, f... by broknbottle on Friday November 29, 2024 @11:34PM (#42279459)

the irony, preventing and killing something that is actually useful, while we let crowdcrap hum along consuming tons of memory and bottlenecking IO so it can do snakeoil things...

[ Parent | on HN ]

Re:Llama.cpp guide – Running LLMs locally on any hardware, f... by yjftsjthsd-h on Saturday November 30, 2024 @12:13AM (#42279670)

Are they specifically flagging LLMs, or do they not like Cosmopolitan Libc / APE?

[ Parent | on HN ]

Re:Llama.cpp guide – Running LLMs locally on any hardware, f... by chatmasta on Saturday November 30, 2024 @12:59AM (#42279875)

Nah nothing to do with LLMs, it’s just because the method of Llamafile is very similar to malware - basically zip up an executable, concatenate it with some stuff, throw it in /tmp and execute it with a randomly generated high entropy name.

(That said, after I explained it to SecOps they did tell me I would need to “consult legal” if I wanted to use a local LLM, but I’ll give them the benefit of the doubt there…)

[ Parent | on HN ]