Llama.cpp guide – Running LLMs locally on any hardware, from scratch

Re:Llama.cpp guide – Running LLMs locally on any hardware, f... by smcleod on Friday November 29, 2024 @04:06PM (#42277109)

Neat to see more folks writing blogs on their experiences. This however does seem like it's an over-complicated method of building llama.cpp.

Assuming you want to do this iteratively (at least for the first time) should only need to run:

  ccmake .

And toggle the parameters your hardware supports or that you want (e.g. if CUDA if you're using Nvidia, Metal if you're using Apple etc..), and press 'c' (configure) then 'g' (generate), then:

  cmake --build . -j $(expr $(nproc) / 2)

Done.

If you want to move the binaries into your PATH, you could then optionally run cmake install.

[ Parent | on HN ]

Re:Llama.cpp guide – Running LLMs locally on any hardware, f... by SteelPh0enix on Friday November 29, 2024 @06:55PM (#42278214)

Wow, i did not know about ccmake. I'll check it out and edit the post if it's really that easy to use, thanks.

[ Parent | on HN ]

Re:Llama.cpp guide – Running LLMs locally on any hardware, f... by moffkalast on Friday November 29, 2024 @07:12PM (#42278327)

Yeah the mingw method on windows is a ludicrous thing to even think about, and llama.cpp still has that as the suggested option in the readme for some weird reason. Endless sourcing of paths that never works quite right. I literally couldn't get it to work when I first tried it last year.

Meanwhile Cmake is like two lines and somehow it's the backup fallback option? I don't get it. And well on linux it's literally just one line with make.

[ Parent | on HN ]

Re:Llama.cpp guide – Running LLMs locally on any hardware, f... by blharr on Friday November 29, 2024 @09:18PM (#42278870)

Building anything on windows without cmake is just... I don't know why anyone would use anything else. I used to spend hours wrestling with build failures, but after setting up cmake, it just works with everything.

[ Parent | on HN ]

Re:Llama.cpp guide – Running LLMs locally on any hardware, f... by SteelPh0enix on Saturday November 30, 2024 @02:25AM (#42280138)

>Yeah the mingw method on windows is a ludicrous thing to even think about

Building most stuff on Windows is ludicrous, that's not something uncommon. I've chosen MSYS there as it's the easiest and least paninful way of installing deps for Vulkan build.

[ Parent | on HN ]

Re:Llama.cpp guide – Running LLMs locally on any hardware, f... by lukev on Friday November 29, 2024 @09:07PM (#42278825)

Actually I think even this makes it look scarier than it is if you're on an M-series Apple.

In that case, the steps to building llama.cpp are:

1. Clone the repo.

2. Run `make`.

To start chatting with a model all you need is to:

1. Download the model you want in gguf format that will fit into your hardware (probably the hardest step, but readily available on HuggingFace)

2. Run `./llama-server -m model.gguf`.

3. Visit localhost:8080

[ Parent | on HN ]

Re:Llama.cpp guide – Running LLMs locally on any hardware, f... by int_19h on Friday November 29, 2024 @11:19PM (#42279386)

On a Mac, if all you want is to just use it directly, it is also readily available from Homebrew.

[ Parent | on HN ]

Re:Llama.cpp guide – Running LLMs locally on any hardware, f... by xyc on Saturday November 30, 2024 @12:30AM (#42279758)

You can get a release binary from https://github.com/ggerganov/llama.cpp/releases too.

[ Parent | on HN ]

Re:Llama.cpp guide – Running LLMs locally on any hardware, f... by freehorse on Saturday November 30, 2024 @03:16AM (#42280321)

Does it autoupdate? I get it from github so I just have to pull and build again every time I want to update it.

[ Parent | on HN ]

Re:Llama.cpp guide – Running LLMs locally on any hardware, f... by smcleod on Saturday November 30, 2024 @01:47AM (#42280041)

Yes, but that's not building it for your system, that's a relatively generic build.

[ Parent | on HN ]