Localllm lets you develop gen AI apps on local CPUs

Re:Localllm lets you develop gen AI apps on local CPUs by gbickford on Thursday February 08, 2024 @02:28AM (#39299309)

Llama.cpp is an inference engine. The author of llama.cpp designed gguf. Funcionary is a model that does function calling. You can download functionary weights in the gguf format and then run it using llama.cpp on low-end machines using CPU or GPU or a mix of both.

[ Parent | on HN ]

Re:Localllm lets you develop gen AI apps on local CPUs by thund on Thursday February 08, 2024 @04:09AM (#39299891)

ok so the answer is no, thanks

[ Parent | on HN ]

Re:Localllm lets you develop gen AI apps on local CPUs by behnamoh on Thursday February 08, 2024 @11:23AM (#39303847)

The GP educated you and you still said the answer was "no"? It's clearly "yes".

[ Parent | on HN ]

Re:Localllm lets you develop gen AI apps on local CPUs by TeMPOraL on Friday February 09, 2024 @12:58AM (#39311855)

I read the original question as "is the 'Functionary' thing something that works with any LLM you can run in llama.cpp, not just only with the special 'Functionary' finetuned models?", to which the answer now is clearly no.

Very interesting question, as I also read top-level post as saying it's a llama-cpp-python feature, and therefore I could shove some random dolphin-mixtral or whatever I have on my hard drive at it, and it will work.

[ Parent | on HN ]