https://github.com/Mozilla-Ocho/llamafile
You can download any .gguf model (not just the ones in their examples) and run it locally (as long as you have the ram for it). I was running 7B models with ease on an old FX8350 and now 13B models on a 5600X (32GB RAM on both machines).
This wrapper spins up a local web server that runs a simple web frontend to use immediately with no code, but also exposes an OpenAI compatible API for dev work and alt frontends (like SillyTavern).
I see llama.cpp in there, but I didn't go deeper than that. Ollama is not the same thing.
Very interesting question, as I also read top-level post as saying it's a llama-cpp-python feature, and therefore I could shove some random dolphin-mixtral or whatever I have on my hard drive at it, and it will work.
Functionary is a series of function calling llama architecture finetuned models, not a new base model, if thats what you mean.
Hence, it can be turned into a gguf like any other llama finetune.
It just has a very esoteric prompting format, but llama-cpp-python is specifically set up to leverage it.
> TBH I would just recommend using that and all its under-the-radar features (like grammar, and built in function calling support with Functionary).
I never heard about Functionary, this comment seems to say “I would use llama.cpp plus Functionary”. Now, since llama.cpp allows to plug in several models, I was wondering if Functionary works with several models too, ie is Functionary a library that allows to add function calling to models?
The following answer to my comment helped clarifying that Functionary is not a lib, but a specific model, or that is what I understand from the answer. So the answer is “no, you can’t use functionary with many models (eg it’s not like llama.cpp that works with multiple models) because Functionary is a model itself”
:-)
Technically a wrapper of a wrapper, though llama-cpp-python is quite excellent... TBH I would just recommend using that and all its under-the-radar features (like grammar, and built in function calling support with Functionary).