https://github.com/ggerganov/llama.cpp/blob/master/grammars/...
Its actually better than a specialized model, during token generation it constrains the possible output tokens to an arbitrary grammar (like, say, JSON syntax). So it will work "perfectly" with any model with a basic understanding of the format.
Kobold.cpp and text-generation-ui already support this, and both will run on your mac.
Though its possible it would output syntactically correct nonsense.
Generation never actually stops, the model just emits a special stop token when stopping is most likely next token. Hence the grammar implementation can prevent this stop token from being emitted prematurely.
There was some discussion of models getting "stuck" where there is no syntactically correct token to emit. Some proposals included a "backspace token" IIRC, but I dunno what they actually did. You can look through the discussion in the PR.
Would be interesting to see how far it would get writing programs, and wrt. the problem of stopping too early on a token, it could resume with some extra context from where it stopped.
Maybe construct program such that it is made by building blocks which can fit into context, such that each block/function has a pre/post condition and invariants are inside it, which the LLM will try to test against each time it runs. I think I just found my next side-project, I know there are many similar tools, but I haven't seen anything trying to couple the compiler, running the program, creating pre/post and invariant checking against emitted code yet.
Would be interesting to test this hypothesis to see if the llm can actually build a program this way. I think about it like humans have a short-term memory as well, and for each sub-structure of a program, the llm would have to work in a similar way. Then reserve a bit of context to control the long-term memory for general goals, or create something like an AST of thoughts which it would recurse through, while reasoning about it.
This is a wrapper around many training methods, and it has yielded many excellent community finetunes already.
total duration: 33.964492834s
load duration: 1.471584ms
prompt eval count: 8 token(s)
prompt eval duration: 385.334ms
prompt eval rate: 20.76 tokens/s
eval count: 418 token(s)
eval duration: 31.887033s
eval rate: 13.11 tokens/s
https://github.com/mistralai/mistral-src/issues/2
edit: nevermind https://ollama.ai
If I wanted to experiment with working with models and finetuning offline should I just save for a 32g vram+ video card?
NotADirectoryError: [Errno 20] Not a directory: 'hipconfig'
Our approach also extends to EBNF grammars and LALR parsing. There's an example of that in the repository. It builds off of the Lark library, so you can use existing grammar specifications instead of starting from scratch.
Question: is there something like this, but with the "function calling api" finetuning? 95% of my uses nowadays deal with input/output of structured data (JSON, basically), and I kind of fear that OpenAI will no longer support thos specialized models after a few months... I know its probably not that glorious compared with multimodal chat UIs, but a workhose like nothing else for automation!