Many options for running Mistral models in your terminal using LLM

Re:Many options for running Mistral models in your terminal ... by vunderba on Tuesday December 19, 2023 @02:41AM (#38692929)

I want to like Ollama, but I wish it didn't obfuscate the actual directives (full prompt) that it sends to the underlying model. Ollama uses a custom templatizing script in its Modelfiles to translate user input into the format that a specific model expects ([INST], etc), but it can be difficult to tell if it's working as expected because it won't show up in the logs at all.

Other than that it's a great project - very easy to get started and has a solid API implementation. I've got it running on both a Win 10 + WSL2 docker and on a Mac M1.

[ Parent | on HN ]

Re:Many options for running Mistral models in your terminal ... by coffeeri on Tuesday December 19, 2023 @05:18AM (#38693894)

You can bypass the templating in raw mode, by setting the request parameter `raw` to true.

https://github.com/jmorganca/ollama/blob/main/docs%2Fapi.md

[ Parent | on HN ]

Re:Many options for running Mistral models in your terminal ... by vunderba on Tuesday December 19, 2023 @12:58PM (#38699031)

yeah I guess I could compare the output at 0.0 temperature with same prompt using Modelfile and then after using raw mode with my best guess of how the modelfile is creating the raw data it's passing to Ollama.

I'd push a PR to the repo itself but I have zero experience with Go...

[ Parent | on HN ]