News for hackers. Stuff that matters.
Re:MistralLite by Amazon Web Services by ReactiveJelly on Wednesday November 01, 2023 @12:19PM (#38100503)
So... this won't work with llama.cpp right? I need a different runtime for it? I'm new to this LLM stuff.
Re:MistralLite by Amazon Web Services by brucethemoose2 on Wednesday November 01, 2023 @12:35PM (#38100802)
Yeah it works fine. Mistral is a llama architecture llm.

There are quantizations already. I would suggest running it on KoboldCPP, or any of the UIs recommended on this page: https://huggingface.co/TheBloke/MistralLite-7B-GGUF

Personally, I run the elx2 quantization with exllamav2 because its faster and more coherent in the same amount of vram. You can do that with Aphrodite or Text-gen-ui: https://huggingface.co/models?search=MistralLite%20exl2

Re:MistralLite by Amazon Web Services by hfjjbf on Wednesday November 01, 2023 @12:48PM (#38101028)
This account has prepackaged and quantized models for everything, really awesome.

https://huggingface.co/TheBloke

Re:MistralLite by Amazon Web Services by brucethemoose2 on Wednesday November 01, 2023 @01:00PM (#38101218)
Indeed.

But TheBloke doesn't (yet) do exl2 quantization. That one is almost better done yourself, as the bpw target is arbitrary, and requires profiling text that should probably fit your intended use (though many just use wikitext as a good default).

Re:MistralLite by Amazon Web Services by simonw on Wednesday November 01, 2023 @01:01PM (#38101240)
Can anyone explain the prompting format for this?

    <|prompter|>{prompt}</s><|assistant|>
Why is that </s> in there?

And how would I prompt it with a previous history of the conversation?

Re:MistralLite by Amazon Web Services by brucethemoose2 on Wednesday November 01, 2023 @01:36PM (#38101791)
</s> is the "end" token, its common in the instruct syntax for many LLMs.

<|prompter|> is the human message (including the instructon/contexts from my tests) and <|assistant|> is the bot's response.

There are llm wrappers that will save chat histories and reformat them appropriately for different LLMs. For instance, see TUI and its instruction template presets: https://github.com/oobabooga/text-generation-webui/tree/main...

Re:MistralLite by Amazon Web Services by simonw on Wednesday November 01, 2023 @01:51PM (#38102070)
Aaah I see - so it's actually more like:

    <|prompter|>... prompt goes here</s>
    <|assistant|> ... assistant response will go here, maybe followed by </s>
Still don't understand why I got a rogue </s> in the reply when I ran a prompt through it (but it continued after that tag): https://news.ycombinator.com/item?id=38101766

And is my guess about </s> at the end of the assistant thing correct? If so then a conversation might look like this:

    <|prompter|>hi</s>
    <|assistant|>Hi to you too. What do you want to talk about?</s>
    <|prompter|>Tell me facts about snakes</s>
    <|assistant|>
But the documentation on these models is so thin! Do I need that </s> at the end of the assistant things or not?
Re:MistralLite by Amazon Web Services by brucethemoose2 on Wednesday November 01, 2023 @02:09PM (#38102348)
> But the documentation on these models is so thin!

Welcome to LLM research land, lol. Mpstly there is no documentation, no reproducability, just random hit-and-run releases with some claimed metrics and if you're lucky a model card with the trained prompt.

> Do I need that </s> at the end of the assistant things or not?

...I dunno. I didn't add it in my prompt template, and you can see that the model itself it outputting that token in the responses. Maybe ask the trainers on the HF page.

Re:MistralLite by Amazon Web Services by simonw on Wednesday November 01, 2023 @01:34PM (#38101766)
I tried running a prompt through it just now:

    <|prompter|>hello</s><|assistant|>
And got back this (truncated):

    hey there!</s>
    introducing the first of its kind, a revolutionary AI...
Why did that </s> come back as part of the response?
Re:MistralLite by Amazon Web Services by 65a on Wednesday November 01, 2023 @02:58PM (#38103189)
You would want '</s>' as a stop token.