News for hackers. Stuff that matters.
Re:MistralLite by Amazon Web Services by simonw on Wednesday November 01, 2023 @01:01PM (#38101240)
Can anyone explain the prompting format for this?

    <|prompter|>{prompt}</s><|assistant|>
Why is that </s> in there?

And how would I prompt it with a previous history of the conversation?

Re:MistralLite by Amazon Web Services by brucethemoose2 on Wednesday November 01, 2023 @01:36PM (#38101791)
</s> is the "end" token, its common in the instruct syntax for many LLMs.

<|prompter|> is the human message (including the instructon/contexts from my tests) and <|assistant|> is the bot's response.

There are llm wrappers that will save chat histories and reformat them appropriately for different LLMs. For instance, see TUI and its instruction template presets: https://github.com/oobabooga/text-generation-webui/tree/main...

Re:MistralLite by Amazon Web Services by simonw on Wednesday November 01, 2023 @01:51PM (#38102070)
Aaah I see - so it's actually more like:

    <|prompter|>... prompt goes here</s>
    <|assistant|> ... assistant response will go here, maybe followed by </s>
Still don't understand why I got a rogue </s> in the reply when I ran a prompt through it (but it continued after that tag): https://news.ycombinator.com/item?id=38101766

And is my guess about </s> at the end of the assistant thing correct? If so then a conversation might look like this:

    <|prompter|>hi</s>
    <|assistant|>Hi to you too. What do you want to talk about?</s>
    <|prompter|>Tell me facts about snakes</s>
    <|assistant|>
But the documentation on these models is so thin! Do I need that </s> at the end of the assistant things or not?
Re:MistralLite by Amazon Web Services by brucethemoose2 on Wednesday November 01, 2023 @02:09PM (#38102348)
> But the documentation on these models is so thin!

Welcome to LLM research land, lol. Mpstly there is no documentation, no reproducability, just random hit-and-run releases with some claimed metrics and if you're lucky a model card with the trained prompt.

> Do I need that </s> at the end of the assistant things or not?

...I dunno. I didn't add it in my prompt template, and you can see that the model itself it outputting that token in the responses. Maybe ask the trainers on the HF page.

Re:MistralLite by Amazon Web Services by simonw on Wednesday November 01, 2023 @01:34PM (#38101766)
I tried running a prompt through it just now:

    <|prompter|>hello</s><|assistant|>
And got back this (truncated):

    hey there!</s>
    introducing the first of its kind, a revolutionary AI...
Why did that </s> come back as part of the response?
Re:MistralLite by Amazon Web Services by 65a on Wednesday November 01, 2023 @02:58PM (#38103189)
You would want '</s>' as a stop token.