News for hackers. Stuff that matters.

A comment on I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros. Parent comment

Re:I made an app that runs Mistral 7B 0.2 LLM locally on iPh... by YetAnotherNick on Sunday January 07, 2024 @08:30PM (#38907210)
Mistral instruct doesn't have system prompt AFAIK. Also llama chat system prompt is very useless in my testing.
[ Parent | on HN ]
Re:I made an app that runs Mistral 7B 0.2 LLM locally on iPh... by nl on Sunday January 07, 2024 @08:51PM (#38907361)
Mistral Instruct does use a system prompt.

You can see the raw format here: https://www.promptingguide.ai/models/mistral-7b#chat-templat... and you can see how LllamaIndex uses it here (as an example): https://github.com/run-llama/llama_index/blob/1d861a9440cdc9...

[ Parent | on HN ]
Re:I made an app that runs Mistral 7B 0.2 LLM locally on iPh... by sp332 on Sunday January 07, 2024 @09:40PM (#38907647)
So the system prompt is just part of the first prompt in a conversation? How is that different from not having a system prompt?
[ Parent | on HN ]
Re:I made an app that runs Mistral 7B 0.2 LLM locally on iPh... by nl on Sunday January 07, 2024 @11:55PM (#38908434)
No, system prompts are surrounded by [INST] [/INST]
[ Parent | on HN ]
Re:I made an app that runs Mistral 7B 0.2 LLM locally on iPh... by sp332 on Monday January 08, 2024 @12:36AM (#38908685)
No, every user input is surrounded by those tags. Scroll down from that link you posted and read the next two example prompts.
[ Parent | on HN ]
Re:I made an app that runs Mistral 7B 0.2 LLM locally on iPh... by nl on Monday January 08, 2024 @04:49AM (#38909947)
No this isn't the case.

Here's the example from https://www.promptingguide.ai/models/mistral-7b#chat-templat...

<s>[INST] What is your favorite condiment? [/INST] "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"</s> [INST] The right amount of what? [/INST]

Note that the sentence starting "Well, I'm quite partial isn't inside the tag.

This example comes from the official system card created by MistralAI available here: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1

You can try it:

   ollama run Mistral "<s>[INST] What is your favorite condiment? [/INST] Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen</s> [INST] The right amount of what? [/INST]"
[ Parent | on HN ]
Re:I made an app that runs Mistral 7B 0.2 LLM locally on iPh... by sp332 on Monday January 08, 2024 @08:55AM (#38911924)
That's the whole context with two user inputs in the INST tags, and one assistant output between/outside of the tags. They're just simulating the beginning of a conversation. You can see this very clearly in the JSON version in the next code block:

  messages = [
    {"role": "user", "content": "What is your favourite condiment?"},
    {"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
    {"role": "user", "content": "Do you have mayonnaise recipes?"}
  ]
[ Parent | on HN ]
Re:I made an app that runs Mistral 7B 0.2 LLM locally on iPh... by nl on Monday January 08, 2024 @04:47PM (#38918657)
Yes, but this whole block of text gets passed to the LLM on each call as the conversation history. The [INST] tags tell the LLM which parts were inputs (system instructions) as opposed to outputs.
[ Parent | on HN ]
Re:I made an app that runs Mistral 7B 0.2 LLM locally on iPh... by sp332 on Monday January 08, 2024 @05:43PM (#38919406)
Well, that's not what a system prompt is. There is a difference between system prompts and regular conversational inputs. https://platform.openai.com/docs/guides/prompt-engineering/t...
[ Parent | on HN ]
Re:I made an app that runs Mistral 7B 0.2 LLM locally on iPh... by nl on Monday January 08, 2024 @08:09PM (#38920928)
In actual fact system prompts are just the instruction tuning prompts that provide instructions formatted in the correct way (ie, with [INST]). It's just text all the way down.
[ Parent | on HN ]
Re:I made an app that runs Mistral 7B 0.2 LLM locally on iPh... by YetAnotherNick on Monday January 08, 2024 @06:57AM (#38910761)
Look into their official page: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1

There is no system prompt. Unless Llamaindex or some other sources cite something from mistral, I am inclined to believe they just copied it from llama.

[ Parent | on HN ]
Re:I made an app that runs Mistral 7B 0.2 LLM locally on iPh... by rrr_oh_man on Sunday January 07, 2024 @08:35PM (#38907245)
It seems to work well on GPT4All (macOS) with system prompts?

Can you link to any doc why it shouldn't work?

[ Parent | on HN ]
Re:I made an app that runs Mistral 7B 0.2 LLM locally on iPh... by refulgentis on Sunday January 07, 2024 @08:48PM (#38907332)
It does; and if it's LLaMa 2 7B Chat stock from Facebook, that was a little rushed imho, doesn't seem as baked in.

(GPTs matters but it's _very_ bizarre who it thinks it's coming from)

[ Parent | on HN ]