Comparing 60 LLMs with a set of 20 prompts

Re:Comparing 60 LLMs with a set of 20 prompts by YetAnotherNick on Saturday September 09, 2023 @11:37AM (#37446621)

I think they messed up the templates.

[ Parent | on HN ]

Re:Comparing 60 LLMs with a set of 20 prompts by detente18 on Saturday September 09, 2023 @11:44AM (#37446684)

Here's the template I'm using - https://github.com/BerriAI/litellm/blob/5ca8b23e22139a4f49bd...

Anything I'm doing incorrectly?

[ Parent | on HN ]

Re:Comparing 60 LLMs with a set of 20 prompts by YetAnotherNick on Saturday September 09, 2023 @09:48PM (#37452148)

The format is really weird for chat models. In your code post message is `<</SYS>>\n [/INST]` which is wrong. Instead the overall format is `[INST] <<SYS>>\n{system}\n<</SYS>>\n\n{instruction} [/INST]`

So both the system message and first instruction are in same [INST] block.

[ Parent | on HN ]