Mistral 7B, the most powerful language model for its size to date, Apache 2.0

Re:Mistral 7B, the most powerful language model for its size... by londons_explore on Wednesday September 27, 2023 @04:19PM (#37680688)

One solution is to come up with a new benchmark yourself.

Manually benchmarking it by coming up with 20 questions and feeding it to a pair of models and blindly choosing the best result can give you a pretty good figure.

And that can probably be done in under 20 mins of human time.

[ Parent | on HN ]

Re:Mistral 7B, the most powerful language model for its size... by poser-boy on Wednesday September 27, 2023 @09:18PM (#37683953)

This is frankly the best benchmark you can use on an LLM.

[ Parent | on HN ]

Re:Mistral 7B, the most powerful language model for its size... by avereveard on Thursday September 28, 2023 @01:49AM (#37685731)

You can ask gpt4 or other high value model to rate two chat logs for coherency etc, not as accurate as human evaluation, but you don't have to read thousand lines of text if comparing many models.

[ Parent | on HN ]

Re:Mistral 7B, the most powerful language model for its size... by brucethemoose2 on Thursday September 28, 2023 @10:22AM (#37690001)

This is problematic if you are comparing a model in the same base family as the evaluator, as it will probably favor itself because it literally has the sequences it would naturally emit.

[ Parent | on HN ]

Re:Mistral 7B, the most powerful language model for its size... by Breza on Monday October 09, 2023 @07:26PM (#37826749)

Especially since most devs have a specific use case in mind. Coming up with tests that are tailored to your needs will always be more informative than off the shelf metrics.

[ Parent | on HN ]