Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva...
by simonw on Tuesday May 28, 2024 @03:30PM
(#40504615)
I've been using LMSYS as the best available high level "these are models to pay the most attention to" benchmark - I've not seen anything else that's nearly as useful on that front.
The other one I consult occasionally is https://gorilla.cs.berkeley.edu/leaderboard.html - the Berkeley Function-Calling Leaderboard - because function calling is a really useful characteristic of smaller models.
I believe the best ways to evaluate these models is with your own set of task-specific evals for your own project, but these are REALLY difficult to construct. I still don't have a good solution there for my own work.
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva...
by jerpint on Tuesday May 28, 2024 @04:01PM
(#40504940)
Completely agree, and itβs been great to see to some extent that the gap between open source and 3rd party models is really starting to close
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva...
by FergusArgyll on Tuesday May 28, 2024 @05:25PM
(#40505779)
Someone made a framework to make and test llms on your own benchmarks
you might find it useful
https://github.com/carlini/yet-another-applied-llm-benchmark
LMSys has the same energy as Amazon reviews of brand name products. It's measuring something, and that something might be valuable. I don't know if it is valuable for answering subjective, spicy questions like, "What LLM is likely to replace most or all of your job right now, giving you some time to arbitrage 1h of chatbot with a full time employment?" which is what everyone actually cares about. That is what this is all about. I can't tell you what benchmark is answering that question or others. I don't know!