News for hackers. Stuff that matters.
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva... by simonw on Tuesday May 28, 2024 @03:30PM (#40504615)
I've been using LMSYS as the best available high level "these are models to pay the most attention to" benchmark - I've not seen anything else that's nearly as useful on that front.

The other one I consult occasionally is https://gorilla.cs.berkeley.edu/leaderboard.html - the Berkeley Function-Calling Leaderboard - because function calling is a really useful characteristic of smaller models.

I believe the best ways to evaluate these models is with your own set of task-specific evals for your own project, but these are REALLY difficult to construct. I still don't have a good solution there for my own work.

Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva... by jerpint on Tuesday May 28, 2024 @04:01PM (#40504940)
Completely agree, and it’s been great to see to some extent that the gap between open source and 3rd party models is really starting to close
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva... by FergusArgyll on Tuesday May 28, 2024 @05:25PM (#40505779)
Someone made a framework to make and test llms on your own benchmarks

you might find it useful

https://github.com/carlini/yet-another-applied-llm-benchmark