News for hackers. Stuff that matters.
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva... by doctorpangloss on Tuesday May 28, 2024 @03:03PM (#40504274)
I don't think LMSys is the end-all be-all of benchmarking

We are only talking about it because people's "vibes" are that GPT-4o is "the best."

If a contrary result were found, the social media public would gravitate to something else written in the aesthetics of a primitive Gradio interface with some academic affiliation that told them the story they wanted to believe.

It doesn't really make sense, one way or another, than totally different models should perform so similarly, or even so differently, from the specific LMSys ranking task of "edify the end user." There's no explanation or analysis for the result. But because we "feel" "like" Anthropic's "big model" and Google's "big model" should perform the same as "OpenAI's big model," but because we also feel that "OpenAI is the best so it's latest model is also should be the best," well the results could say anything in any benchmarking approach, but we talk about LMSys's because they agree with our feelings, and we don't talk about other results because they do not.

If someone had a good way to benchmark LLMs, they would instead author training data.

Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva... by jsnell on Tuesday May 28, 2024 @03:27PM (#40504570)
I've been seeing links on HN to Chatbot Arena results pretty regularly after any kind of model releases or updates for a long time, at least six months, probably nine. That's way before GPT-4o was released, so it seems unlikely your theory is correct.
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva... by minimaxir on Tuesday May 28, 2024 @03:17PM (#40504433)
LMSys is not a perfect benchmark, but it's much better than the more academic benchmarks which do not reflect real-world use cases and may or may not be trained upon by the LLM.
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva... by scrollop on Tuesday May 28, 2024 @03:36PM (#40504674)
IMO Lmsys benchamarks are essentially:

How AI nerds (like us, so a particular subclass of the "normal" population) rate responses from various AIs - this rating is subjective, of course, and an LLM that produces answers that read well though is less accurate/"intelligent" would likely perform better than a model that propduces answers that read worse, though is more "intelligent".

AI nerds/IT nerds likely have higher prevalence of ASD/Asperger's etc (or higher up the spectrum), so likely not the best way to rank LLMs.

It's fun, though!

Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva... by simonw on Tuesday May 28, 2024 @03:11PM (#40504370)
"[...] we talk about LMSys's because they agree with our feelings, and we don't talk about other results because they do not."

What other benchmarks do you think are being overlooked?

Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva... by doctorpangloss on Tuesday May 28, 2024 @03:24PM (#40504530)
No idea. Better to ask the people training models what they use internally, not me. A good benchmark itself could be worth millions.

LMSys has the same energy as Amazon reviews of brand name products. It's measuring something, and that something might be valuable. I don't know if it is valuable for answering subjective, spicy questions like, "What LLM is likely to replace most or all of your job right now, giving you some time to arbitrage 1h of chatbot with a full time employment?" which is what everyone actually cares about. That is what this is all about. I can't tell you what benchmark is answering that question or others. I don't know!

Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva... by simonw on Tuesday May 28, 2024 @03:30PM (#40504615)
I've been using LMSYS as the best available high level "these are models to pay the most attention to" benchmark - I've not seen anything else that's nearly as useful on that front.

The other one I consult occasionally is https://gorilla.cs.berkeley.edu/leaderboard.html - the Berkeley Function-Calling Leaderboard - because function calling is a really useful characteristic of smaller models.

I believe the best ways to evaluate these models is with your own set of task-specific evals for your own project, but these are REALLY difficult to construct. I still don't have a good solution there for my own work.

Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva... by jerpint on Tuesday May 28, 2024 @04:01PM (#40504940)
Completely agree, and it’s been great to see to some extent that the gap between open source and 3rd party models is really starting to close
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva... by FergusArgyll on Tuesday May 28, 2024 @05:25PM (#40505779)
Someone made a framework to make and test llms on your own benchmarks

you might find it useful

https://github.com/carlini/yet-another-applied-llm-benchmark

Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva... by bravura on Tuesday May 28, 2024 @03:59PM (#40504921)
Predicting the future.

I recently saw a paper getting LLM evaluation completely right in many respects: https://arxiv.org/abs//2402.00861

"Evaluating Large Language Models for Generalization and Robustness via Data Compression"

They evaluate LLMs by seeing how well they compress text/news/science from the future, over time. In a sense, this measures how "surprised" the LLM is by future developments.

This is, I believe, an important and strong task. A top researcher or economist will have a preternatural ability to "see where the puck is going" in a way that would be more surprising (perplexing) to a junior person or a layperson. Thus, things after the knowledge cutoff being more "expected" is a great sign of predictive generality.

Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva... by mejutoco on Tuesday May 28, 2024 @05:25PM (#40505783)
Wouldn t this get you survivorship bias but with models instead of investors?