Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva...
by minimaxir on Tuesday May 28, 2024 @03:17PM
(#40504433)
LMSys is not a perfect benchmark, but it's much better than the more academic benchmarks which do not reflect real-world use cases and may or may not be trained upon by the LLM.
[ Parent
| on HN ]
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva...
by scrollop on Tuesday May 28, 2024 @03:36PM
(#40504674)
IMO Lmsys benchamarks are essentially:
How AI nerds (like us, so a particular subclass of the "normal" population) rate responses from various AIs - this rating is subjective, of course, and an LLM that produces answers that read well though is less accurate/"intelligent" would likely perform better than a model that propduces answers that read worse, though is more "intelligent".
AI nerds/IT nerds likely have higher prevalence of ASD/Asperger's etc (or higher up the spectrum), so likely not the best way to rank LLMs.
It's fun, though!