Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva...
by huevosabio on Tuesday May 28, 2024 @03:51PM
(#40504855)
It wouldn't be hard to do a basic version either.
Just check run the output of both outputs through the model and pick the one with highest likelihood according to the model you are trying to promote (since arguably you're trying to promote a model you have access to).
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva...
by minimaxir on Tuesday May 28, 2024 @03:46PM
(#40504800)
That's why the sample size is important, since a large sample size would offset a single bad actor.
LMSys does have some anti-bot measures (e.g. Cloudflare) which makes it infeasible to manipulate the leaderboard for marketing.
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva...
by Tiberium on Tuesday May 28, 2024 @06:34PM
(#40506320)
LMSys' Cloudflare protection is sadly trivial to bypass - just search for "LMSYS Chatbot Arena: Benchmarking LLMs in the Wild" (with quotes) in Censys and you'll find the origin host. It's a common misconfiguration for services that use Cloudflare as a proxy but then forget to actually make the origin server only respond to requests originating from CF IPs. Although I don't know if anyone's actually botting it.
IF it does this hundreds of time, it could alter the results.
Do AI companies care enough to do this vs the risks of discovery? Might be a fun project, and I can imagine that the people who make the decisions on which AI to use for their project/company, may look at this leaderboard.