News for hackers. Stuff that matters.
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva... by minimaxir on Tuesday May 28, 2024 @03:46PM (#40504800)
That's why the sample size is important, since a large sample size would offset a single bad actor.

LMSys does have some anti-bot measures (e.g. Cloudflare) which makes it infeasible to manipulate the leaderboard for marketing.

Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva... by Tiberium on Tuesday May 28, 2024 @06:34PM (#40506320)
LMSys' Cloudflare protection is sadly trivial to bypass - just search for "LMSYS Chatbot Arena: Benchmarking LLMs in the Wild" (with quotes) in Censys and you'll find the origin host. It's a common misconfiguration for services that use Cloudflare as a proxy but then forget to actually make the origin server only respond to requests originating from CF IPs. Although I don't know if anyone's actually botting it.