News for hackers. Stuff that matters.
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva... by minimaxir on Tuesday May 28, 2024 @03:12PM (#40504389)
The performance of Gemini 1.5 Flash is the real takeaway. It's priced roughly similar to Claude Haiku, but in this leaderboard it's performing better than Claude Sonnet.

I may have to run tests to see if Flash is really that much better than Haiku. From the win rate heat map (https://x.com/lmsysorg/status/1795512212456645074) Flash has a 62% win rate vs. Haiku.

Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva... by YetAnotherNick on Tuesday May 28, 2024 @03:28PM (#40504588)
Better comparison would be llama 3 70b whose pricing is $0.59/$0.79[1]. Gemini Flash pricing is $0.35/$1.05. For similar quality and price I would select open model, if only for the certainty that the model would perform exactly the same and can't be tweaked/deleted without warning.

[1]: https://wow.groq.com/ [2]: https://ai.google.dev/pricing

Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva... by harlanlewis on Tuesday May 28, 2024 @04:14PM (#40505078)
I found myself making these types of comparisons over and over again to keep up with new models and pricing changes, so I made a Google Sheet for it with some charts to visualize outliers and trends in terms of model capability, throughput, and token cost. It's sorted by the Artificial Analysis Index [1], which incorporates both static benchmarks (MMLU) and dynamic crowdsourced elo (chatbot arena). Hope someone finds it useful:

https://docs.google.com/spreadsheets/d/1foc98Jtbi0-GUsNySddv...

Specific to Groq - the service is an incredible outlier for speed and cost, but even for quite small projects I've run into rate limits and similar errors. Great option for something like a bring-your-own-keys personal chatbot, but I wouldn't build a service on it. Groq Llama 3 70B and Groq Mixtral 8x7B are in the sheet for comparison but omitted from charts.

[1]: https://artificialanalysis.ai/leaderboards/providers

Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva... by nycdatasci on Tuesday May 28, 2024 @04:56PM (#40505513)
This is great - thanks!
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva... by dilap on Tuesday May 28, 2024 @04:14PM (#40505073)
groq is super fast and cheap, but it seems to have (for normal plans) a very-low 30 requests/minute rate limit; to get a higher-limit, it's "contact us" enterprise pricing (which I haven't done, so I'm not sure what the results will be).

edit: ah, currently it's "free beta access with low rate limits" and "enterprise" that is available, but they are planning to release "on demand pay per token" with higher rate limits soon; exact pricing unknown.

https://pasteboard.co/OnLHjusiv6QT.png

another limitation of groq, which may or may not matter for your use case, is there's no way to pre-fill part of the llm response, which can be a very useful technique (there's also no support for the more general feature of "restrict sampling to a grammar", tho that is a much bigger ask).