Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva...
by minimaxir on Tuesday May 28, 2024 @03:12PM
(#40504389)
The performance of Gemini 1.5 Flash is the real takeaway. It's priced roughly similar to Claude Haiku, but in this leaderboard it's performing better than Claude Sonnet.
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva...
by YetAnotherNick on Tuesday May 28, 2024 @03:28PM
(#40504588)
Better comparison would be llama 3 70b whose pricing is $0.59/$0.79[1]. Gemini Flash pricing is $0.35/$1.05. For similar quality and price I would select open model, if only for the certainty that the model would perform exactly the same and can't be tweaked/deleted without warning.
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva...
by harlanlewis on Tuesday May 28, 2024 @04:14PM
(#40505078)
I found myself making these types of comparisons over and over again to keep up with new models and pricing changes, so I made a Google Sheet for it with some charts to visualize outliers and trends in terms of model capability, throughput, and token cost. It's sorted by the Artificial Analysis Index [1], which incorporates both static benchmarks (MMLU) and dynamic crowdsourced elo (chatbot arena). Hope someone finds it useful:
Specific to Groq - the service is an incredible outlier for speed and cost, but even for quite small projects I've run into rate limits and similar errors. Great option for something like a bring-your-own-keys personal chatbot, but I wouldn't build a service on it. Groq Llama 3 70B and Groq Mixtral 8x7B are in the sheet for comparison but omitted from charts.
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva...
by dilap on Tuesday May 28, 2024 @04:14PM
(#40505073)
groq is super fast and cheap, but it seems to have (for normal plans) a very-low 30 requests/minute rate limit; to get a higher-limit, it's "contact us" enterprise pricing (which I haven't done, so I'm not sure what the results will be).
edit: ah, currently it's "free beta access with low rate limits" and "enterprise" that is available, but they are planning to release "on demand pay per token" with higher rate limits soon; exact pricing unknown.
another limitation of groq, which may or may not matter for your use case, is there's no way to pre-fill part of the llm response, which can be a very useful technique (there's also no support for the more general feature of "restrict sampling to a grammar", tho that is a much bigger ask).
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva...
by doctorpangloss on Tuesday May 28, 2024 @03:03PM
(#40504274)
I don't think LMSys is the end-all be-all of benchmarking
We are only talking about it because people's "vibes" are that GPT-4o is "the best."
If a contrary result were found, the social media public would gravitate to something else written in the aesthetics of a primitive Gradio interface with some academic affiliation that told them the story they wanted to believe.
It doesn't really make sense, one way or another, than totally different models should perform so similarly, or even so differently, from the specific LMSys ranking task of "edify the end user." There's no explanation or analysis for the result. But because we "feel" "like" Anthropic's "big model" and Google's "big model" should perform the same as "OpenAI's big model," but because we also feel that "OpenAI is the best so it's latest model is also should be the best," well the results could say anything in any benchmarking approach, but we talk about LMSys's because they agree with our feelings, and we don't talk about other results because they do not.
If someone had a good way to benchmark LLMs, they would instead author training data.
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva...
by jsnell on Tuesday May 28, 2024 @03:27PM
(#40504570)
I've been seeing links on HN to Chatbot Arena results pretty regularly after any kind of model releases or updates for a long time, at least six months, probably nine. That's way before GPT-4o was released, so it seems unlikely your theory is correct.
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva...
by minimaxir on Tuesday May 28, 2024 @03:17PM
(#40504433)
LMSys is not a perfect benchmark, but it's much better than the more academic benchmarks which do not reflect real-world use cases and may or may not be trained upon by the LLM.
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva...
by scrollop on Tuesday May 28, 2024 @03:36PM
(#40504674)
IMO Lmsys benchamarks are essentially:
How AI nerds (like us, so a particular subclass of the "normal" population) rate responses from various AIs - this rating is subjective, of course, and an LLM that produces answers that read well though is less accurate/"intelligent" would likely perform better than a model that propduces answers that read worse, though is more "intelligent".
AI nerds/IT nerds likely have higher prevalence of ASD/Asperger's etc (or higher up the spectrum), so likely not the best way to rank LLMs.
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva...
by doctorpangloss on Tuesday May 28, 2024 @03:24PM
(#40504530)
No idea. Better to ask the people training models what they use internally, not me. A good benchmark itself could be worth millions.
LMSys has the same energy as Amazon reviews of brand name products. It's measuring something, and that something might be valuable. I don't know if it is valuable for answering subjective, spicy questions like, "What LLM is likely to replace most or all of your job right now, giving you some time to arbitrage 1h of chatbot with a full time employment?" which is what everyone actually cares about. That is what this is all about. I can't tell you what benchmark is answering that question or others. I don't know!
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva...
by simonw on Tuesday May 28, 2024 @03:30PM
(#40504615)
I've been using LMSYS as the best available high level "these are models to pay the most attention to" benchmark - I've not seen anything else that's nearly as useful on that front.
The other one I consult occasionally is https://gorilla.cs.berkeley.edu/leaderboard.html - the Berkeley Function-Calling Leaderboard - because function calling is a really useful characteristic of smaller models.
I believe the best ways to evaluate these models is with your own set of task-specific evals for your own project, but these are REALLY difficult to construct. I still don't have a good solution there for my own work.
"Evaluating Large Language Models for Generalization and Robustness via Data Compression"
They evaluate LLMs by seeing how well they compress text/news/science from the future, over time. In a sense, this measures how "surprised" the LLM is by future developments.
This is, I believe, an important and strong task. A top researcher or economist will have a preternatural ability to "see where the puck is going" in a way that would be more surprising (perplexing) to a junior person or a layperson. Thus, things after the knowledge cutoff being more "expected" is a great sign of predictive generality.
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva...
by scrollop on Tuesday May 28, 2024 @03:40PM
(#40504731)
I wonder if you could create an AI bot that interacts in the arena, and knows the telltale signs of a particular LLM, and when it sees that LLM answer, always votes for it.
IF it does this hundreds of time, it could alter the results.
Do AI companies care enough to do this vs the risks of discovery? Might be a fun project, and I can imagine that the people who make the decisions on which AI to use for their project/company, may look at this leaderboard.
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva...
by huevosabio on Tuesday May 28, 2024 @03:51PM
(#40504855)
It wouldn't be hard to do a basic version either.
Just check run the output of both outputs through the model and pick the one with highest likelihood according to the model you are trying to promote (since arguably you're trying to promote a model you have access to).
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva...
by Tiberium on Tuesday May 28, 2024 @06:34PM
(#40506320)
LMSys' Cloudflare protection is sadly trivial to bypass - just search for "LMSYS Chatbot Arena: Benchmarking LLMs in the Wild" (with quotes) in Censys and you'll find the origin host. It's a common misconfiguration for services that use Cloudflare as a proxy but then forget to actually make the origin server only respond to requests originating from CF IPs. Although I don't know if anyone's actually botting it.
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva...
by ein0p on Tuesday May 28, 2024 @03:28PM
(#40504590)
The only thing this tells me: your metrics are bullshit, you need better ones. There’s no way Gemini of any flavor is “closing in” on GPT4o to anyone who uses both
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva...
by resource_waste on Tuesday May 28, 2024 @03:12PM
(#40504395)
Haha what total crap.
Anyone who uses/used Gemini knows its
>Not good, the quality of results are quite generic and low quality. It talks more than it answers. When it talks, it gives you the straight and narrow, nice, happy answer.
>It denies seemingly half of my requests, so I have to reword it.
>It denies genuine requests, even when reworded.
I have chatgpt4 for 99% of things, including the denied requests. I have 7B Berkley Sterling for anything chatGPT4 doesnt answer.
I continue to try Google's, Microsofts, Claude, etc... I am always looking for better, I always pay for premium, I've ran 100B param offline models. Gemini is inferior to their direct competitors, there is nothing redeeming.
I'm convinced Gemini being at the top of any ranking system is some sort of over-fitting error or dealings with Google to promote their product.
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva...
by jxy on Tuesday May 28, 2024 @03:33PM
(#40504656)
Does LMSys even claim to be using the public API? What API endpoint is "Gemini-Advanced-0514"?
Is LMSys "Gemini-1.5-Flash-API-0514" the same as what the public API endpoint "gemini-1.5-flash-001" provides, or "gemini-1.5-flash-preview-0514"? And is LMSys named "Gemini-1.5-Pro-API-0514" the same as "gemini-1.5-pro-preview-0514"?
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva...
by theropost on Tuesday May 28, 2024 @03:42PM
(#40504761)
Maybe that's a problem for me every time I've tried it completely denies such basic requests, I don't have to worry about how I word things I just want to get my answer based on my inquiry without having to subscribe to a proper way of speaking.
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva...
by resource_waste on Tuesday May 28, 2024 @04:44PM
(#40505425)
You just reminded me to sell my Google stock. I got it cheap after it hallucinated in that public press conference. LLMs hallucinate, it was no big deal at the time.
I wrong, the quality IS terrible. But my stock did well. Time to take profits and move it to something like VT or SPY.
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva...
by jmclnx on Tuesday May 28, 2024 @02:35PM
(#40503967)
I really wish Google chose a better name than Gemini. Gemini was around years before Googles thing. Too bad it is so hard to fight a company like Google:
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva...
by tiagod on Tuesday May 28, 2024 @02:57PM
(#40504202)
To be honest, I don't think you cant get a lot more generic than "Gemini". I can think of many things sharing that name that are more well known than a niche HTTP alternative...
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva...
by FergusArgyll on Tuesday May 28, 2024 @05:22PM
(#40505757)
I quite like Gemini Pro. You can get a free API key, turn off the "safety" settings and Write your own system prompt. the free key is rate limited at 60 rpm, more than enough for my personal use.
I guess my preference is more about the product. I don't really have a strong opinion on "IQ" of gpt4 v gemini v claude, but if they gave me a free API key I'd prob use it
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva...
by viking123 on Wednesday May 29, 2024 @06:48AM
(#40510513)
Does it actually allow all sort of content if you turn off the safety? Like in roleplaying I hate if the models complain about violence or sexual content. I remember Altman mentioning they want to allow it too..
I may have to run tests to see if Flash is really that much better than Haiku. From the win rate heat map (https://x.com/lmsysorg/status/1795512212456645074) Flash has a 62% win rate vs. Haiku.
[1]: https://wow.groq.com/ [2]: https://ai.google.dev/pricing
https://docs.google.com/spreadsheets/d/1foc98Jtbi0-GUsNySddv...
Specific to Groq - the service is an incredible outlier for speed and cost, but even for quite small projects I've run into rate limits and similar errors. Great option for something like a bring-your-own-keys personal chatbot, but I wouldn't build a service on it. Groq Llama 3 70B and Groq Mixtral 8x7B are in the sheet for comparison but omitted from charts.
[1]: https://artificialanalysis.ai/leaderboards/providers
edit: ah, currently it's "free beta access with low rate limits" and "enterprise" that is available, but they are planning to release "on demand pay per token" with higher rate limits soon; exact pricing unknown.
https://pasteboard.co/OnLHjusiv6QT.png
another limitation of groq, which may or may not matter for your use case, is there's no way to pre-fill part of the llm response, which can be a very useful technique (there's also no support for the more general feature of "restrict sampling to a grammar", tho that is a much bigger ask).
We are only talking about it because people's "vibes" are that GPT-4o is "the best."
If a contrary result were found, the social media public would gravitate to something else written in the aesthetics of a primitive Gradio interface with some academic affiliation that told them the story they wanted to believe.
It doesn't really make sense, one way or another, than totally different models should perform so similarly, or even so differently, from the specific LMSys ranking task of "edify the end user." There's no explanation or analysis for the result. But because we "feel" "like" Anthropic's "big model" and Google's "big model" should perform the same as "OpenAI's big model," but because we also feel that "OpenAI is the best so it's latest model is also should be the best," well the results could say anything in any benchmarking approach, but we talk about LMSys's because they agree with our feelings, and we don't talk about other results because they do not.
If someone had a good way to benchmark LLMs, they would instead author training data.
How AI nerds (like us, so a particular subclass of the "normal" population) rate responses from various AIs - this rating is subjective, of course, and an LLM that produces answers that read well though is less accurate/"intelligent" would likely perform better than a model that propduces answers that read worse, though is more "intelligent".
AI nerds/IT nerds likely have higher prevalence of ASD/Asperger's etc (or higher up the spectrum), so likely not the best way to rank LLMs.
It's fun, though!
What other benchmarks do you think are being overlooked?
LMSys has the same energy as Amazon reviews of brand name products. It's measuring something, and that something might be valuable. I don't know if it is valuable for answering subjective, spicy questions like, "What LLM is likely to replace most or all of your job right now, giving you some time to arbitrage 1h of chatbot with a full time employment?" which is what everyone actually cares about. That is what this is all about. I can't tell you what benchmark is answering that question or others. I don't know!
The other one I consult occasionally is https://gorilla.cs.berkeley.edu/leaderboard.html - the Berkeley Function-Calling Leaderboard - because function calling is a really useful characteristic of smaller models.
I believe the best ways to evaluate these models is with your own set of task-specific evals for your own project, but these are REALLY difficult to construct. I still don't have a good solution there for my own work.
you might find it useful
https://github.com/carlini/yet-another-applied-llm-benchmark
I recently saw a paper getting LLM evaluation completely right in many respects: https://arxiv.org/abs//2402.00861
"Evaluating Large Language Models for Generalization and Robustness via Data Compression"
They evaluate LLMs by seeing how well they compress text/news/science from the future, over time. In a sense, this measures how "surprised" the LLM is by future developments.
This is, I believe, an important and strong task. A top researcher or economist will have a preternatural ability to "see where the puck is going" in a way that would be more surprising (perplexing) to a junior person or a layperson. Thus, things after the knowledge cutoff being more "expected" is a great sign of predictive generality.
IF it does this hundreds of time, it could alter the results.
Do AI companies care enough to do this vs the risks of discovery? Might be a fun project, and I can imagine that the people who make the decisions on which AI to use for their project/company, may look at this leaderboard.
Just check run the output of both outputs through the model and pick the one with highest likelihood according to the model you are trying to promote (since arguably you're trying to promote a model you have access to).
LMSys does have some anti-bot measures (e.g. Cloudflare) which makes it infeasible to manipulate the leaderboard for marketing.
Anyone who uses/used Gemini knows its
>Not good, the quality of results are quite generic and low quality. It talks more than it answers. When it talks, it gives you the straight and narrow, nice, happy answer.
>It denies seemingly half of my requests, so I have to reword it.
>It denies genuine requests, even when reworded.
I have chatgpt4 for 99% of things, including the denied requests. I have 7B Berkley Sterling for anything chatGPT4 doesnt answer.
I continue to try Google's, Microsofts, Claude, etc... I am always looking for better, I always pay for premium, I've ran 100B param offline models. Gemini is inferior to their direct competitors, there is nothing redeeming.
I'm convinced Gemini being at the top of any ranking system is some sort of over-fitting error or dealings with Google to promote their product.
The web apps tend to have extra security layers.
Is LMSys "Gemini-1.5-Flash-API-0514" the same as what the public API endpoint "gemini-1.5-flash-001" provides, or "gemini-1.5-flash-preview-0514"? And is LMSys named "Gemini-1.5-Pro-API-0514" the same as "gemini-1.5-pro-preview-0514"?
I wrong, the quality IS terrible. But my stock did well. Time to take profits and move it to something like VT or SPY.
https://en.wikipedia.org/wiki/Gemini_(protocol)
If they followed MS naming style.
I guess my preference is more about the product. I don't really have a strong opinion on "IQ" of gpt4 v gemini v claude, but if they gave me a free API key I'd prob use it