News for hackers. Stuff that matters.

I just spent the past 5 hours comparing LLMs

The top 2 takeaways I took from this is that.

1. No one should still be using GPT 3.5 or Gemini 1.0, They may get the job done, but 98% of the time one of these LLMs (Gemini 1.5 Pro, Bing Copilot, ChatGPT4o, and Perplexity) will give you a better response.

2. The best way to always get the best response is by always comparing multiple AI Chatbots and choosing the best answer. This is because

If you are only using ChatGPT-3.5, llama3, claude sonnet, or mistral 100% of the time there is another LLM with a better answer. If you are only using Gemini 1.0 96% of the time there is another LLM with a better answer.

If you are only using perplexity 86% of the time there is another LLM with a better answer.

If you are only using ChatGPT-4o 73% of the time there is another LLM with a better answer.

If you are only using Gemini 1.5 Pro 73% of the time there is another LLM with a better answer.

If you are only using Bing Copilot 73% of the time there is another LLM with a better answer.

I personally used chatplayground.ai to compare all of them.

I stopped using chatgpt 3.5 a long time ago because this gives you all the pro llms for the same price as gpt4

It is very important to note this research was done on a very small dataset of 22 questions.

The best LLM answer for each question was decided by me simply putting myself in the choose of the person asking that question and deciding which output is the most helpful.

Perplexity is good at giving you a completely answer, most llms will give you bullet point answers, perplexity perfers to write out a complete answer rather than just giving bullet points

Bing Copilot is great because it cites its resources, will even recommend videos to help you

ChatGPT-4o has really descriptive and usually longer answers, but prefers to answer in bullet points

Gemini 1.5 pro is great because it feels like it understands the context of your question more by having a conversational tone

Bing copilot can be great but 20-30% of the time the answers it gives are not even usable

Because Bing copilot is using sources to give you answers the answers feel much more human like and at times more useful then other LLMs that just list a bunch of basic bullet points

Their should be no reason anyone is still using ChatGPT 3.5 today

Gemini 1.0 gives good answers but not as detailed and helpful as Gemini 1.5 Pro

ChatGPT-4o was able to generate really nice data tables, that Gemini 1.5 Pro wasn't able to.

ChatGPT-3.5 ChatGPT-4o Gemini 1.0 Gemini 1.5 Pro Bing Copilot Claude Sonnet Llama 3 Mixtral 8x7b Mistral Large Perplexity

( 7 comments | 4 points | on HN )
Re:I just spent the past 5 hours comparing LLMs by Venkatesh10 on Saturday May 18, 2024 @08:48AM (#40398577)
What's the reasoning for the data? Show the reference, table, graph to backup.
Re:I just spent the past 5 hours comparing LLMs by navjack27 on Saturday May 18, 2024 @08:56AM (#40398633)
Your number one that you bring up shouldn't be a universal. You're basing everything, it seems, upon LLMS as being knowledge retrieval systems. You are dismissing situations where someone might want more creative output without having to guide the LLM with a more strict prompt.
Re:I just spent the past 5 hours comparing LLMs by Jimmc414 on Saturday May 18, 2024 @10:52PM (#40403840)
GPT 3.5 and haiku are great for prototyping
Re:I just spent the past 5 hours comparing LLMs by qup on Sunday May 19, 2024 @06:13AM (#40405783)
3.5 great for summarizing
Re:I just spent the past 5 hours comparing LLMs by karanveer on Sunday May 19, 2024 @09:21AM (#40406776)
you also recently did a post on reddit, didnt u? you're associated with chatplayground.ai where u did 80k views and got 8 meaningful conversions in the end or so...is that u?
Re:I just spent the past 5 hours comparing LLMs by smarri on Monday May 20, 2024 @01:21PM (#40417755)
Thanks. Can you share a sample of the questions you used? Were you comparing the LLMs to answers you knew already, or comparing the new answers which were output by the LLMs
Re:I just spent the past 5 hours comparing LLMs by atleastoptimal on Monday May 20, 2024 @05:47PM (#40420827)
GPT-4T/o is the smartest

Opus has the best vocabulary/apparent creativity

Gemini is the most "neutral"