News for hackers. Stuff that matters.
Re:Llama 2 – Meta AI by baby on Tuesday July 18, 2023 @01:47PM (#36776518)
I'm wondering how do people compare different models? I've been trying chatGPT 3.5, bing chat (chatgpt 4 I believe?), and bard, and now this one, and I'm not sure if there's a noticeable difference in terms of "this is better"
Re:Llama 2 – Meta AI by losteric on Tuesday July 18, 2023 @01:59PM (#36776711)
Develop a set of queries for the use-case with human review of outputs. My team has an internal (corporate) tool where we drop in an S3 file, complete text over K models, then evaluate the completions with appropriate humans labor pools. Each evaluator gets a pair of outputs for the same prompt and picks the best.
Re:Llama 2 – Meta AI by jimmySixDOF on Tuesday July 18, 2023 @01:58PM (#36776708)
Try the Chat Arena with ELO ratings based on end user side by side blind tests. It run out of UC Berkley by lmsys the same team that released Vicuna.

https://arena.lmsys.org/

Re:Llama 2 – Meta AI by stavros on Tuesday July 18, 2023 @03:13PM (#36777774)
That's a terrible system, it doesn't represent gaps in performance. If the first model is orders of magnitude better than the second, that system still says "99% as good" or whatever.
Re:Llama 2 – Meta AI by sebzim4500 on Tuesday July 18, 2023 @05:48PM (#36779553)
The relative difference between elo ratings is meaningless, you need to look at the absolute difference.
Re:Llama 2 – Meta AI by baby on Tuesday July 18, 2023 @08:19PM (#36780719)
This is awesome! So basically GPT-4 is the winner far ahead of alternatives. I don't see Bard in the ranking though
Re:Llama 2 – Meta AI by netsec_burn on Wednesday July 19, 2023 @08:48AM (#36785453)
It's outdated.
Re:Llama 2 – Meta AI by kcorbitt on Tuesday July 18, 2023 @02:01PM (#36776756)
It depends -- do you mean as a general end-user of a chat platform or do you mean to include a model as part of an app or service?

As an end user, what I've found works in practice is to use one of the models until it gives me an answer I'm unhappy with. At that point I'll try another model and see whether the response is better. Do this for long enough and you'll get a sense of the various models' strengths and weaknesses (although the tl;dr is that if you're willing to pay GPT-4 is better than anything else across most use cases right now).

For evaluating models for app integrations, I can plug an open source combined playground + eval harness I'm currently developing: https://github.com/openpipe/openpipe

We're working on integrating Llama 2 so users can test it against other models for their own workloads head to head. (We're also working on a hosted SaaS version so people don't have to download/install Postgres and Node!)

Re:Llama 2 – Meta AI by dotancohen on Tuesday July 18, 2023 @01:52PM (#36776599)
Depends on the task. For code, ask it to implement a not-difficult but not-trivial feature. "Please add hooks to the AnkiDroid source code so that addons would be viable" might be a good start, for something that is on my mind. Then compare implementations.

For checking hallucinations, ask it about events and trivia that happened eons ago, and also within the last decade. Try some things that it can not possibly know, like how much celery Brad Pitt likes in his salad.

Re:Llama 2 – Meta AI by rajko_rad on Tuesday July 18, 2023 @02:00PM (#36776732)
This is an emerging space with lots of interesting tools coming out... There are many established benchmarks out there (i.e. included on front page of llama2 release), but most product builders have their own sets of evals that are more relevant to them...

Here is a thread exploring differences between llama-v2 vs. gpt3.5: https://twitter.com/rajko_rad/status/1681344850510376960