News for hackers. Stuff that matters.
Re:Llama 2 – Meta AI by cheeseface on Tuesday July 18, 2023 @12:12PM (#36774834)
Would really want to see some benchmarks against ChatGPT / GPT-4.

The improvements in the given benchmarks for the larger models (Llama v1 65B and Llama v2 70B) are not huge, but hard to know if still make a difference for many common use cases.

Re:Llama 2 – Meta AI by majorbadass on Tuesday July 18, 2023 @12:43PM (#36775396)
"In addition to open-source models, we also compare Llama 2 70B results to closed-source models. As shown in Table 4, Llama 2 70B is close to GPT-3.5 (OpenAI, 2023) on MMLU and GSM8K, but there is a significant gap on coding benchmarks. Llama 2 70B results are on par or better than PaLM (540B) (Chowdhery et al., 2022) on almost all benchmarks. There is still a large gap in performance between Llama 2 70B and GPT-4 and PaLM-2-L."
Re:Llama 2 – Meta AI by gentleman11 on Tuesday July 18, 2023 @01:03PM (#36775739)
it's not open source
Re:Llama 2 – Meta AI by elaus on Tuesday July 18, 2023 @01:08PM (#36775833)
This quote does not talk about Llama being open source.
Re:Llama 2 – Meta AI by messe on Tuesday July 18, 2023 @01:16PM (#36775973)
The commenter you're replying to did not say that it was.
Re:Llama 2 – Meta AI by chaxor on Tuesday July 18, 2023 @12:35PM (#36775254)
It would be nice to see 6 of them trained for different purposes by combining 5 of their outputs together and 1 trained to summarize for the most complete and correct output. If we are to trust the leaks about GPT-4, this may be a more fair comparison, even if it is only ~10-20% of the size or so.
Re:Llama 2 – Meta AI by pedrovhb on Tuesday July 18, 2023 @09:00PM (#36780979)
Isn't that essentially beam sampling?
Re:Llama 2 – Meta AI by jmiskovic on Tuesday July 18, 2023 @12:31PM (#36775189)
Then why not read their paper?

"The largest Llama 2-Chat model is competitive with ChatGPT. Llama 2-Chat 70B model has a win rate of 36% and a tie rate of 31.5% relative to ChatGPT."

Re:Llama 2 – Meta AI by capableweb on Tuesday July 18, 2023 @12:34PM (#36775244)
Do they specify which GPT version they used? Could Llama 2 really beat GPT-4?
Re:Llama 2 – Meta AI by jmiskovic on Tuesday July 18, 2023 @12:41PM (#36775357)
The 70B Llama2 model ties in with 173B ChatGPT-0301 model. The GPT-4 still stands unchallenged.
Re:Llama 2 – Meta AI by sebzim4500 on Tuesday July 18, 2023 @02:28PM (#36777154)
Source on the 173B parameters?
Re:Llama 2 – Meta AI by exradr on Tuesday July 18, 2023 @09:09PM (#36781030)
The wikipedia article for GPT-4 has this as its source: https://the-decoder.com/gpt-4-architecture-datasets-costs-an...
Re:Llama 2 – Meta AI by jmiskovic on Thursday July 20, 2023 @04:38AM (#36798149)
Re:Llama 2 – Meta AI by davidkunz on Tuesday July 18, 2023 @12:44PM (#36775417)
They used ChatGPT-0301, it can't beat GPT-4.
Re:Llama 2 – Meta AI by illnewsthat on Tuesday July 18, 2023 @12:46PM (#36775444)
The paper[1] says this in the conclusion:

> [Llama 2] models have demonstrated their competitiveness with existing open-source chat models, as well as competency that is equivalent to some proprietary models on evaluation sets we examined, although they still lag behind other models like GPT-4.

It also seems like they used GPT-4 to measure the quality of responses which says something as well.

[1] https://ai.meta.com/research/publications/llama-2-open-found...

Re:Llama 2 – Meta AI by janejeon on Tuesday July 18, 2023 @12:48PM (#36775483)
In the paper, I was able to find this:

> In addition to open-source models, we also compare Llama 2 70B results to closed-source models. As shown in Table 4, Llama 2 70B is close to GPT-3.5 (OpenAI, 2023) on MMLU and GSM8K, but there is a significant gap on coding benchmarks. Llama 2 70B results are on par or better than PaLM (540B) (Chowdhery et al., 2022) on almost all benchmarks. There is still a large gap in performance between Llama 2 70B and GPT-4 and PaLM-2-L.