News for hackers. Stuff that matters.

A comment on Gemini: Google's most capable AI model yet. Parent comment

Re:Gemini: Google's most capable AI model yet by hulium on Wednesday December 06, 2023 @12:54PM (#38547262)
Why is that misleading? It shows Gemini with CoT is the best known combination of prompt and LLM on MMLU.

They simply compare the prompting strategies that work best with each model. Otherwise it would be just a comparison of their response to specific prompt engineering.

[ Parent | on HN ]
Re:Gemini: Google's most capable AI model yet by noway421 on Wednesday December 06, 2023 @09:21PM (#38552051)
> They simply compare the prompting strategies that work best with each model

Incorrect.

# Gemini marketing website, MMLU

- Gemini Ultra 90.0% with CoT@32*

- GPT-4 86.4% with 5-shot* (reported)

# gemini_1_report.pdf, MMLU

- Gemini Ultra 90.0% with CoT@32*

- Gemini Ultra 83.7% with 5-shot

- GPT-4 87.29% with CoT@32 (via API*)

- GPT-4 86.4% with 5-shot (reported)

Gemini marketing website compared best Gemini Ultra prompting strategy with a worse-performing (5-shot) GPT-4 prompting strategy.

[ Parent | on HN ]