News for hackers. Stuff that matters.

A comment on Claude 3.5 Sonnet.

Re:Claude 3.5 Sonnet by zone411 on Thursday June 20, 2024 @11:32AM (#40739876)
Slightly better on the NYT Connections benchmark (27.9) than Claude 3 Opus (27.3) but massively improved over Claude 3 Sonnet (7.8).

GPT-4o 30.7

Claude 3.5 Sonnet 27.9

Claude 3 Opus 27.3

Llama 3 Instruct 70B 24.0

Gemini Pro 1.5 0514 22.3

Mistral Large 17.7

Qwen 2 Instruct 72B 15.6

[ Parent | on HN ]
Re:Claude 3.5 Sonnet by zsmizzle on Thursday June 20, 2024 @02:55PM (#40741970)
It still fails to be the moderator of a WORDLE board. That is always the first test I do of these new models.
[ Parent | on HN ]