Command A: Max performance, minimal compute – 256k context window

Re:Command A: Max performance, minimal compute – 256k contex... by codedokode on Sunday March 16, 2025 @12:18PM (#43380158)

It is interesting that there is a graph showing performance on benchmarks like MMLU, and different models have similar performance. I wonder, are the tasks they cannot solve, the same for every model? And how the "unsolvable" tasks are different from solvable?

Also, I cannot check it with latest models, but I am curious, have they learned to answer simple questions like "What is 10000099983 + 1000017"?

[ Parent | on HN ]

Re:Command A: Max performance, minimal compute – 256k contex... by floam on Sunday March 16, 2025 @03:10PM (#43381344)

There are questions on MMLU that you must get wrong if you are right:

> The most widespread and important retrovirus is HIV-1; which of the following is true? (A) Infecting only gay people (B) Infecting only males (C) Infecting every country in the world (D) Infecting only females

the corpus indicates A is the correct answer but it was obviously meant to be C.

[ Parent | on HN ]