News for hackers. Stuff that matters.

A comment on Gemini: Google's most capable AI model yet. Parent comment

Re:Gemini: Google's most capable AI model yet by belval on Wednesday December 06, 2023 @12:24PM (#38546856)
Don't look at absolute number, instead think of it in terms of relative improvement.

DocVQA is a benchmark with a very strong SOTA. GPT-4 achieves 88.4, Gemini 90.9. It's only 2.5% increase, but a ~22% error reduction which is massive for real-life usecases where the error tolerance is lower.

[ Parent | on HN ]
Re:Gemini: Google's most capable AI model yet by machiaweliczny on Wednesday December 06, 2023 @12:44PM (#38547124)
This + some benchmarks are shitty thus rational model should be allowed to not answer them but ask claryfying questions.
[ Parent | on HN ]
Re:Gemini: Google's most capable AI model yet by belval on Wednesday December 06, 2023 @12:49PM (#38547202)
Yes, a lot of those have pretty egregious annotation mistakes. Once you get in high percentage it's often worth going through your dataset with your model prediction and compare. Obviously you can't do that on academic benchmarks (though some papers still do).
[ Parent | on HN ]