News for hackers. Stuff that matters.

A comment on Falcon 180B Released. Parent comment

Re:Falcon 180B Released by SkyPuncher on Wednesday September 06, 2023 @10:12AM (#37405448)
I don’t think this is a particularly useful benchmark.

It’s well known that LLMs are bad at math. The token based weighting can’t properly account for numbers that can vary wildly. Numbers are effectively wildcards in the LLM world.

[ Parent | on HN ]
Re:Falcon 180B Released by ben_w on Wednesday September 06, 2023 @10:30AM (#37405702)
Surely this is a "didn't read the question properly" problem rather than a "didn't maths right" problem?

And that (understanding a natural language question) is the USP for LLMs.

[ Parent | on HN ]