Re:The A.I. Bubble is Bursting with Ed Zitron [video]
by xracy on Sunday July 07, 2024 @03:02PM
(#40899657)
I'm not asking for someone to overfit to being able to properly answer the question of "What day is it today?" I'm giving an example of a simple question that all LLMs need to be able to answer correctly.
But like, people are on here saying that this will make scientific improvements, and until it can get past the basic stuff, it's not in the ballpark of anything more complicated. Right now, we're basically at the stage of 10 million monkeys on 10 million typewriters for 10 million hours. Like, maybe we'll get Shakespeare out of it, but are we willing to sort through all of the crap it will generate along the way, when it can't actually create a useful answer to simple questions?
But has anything changed? Well no, because it's obviously trivially possible for them to play a decent game of chess (or correctly assess the date), but it's an example of a more general issue of LLMs being generally incapable of consistently engaging in simple tasks across arbitrary domains. So you have software that can score some high thing on the LSAT or whatever, but can't competently engage in a game children can play.
The over-specialization for the sake of generating headlines and over-fitting benchmarks is, IMO, not productive. At least not in terms of creating optimal systems. If the goal is to generate money, which I guess it is, then it must be considered productive.