Re:Mass attempted replication of old findings of LLMs with n...
by dwroberts on Tuesday November 03, 2026 @09:12AM
(#50352232)
> Mostly capability failures that newer models no longer make: counting letters
Have LLMs fundamentally actually changed or improved to resolve problems like this, or have the big models been aggressively tuned / trained on specific examples so it’s less embarrassing?
Because I’ve noticed that Google’s stuff always seems to generate code to do counting tasks, even when not asked for, so I wondered if the LLM itself even generated the answer or it’s just being instructed to use a tool? (Which IMO is a workaround)
Have LLMs fundamentally actually changed or improved to resolve problems like this, or have the big models been aggressively tuned / trained on specific examples so it’s less embarrassing?
Because I’ve noticed that Google’s stuff always seems to generate code to do counting tasks, even when not asked for, so I wondered if the LLM itself even generated the answer or it’s just being instructed to use a tool? (Which IMO is a workaround)