LLMs work on tokens. They literally cannot see individual letters. Yet, unlike humans, they can instantly create a tool to solve this task for them and never be stumped by it again.
I'm so tired of this brain-dead gotcha comment.
Do you think your legislators or most users understand tokens and predictions and what is going on? Do you think that those who bombed a school using it know this?
Until frontier models can account for the disparity between what they can do and what the person (or other agent) is asking, we will continue to suffer bad information used like good information.
That tokens cannot count letters like that is not a brain-dead gotcha. Ignoring how humans are using these models is lacking in critical understanding of the real world. People are literally going to die due to humans not understanding how this works (and already have). True brain-dead, and body-dead.
People formed their opinions on LLMs in late 2022/early 2023 when ChatGPT was first released and taking the world by storm and then never updated them after nearly 4 years of improvements.
Hell, just being able to search the Internet did MASSIVE things for reducing hallucinations.
Do we really understand what AI is and is not? For example we all know AI can hallucinate, but how does that knowledge help us in practice? If I'm leaning something new I would have to fact check everything which would take about as much time as learning from verified sources.