I believe AGI/ASI level of an artificial intelligence is an interception of 1, 2, 3. LLM is NN which is not a true Expert system, so it will never get rid of hallucinations. But one ability impresses me greatly - an ability to read long text and to make a digest, I craved this tool when I was a student.
[1] https://en.wikipedia.org/wiki/Expert_system [2] https://en.wikipedia.org/wiki/Lisp_(programming_language) [3] https://en.wikipedia.org/wiki/Neural_network
One could point out that humans suffer from the same problem. Hallucinations should not be a problem for declaring something AGI. Additionally saying "science advances" is roughly the equivalent to "we've just realized we've all been hallucinating about X".
And there's in the extreme Turing's theorems. They do mean even a perfect ChatGPT would still not know everything (and practically, it would need time, a lot of time, before it really knew more than humans do).
The fact that LLMs appear so intelligent to humans is really a reflection of our inability to imagine effects of scale. We can understand simple linear predictions as trivial calculations, but when language-based pattern discovery is many layers deep and those patterns are combined in nontrivial (but non-intelligent) ways, we project intelligence onto the result.