> I mean, it probably is right, but this "I don't know something, but I trust what the chatbot told me" is what worries me about the rise of the LLMs.

I share the same worry with regards to the way humans will use AI, along with worries about enabling various antisocial behaviours at scale.