> They all seem aware to some extent that these tools can occasionally generate nonsense.

You have too many smart people in your circle, many people are somewhat aware that "chatgpt can be wrong" but fail to internalize this.

Consider machine translation: we have a lot of evidence of people trusting machines for the job (think: "translate server error" signs) , even tho everybody "knows" the translation is unreliable.

But tbh moral and truth seem somewhat orthogonal issues here.