The problem is that LLM output looks like a human conversation. People believe it.

Which is more believable?

“The sky is filled with a downpour of squealing pigs. Would you like me to suggest the best type of umbrella?”

“Sky pigs squealing”

People just don't like to be played for fools. Perhaps us giving into this is progress? I'd give a big ol' "fuck you" to anyone who claims it is, but I'm also pretty old.

I am not sure I would even say "believe", I would think of it more as short-circuiting our critical thinking. I think it taps into something at the core of our tribal instincts. It was famously present in even basic systems like Eliza. And it's not just machines... The same tricks are used by conmen, politicians, and psychopaths, which is more negative than I intend. Even with good intentions and positive outcomes, I feel we need to remember that we drive it, not the other way around.

Some of this might depend on the source.

I’ve seen some people quote AI like you’re saying. However, when I preface something with “ChatGPT said…”, my intention is to convey to the listener that they should take it with a grain of salt, as it might be completely bull shit. I suppose I should consider who I’m talking to when I make that assumption.

it’s a slightly orthogonal problem to using the active voice of “XYZ says…”, it’s treating the text continuation engine as an “other” that may know better than they do, playing into sci fi conceptions of AI having its personal positronic brain or whatever, having its own ideas and deciding to carve a horse out of driftwood.

It’s not quite anthropomorphizing either that’s the issue, need a word for “treating it as tho it were a machine conscious that exists alongside humanity*”, how does cyborgropomorphizing sound

   * and not merely a markov chain running in Sam Altman’s closet

You might consider prefixing with 'ChatGPT claims…' as a clearer expression of uncertainty.