In which case we would expect an LLM-based system to output something like,

"(Fact that isn't true)[Source that does not make that claim but comes close enough that you wouldn't notice it just by skimming]"

leading to even more people thinking its true than otherwise.

Again this is already something that the human equivalents (low-end PR etc.) do all the time.

This is going to get real nasty when you start getting into science of various sorts, where an increasing number of papers, particularly in the social sciences, tend to make extreme and outlandish claims, but then walk them back extensively in the actual paper itself emphasizing it's just a correlation, or otherwise demonstrating an extremely marginal effect.

Yet the media tends to miss the internationally understated nuance and run with the claims at face value, which an LLM will then pick up on and state, "Yes scientists have proven that [x] does cause [y]. [1][2][3][4][5]" That claim then gets repeated elsewhere, and eventually that "elsewhere" goes on to become part of the LLM's new training material where it's basically training off its own output.

It'd be ironic if something that's ideally designed to make the breadth of human knowledgeable more readily available and accessible than ever before, ends up just making society vastly more misinformed than ever before.