They certainly don't make hyperspecific claims like "this YouTuber traveled to Israel and changed his mind about the war there, as documented in a video he posted on August 18".
If a Fortune teller published articles claiming false things about random prople, gave dangerous medical advice, claiming to be a Nigerian prince, or convinced you to put all your savings into bitcoin; the "entertainment purposes" shield dissolves quite quickly.
Google makes an authorative statement on top of the worlds most used search engine, in a similar way they previously did with Wikipedia for relevant topics.
The little disclaimer should not shield them from doing real tangible harm to people.
The Google disclaimer should probably be upfront and say something more like, “The following statements are fictional, provided for entertainment purposes only. Any resemblance to persons living or dead are purely coincidental.”
And even if they did, it wouldn't really matter. The way Google search is overwhelmingly used in practice, misinformation spread by it is a public hazard and needs to be treated as such.
So you accept that all of this is just a quibble over what the disclaimer says? Rather than "AI generated, might contain mistakes", it should just say "for entertainment purposes only" and they'll be in the clear?
Fortune tellers and tarrot card readers are quite authoritative as well. They ostensibly claim to be able to tell the future, but they get away with it by adding a tiny disclaimer that the advice they dispense are for "entertainment purposes only".