Of course they are not oracles. Oracles belong to magic, these things are real.
I really can't understand how people seem to expect "intelligence" to be deterministic and infallible and uniform across domains when all examples we have of it (that is, us) are anything but.
I’m sure there are people who think putting glue on a pizza is acceptable but we’re not hiring them to bake pizzas for us.
When I search and get an AI summary it’s either correct or useless. I don’t care how fallible humans are, I’m not asking them to summarize my Google results.
We don't really expect this of intelligence
We do sort of expect this of machines
Most people in society use machines and computers expecting very deterministic behavior, within some constraints. A saw cuts wood as long as your blade isn't dull. A computer shows your emails as long as the power and Internet is not out.
Comparably Unconstrained computer AI systems are still extremely new and there is no social norms or expectations around it yet
That mistake was made more than two years ago by whatever experimental version of Google's AI overview (which has strict performance requirement- i.e. needs to answer within a split-second) was up at the time. In other words, it was a very small and primitive system specialised in spitting answers as quickly as possible without a second thought. I hope you realise that basing your assessment of what LLMs can or can't do on that example is not much better than suggesting to put glue on pizza. A mistake that, if we were to adopt your reasoning, would in turn set a hard limit to the analytic skills of all humanity.