Or… the default LLM Google uses for search has been quantized to s**. Ask a proper Thinking model, with browsing enabled, and odds of a correct answer are much higher. There’s been substantial improvement in AI in even the last year.

Ask a human a question like this, and they also have a chance of getting it wrong, even when confident.

> Ask a human a question like this

Why would a human know specs for a random phone off the top of their head? The human response is either "I don't know" or "let me look that up", not a hallucination.

*so long as an accurate answer exists on the internet

Claude is OK at saying when it can’t find good information, but it’s still 50/50 on citing a source that has nothing to do with its claim.

I think that it feels a little wasteful to go to Google search to ask a question like this, only for the AI that's giving you an answer instead of page results to perform its own web search to get you the response.

Also, I asked a thinking model with browsing enabled and got this:

> The Google Pixel 10 is expected to support Wi-Fi 7 (802.11be), based on the Qualcomm Snapdragon 8 Gen 4 / Tensor G5 chipset it will likely use, which includes an integrated Wi-Fi 7 modem. Specific finalized specs aren't confirmed until Google's official announcement.

(Model GLM-5-Turbo - two months old - using Kilo Code in the "Ask" profile; in its thinking token churn it reasoned that it should keep the response brief and direct. Perhaps not the best suite of model+harness for this task, but it's what I had to hand that's not quantized to shit, is a thinking model, and has a web search tool available to it.)

> Ask a human a question like this, and they also have a chance of getting it wrong, even when confident.

We google something specifically because the humans within reach don't know. The goal of searching is, well, to search pages - we're trying to find a site when we use google search.

The goal when using an LLM is generally different; we want an answer, not a site.

LLMs are not a site. They are a clever person that can point you to sites. They, like humans, are fallible.

LLMs can not point you to sites, only in a general direction. That is because complete URLs do not exist as single tokens in any of the large models. It can synthesize a plausible-looking url, and if you're lucky that URL might even exist. But that doesn't mean that there is any relation between between the text surrounding a hyperlink in LLM output and the text on the linked page.

AI agents can verify and summarize URLs, but a plain LLM can not.

I bow to your correction. I was using LLMs as a sloppy shorthand for modern AI agents with best interfaces.

its bad in dev as well... i've seen llm code review bots tell me things that are flat-out not true; this like "this wont compile because windows 11 doesn't exist" like wtf am i paying for this again?