Why can't it just check the URL to see if it's valid and actually contains the data it thinks it contains before recommending it to the user.

Don't modern LLMs like GPT4 have access to load web pages?

The downside is it will slow down the output I guess.

LLMs aren't "trained to pick a url from a database of known urls". Or at least, we have not collectively seen this yet, perhaps I am wrong. We explicitly wire things so we can use LLMs to generate data useful to index with existing search databases....

acting like LLMs are responsible for the reliability of identifying the source document is disingenuous when a large part of the value of an LLM as an asset is its role laundering other peoples' intellectual property without attribution.

Try out phind.com for an example. The LLM generates a query for the search engine to fetch web results. It can include clickable links to the sources in its response.

They accomplish this by training/fine-tuning the LLM to output special tokens, or special syntax, which is interpreted in code to perform some action, such as calling out to an API. This is how chatgpt and Gemini are able to automatically search the web and generate images and such. Yes, LLMs just output tokens, but those tokens can be interpreted to perform actions in code.