It does citations (Grok and Claude etc do too) but I've found when I read the source on some stuff (GitHub discussions and so on) it sometimes actually has nothing to do with what the LLM said. I've actually wasted a lot of time trying to find the actual spot in a threaded conversation where the example was supposedly stated.

Same experience with Google search AI. The links frequently don’t support the assertions, they’ll just say something that might show up in a google search for the assertion.

For example if I’m asking about whether a feature exists in some library, the AI says yes it does and links to a forum where someone is asking the same question I did, but no one answered (this has happened multiple times).

It is funny, Perplexity seems to work much better in this use case for me. When I want some sort of "conclusive answer", I use Gemini pro (just what I have available). It is good with coding and formulating thoughts, rewriting text, so on.

But when I want to actually search for content on the web for, say, product research or opinions on a topic, Perplexity is so much better than either Gemini or google search AI. It lists reference links for each block of assertions that are EASILY clicked on (unlike Gemini or search AI, where the references are just harder to click on for some reason, not the least of which is that they OPEN IN THE SAME TAB where Perplexity always opens on a new tab). This is often a reddit specific search as I want people's opinions on something.

Perplexity's UI for search specifically is the main thing it does just so much better than google's offering is the one thing going for it. I think there is some irony there.

Full disclosure, I don't use Anthropic or OpenAI, so this may not be the case for those products.

In my experience, 80% of the links it provides are either 404, or go to a thread on a forum that is completely unrelated to the subject.

Im also someone who refuses to pay for it, so maybe the paid versions do better. who knows.

The 404 links are truly bizarre. Nearly every link to github.com seems to be 404. That seems like something that should be trivial for a tool to verify.

Yeah. The fact that I can't ask ChatGPT for a source makes the tool way less useful. It will straight up say "I verified all of these links" too.

As you identified, not paying for it is a big part of the issue.

Running these things is expensive, and they're just not serving the same experience to non-paying users.

One could argue this is a bad idea on their part, letting people get a bad taste of an inferior product. And I wouldn't disagree, but I don't know what a sustainable alternative approach is.

Surely the cost of sending a few HTTP requests and seeing if they 404 is negligible compared to AI inference.

I would have no issue if the free version of ChatGPT told me straight up “You gotta pay for links and sources”. It doesn’t do that.

100% agree with that, as I alluded to in my last sentence. And that honestly seems like it might be a good product strategy in the short term.

> The 404 links are truly bizarre. Nearly every link to github.com seems to be 404. That seems like something that should be trivial for a tool to verify. reply

Same issue with Gemini. Intuitively I'd also assume it's trivial to fix but perhaps there's more going on than we think. Perhaps validating every part of a response is a big overhead both financially and might even throw off the model and make it less accurate in other ways.

That's a thing I've experienced, but not remotely at 80% levels.

It might have been the subject I was researching being insanely niche. I was using it to help me fix an arcade CRT monitor from the 80’s that wasn’t found in many cabinets that made it to the USA. It would spit out numbers that weren’t on the schematic, so I asked for context.