This is the mental leap folks need to be making, the question is when will this type of configurably factual large language model be research benchmarked and productionized.

I have read many similar comments to the grandparent, all saying the same thing, “I tried it, and it turned up wrong facts” all the while they forgot that they were deeply engaged with the content it was producing. Just think of how much our attention is monetized already, money will be poring into this space from here on. It’s only a matter of time.

My argument isn't that I tried it and it didn't work, my argument is that an LLM alone fundamentally cannot work for reliable information retrieval.

Producing hallucinations or not isn't just a setting you can tweak in the model. The entire function of GPT-3 is to guess the next word from the words that came before. It's a hallucination machine that has been trained on so much real-world data and has such a large parameter set that for commonly known and discussed information it does a remarkably good job at creating factual sentences. But as soon as you get out into territory that it doesn't have memorized, it will do the next best thing and produce credible-sounding new material that may or may not be complete nonsense. Again, this isn't a setting you can tune, it's just what a transformer does. It models human language, not all of human cognition.

ChatGPT isn’t trained on next word prediction alone, it was trained on human feedback. The evaluation is next word search, but the model has evolved to give “correct” answers when evaluated that way. So it’s not just the most likely answer anymore.

Also, since we can watch the model evaluation and see all its weights, there’s research that shows you can determine if any answer is a “retrieval” of existing knowledge or a hallucination.

For the same reason why it might produce a bogus answer: it's a language model, not a general AI. Prompt it for footnotes and it will generate footnotes for books that don't exist or that don't actually contain the citation. They'll match the form of a footnote, because that's what the model was trained to do, but the citations won't resolve.

For example, I asked who the first president of Argentina was. It gave the wrong name (Juan Manuel de Rosas) with a half-true bio. I then asked it for books where I can learn more about this person.

It produced five books. None of the book-author pairings were correct, though three of the books were real and four of the authors actually wrote books. Of the four authors that existed, three actually did write about Argentina's history, while one is a professor of Spanish and gender studies.

This is my point: ChatGPT was trained to produce text, not to produce facts. What I'm seeing here isn't just something that can be optimized away, it's fundamental to the model's design. It was basically told to produce text that looks like it could have been written on the internet. Its text is really convincing: when I saw the list of books I thought for a minute it had actually done it! But it's still just inventing text that reads well, because that's what its training was optimizing for.

It could be paired with a different system that searches for facts, but then it's just a friendly abstraction layer on top of Google or some other system, not a revolution in search.

That makes sense. The only thing about gpt that impressed me was when I asked it to condense an answer into one sentence, it was able to do it.

Other than that it betrayed no understanding of the material. It rather resembled a high school kid who had read a few too many books and spouted platitudes about any and all subjects without being aware of what was really going on.

OTOH I could imagine a hybrid situation where the AI Bot searches Google for you and restates the millions of results Google gives it

I agree that a hybrid could work, although ChatGPT struggles to keep context with more than a dozen paragraphs, so I doubt it could summarize millions of results.

And in either case, you're not solving the problem that commenters here are trying to solve: Google's bad results will still affect your search, so ChatGPT will be fed largely by dubious blogspam.

Even in a small number of paragraphs language models hallucinate. For example Barack Obama, it’s not hard for me to trick it into saying he was born in Chicago. He was not. But there’s a ton of text about Barack Obama in Chicago. Plus, Michelle was born there. When predicting the most likely next token for “Barack Obama was born in ______” there’s a lot of confounding text.

No, there’s a fundamental issue in that statistical language models don’t start with facts to form sentences, they form sentences based on previous tokens and statistical weights. The weights are not auditable or easily updatable.

And any kind of "tuning" of the weights to get better results is going to be so bespoke that you might as well just curate the response yourself

So LLM's are pretty much Mark V Shaney at heart, only with enough weights to get to the far side of Eliza?

Though for that matter a ChatGPT based product would be even more vulnerable to the same thing that made Google search suck: content sources gaming its process

This was my immediate thought as well. I don’t need millions of searches done, I need 20, then a language model to look through them and collect text that helps me find what I’m looking for.

The lucky thing for Google if they got it working, is they are half there already. As much as I despise Google, I have to admit GPT won’t be their death.

Google already does this it is called Knowledge Graph[0].

[0] https://support.google.com/knowledgepanel/answer/9787176?hl=...

> This is my point: ChatGPT was trained to produce text, not to produce facts.

That's only because it was trained on non-facts also.

But that holds for Google Search just as well ...

I think there's a difference in presentation.

Google Search gives you links to content that might have the answer (excluding their own box). You can see the differences between the authors of said content.

ChatGPT gives you content that looks like the answer. You can't see the authors of the generated content.

Do you think that ChatGPT can't be trained to output links to sources?

Because to make it focus on always producing accurate results (or rather a very low false positive rate) you need to dramatically change and tune the model. Which is much harder than what Google does because it means you'd also need to vet a lot of facts as well (by weighting certain sources, not individually). Or allowing it to search the internet (which also comes with other issues and if it is using Google to do that then it doesn't really kill Google). I do think Google's knowledge graph is more in line with what you're actually thinking. Lots of Google searches can be done without actually clicking links.

There's also the ethical question. For example, duckduckgo will show you stack overflow answers. Google could easily do this but they have already gotten flack for displaying answers and taking away clicks from sites, which means reduced ad revenue for the resource actually being used. We could make even stronger arguments about services like ChatGPT in their current form, let alone the one we're talking about.

If we think a bit further in the future there's another extension that might not be what we want. Suppose our model contains all current human knowledge and can accurately give you answers and even produce paragraphs at will. This kills off a lot of websites. Is this a good thing? Suppose we incorporate an image generator along similar lines and now we have text and image search. That's a huge centralization force. There are a lot of great benefits to this that I don't want to downplay, but we should also think about if this is actually what we want because there are downsides too.

I thought I should also add that I had an at length discussion of these LLMs being Stochastic Parrots (an how we know they are) in a thread from 2 days ago https://news.ycombinator.com/item?id=34113011

What we want is not what we get unless it makes bank. Nobody (generalizing here) works to better humanity, they work to get money. if it aligns well that’s just shear luck. The world needs a lot more people who want see the world to succeed.

I think this is a highly pessimistic view. While I know many that are profit motivated and I won't argue that this isn't the majority, I know many people that just want to pursue knowledge for the sake of knowledge. I'll tell you that personally my dream is to just be able to research and not worry about constraints like publishing (I'd rather open blog) or computational constraints (don't need a data center. A few dozen A100s is more than enough). There are a lot of people like me and a lot that would be like that if they didn't have financial constraints. Having this dream is a privilege and even I didn't have it when I was living paycheck to paycheck.

So I think if you want to see more people with my dream then push towards a post scarce world. We'll still have plenty of resource constraints, but it does change the game a lot. I think you'll see more artists as well. A new renaissance. But I fear the transitionary period might be quite difficult and disruptive. I do think such a future has a fairly good probability of happening within many of our lifetimes and almost guaranteed if we actively pushed for it (or solved adjacent problems like HLI).

So don't be so pessimistic. There's millions of people out there that are interested in pushing the bounds of human knowledge because that's what they want. I'd argue that physicists and mathematicians wouldn't exist if this weren't true. They get paid shit for extremely difficult work. We could say the same about many sciences. While not all of these people are in it for bettering humanity (just for pursuing knowledge) it often aligns and there is a decent portion who believe this is the point of science and our purpose, as humans, in life.

While I see what you are saying and there are people doing the good work, when I read of companies circumventing the Russian embargo knowing full well their stuff gets turned into weapons, and people without thought buying the cheapest shit from amazon and alibaba made in dire circumstances, I am not so sure, or at least, do not see enough awareness for a treshold changing outcome. Some play the economic game on a criminal level, and some profit without giving back and without scruples.

> ChatGPT in its current form can’t replace google but plausible to think in its next few incarnations it will get there.

I don’t think it is.

> Why can’t it have a mode where it returns more rigorous results, and includes some system of links as citation?

Because it is a statistical text-generation model; connection of results to specific sources is fundamentally not part of the way it works, so it can’t just be asked to supply them. You’d need a fundamentally different approach to do that.

Contrary to all the naysayers I think a future version of GTP will be trained to evaluate its own certainty and express this verbally. Probably through an extra training channel.

It has no concept of certainty, unless it was embedded in the original data. What you are describing is an more general AI. This is a language model. I like to think of it of a cognitive intelligence: the ability to process input and compare it to a stunning array of trained data that will have many parallels to reality but has no concept of it.

It can be trained some form of trustfulness, and a measure of certainty is intrinsically part of the model (compare with image to text models etc "70% cheetah 10% dog")

In any case it can be trained to assert confidence via a side channel