ChatGPT relies very much on confirmation bias to do its magic. You ask something trivial you already know (and could have probably Googled in 15 seconds), you get back something glib and smooth in reply and you are wowed by how smart GPT seems to be.

It is significantly less impressive when you ask something you don't already know and can't Google easily.

I actually love bing bot when searching for something very nuanced ("no, not like that ... [explanation]"). I haven't used Google's AI search yet; why learn a new tool?

How would you even know that it's less impressive if you don't know the subject matter?

Because it can be easy to test the result even if you don't know how to come up with a solution ?

If I asked you to build a light switch I don't need to know anything about EE to see that the light turns on/off when I flip the switch.

LLMs often fail to provide functional solutions for non trivial stuff (not impressive) - and once you know more about the domain, you usually figure out that the light switch will likely burn down your house in few hours because it's overheating.

Could you discern a good teacher from a bad teacher when you were in school and the topics were new? Did you ever find out someone was unreliable after they gave you some facts and later you discovered they were wrong? You won't have a 100% rate of detection and it'll be lower than in subjects you know about, but after a few times you can see if the new stuff it tells you is right or not.

I use it mostly for complex queries I can't Google and that's where it excels for me.

I use it _after_ doing complex googling to get an answer, and it is almost universally wrong, or misses the point completely.

It often just takes a few minutes of verifying against your understanding and use case, and ask a few more questions, and you can identify incorrect answers and arrive at the verified correct answer, or useful answer if there is no correct answer, rather quickly- which should be how most people operate in most interactions with information. Not to mention it can now be forced to search the web and cite its sources.

I think at some point it just becomes a preference of your interface to the data your searching, and I've been enjoying natural language

That's more or less my experience with code generation. To be fair, I haven't paid for ChatGPT-4, which I'm told is much better. Still, if I ask 3.5 a leetcode style problem I've done, it comes up with a good solution. If I ask it a leetcode style problem I just made up, it generally comes up with garbage -- the solution doesn't work, solves a different problem than I asked, or ignores a restriction I imposed.

The place where I've found it really handy is if I want to make stuff up. Things like "Give me a mundane event that could happen to a 25-year-old leatherworker for a story I am writing." or "What should I name a piece of software that allows people to collaboratively draw a landscape scene?"

> It is significantly less impressive when you ask something you don't already know and can't Google easily.

What's more, you absolutely cannot trust it with that kind of query.

Its insane to me that LLMs are primarily being pitched as fact givers. That is such a bad fit... They are so much better at writing poetry and fiction in chunks, or maybe analyzing stuff from the context.

Yea ChatGPT wasn't better at non hallucinating, but RLHF made the model much more useful for the average consumer, so consumers used it, instead of just other ML people. And, like you said, consumers are a bit less critical than experts from the field.

> the failure of these statistical language models to have a grounding in facts

I feel the "grounding in facts" was a big mistake, and to a large extent still is. The worst case of it was symbolic linguistic and knowledge graphs. Why? Because symbols don't exist, facts and concepts and words are fuzzy boundaries, their meaning defined mostly or entirely through associations with other words. We don't learn symbolically, we don't have strong "grounding in facts" - not at the language layer.

I'd rather say that language and understanding, the way we do it, is statistical in nature, and "grounding in facts" is done through feedback.

Grounding in facts is the one thing where machines could actually improve upon humans (in the same way a calculator doesn't fail on every 10th calculation).

In the context of LLMs it's only ever going to work out that wath math/stats, because garbage in garbage out will taint LLM output roughly to the same degree as human generation. Because humans are the garbage producers in both scenarios.

To bypass the garbage in problem, you'll need a different AI structure from LLMs almost certainly, a proper logic model.

Is it the fundamental architecture of an LLM that makes this problematic, or the input?

Based on how it's reported that they work, I assume that even without garbage in their statistical reasoning of the next word would occasionally come up with bad output. E.g. they might come up with 2 + 2 = 5 because of mathematical inputs such as "1 + 2 + 2 = 5".

Except it's unlikely because they have a bit larger context and are modeling a bit more than one-dimensional probabilities of "what comes after this?".

I'd say that LLMs are more resistant to GIGO, as long as the fraction of garbage in training data is small - it'll look like outliers to the larger model.

>“write X in the style of Y”

"Write this proof in the style of a program"

I think there is something a lot deeper behind this

Meta/Yann LeCun were generating the hype around Galactica that caused expectations to be different. You can't say things like 'Type a text and Galactica will generate a paper with relevant references, formulas, and everything'[1] and expect your LLM not to make you look foolish. It wasn't about pushing the model to the limit, it was about setting realistic expectations of what it could do.

[1] https://statmodeling.stat.columbia.edu/2022/11/23/bigshot-ch...