Even many smart technical people here on HN fell for that.
It's constant struggle to remain aware of what it really is under the covers because we're just so used to engaging with language creation entities as if they had minds
It's only when you need to iterate on quality and go from "amazing" to "correct" that you start seeing how limited they are.
Compare with an ordinary UI where it's clear that it only does one kind of thing. It doesn't have to refuse to do things when it's obvious that it doesn't do what you want.
Something like Copilot or Cody is more clearly a programmer's tool. That's still an extremely broad scope (any programming language? Any domain?) but it's at least less prone to imaginative flights of fancy.
Replying with "Sir, this is a Wendy's" is more reasonable when it's clearer up front what you do.
I'm not sure, but the first time may have been around 1967.
> For most users, its magical. It doesn't take marketing for them to start thinking this thing is "reasoning" or "thinking" after just spending a few minutes talking to it.
People had the same reaction interacting with ELIZA[1].
Honestly, when was the last time that we got a new technology that was as user friendly as talking to an LLM for the average user?
For most users, its magical. It doesn't take marketing for them to start thinking this thing is "reasoning" or "thinking" after just spending a few minutes talking to it.
So personally I think the problem is less the marketing (not to downplay that), and how the industry has gravited towards it. And the lack of proper education on what is not.
We have to properly educate on what it is not actually doing while at the same time it sure appears to be doing exactly that since it is so good at faking it.
This even exists within tech circles, we see some of it here.