News for hackers. Stuff that matters.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by cl42 on Thursday January 26, 2023 @02:04PM (#34536972)
It's amazing how many researchers underestimate the importance of UX and design.

Personally, seeing non-technical people using prompts for the first time -- and getting results that make sense -- is so incredible. Their eyes light up, they are surprised, and they want to keep playing with it. Amazing!

A sizeable part of the population just got access to an incredibly complex AI model and can play with it. What a collective experience.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by smoldesu on Thursday January 26, 2023 @02:12PM (#34537092)
Clarke's third law: "Any sufficiently advanced technology is indistinguishable from magic."

Most people don't understand that ChatGPT has no idea what they're talking about. It lacks it's own thought and heuristic patterns, and only gives you the most-likely response to your prompt. People don't know that though, so they think the Mechanical Turk is actually playing chess.

I mostly agree with the headline here. ChatGPT is hardly any more innovative than a Markov chain.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by hbrn on Thursday January 26, 2023 @02:30PM (#34537441)
> ChatGPT has no idea what they're talking about. It lacks it's own thought and heuristic patterns, and only gives you the most-likely response to your prompt

Funny thing is, same could be said about lots of people. Try listening to any political debate.

Which leads me to believe that the thing that's missing from AI is the same thing that we miss in those political debates: ability to explain and justify your own thought process.

As long as AI is a blackbox, we won't consider it to be a real intelligence.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by Lambdanaut on Thursday January 26, 2023 @02:38PM (#34537564)
There's the argument that any AI built on silicon is doomed to be analogous to the Chinese Room* because it is reducible to its parts.

It's interesting that I am also reducible to my atomic parts.

I may be a Chinese Room that has not yet experienced fully looking down at my self. I don't have enough self-recursive input yet to see through the illusion.

* https://en.wikipedia.org/wiki/Chinese_room#Chinese_room_thou...

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by hbrn on Thursday January 26, 2023 @02:47PM (#34537695)
An interesting way to escape being reducible to your atomic parts is quantum consciousness. Not that I believe in it, I just find the theory too beautiful to ignore.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by visarga on Thursday January 26, 2023 @02:55PM (#34537807)
A beauty that means to blind the mind and stop you looking further. Penrose is a dead end.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by hbrn on Thursday January 26, 2023 @04:04PM (#34538811)
You can definitely choose to interpret it that way, but you don't have to.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by hgsgm on Thursday January 26, 2023 @05:49PM (#34540014)
Searle didn't understand his own model. The Chinese Room is intelligent, even if the librarian isn't.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by kgarten on Thursday January 26, 2023 @07:00PM (#34540637)
in my opinion, the person/entity who wrote the books with the lookup tables is intelligent, the room is not …
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by fnordpiglet on Thursday January 26, 2023 @02:40PM (#34537590)
If it weren’t that remarkable, how come Siri, alexa, and whatever google calls it’s unused voice assistant are so useless? If it were not a substantial advance, why are chatbots so useless? The reality is, if Alexa had half the ability to “understand” as chatgpt it wouldn’t be nearly as frustrating. It’s not like Amazon, apple, and google haven’t dumped metric tons of investment into these things.

Y’all simply have lost the ability to be amazed.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by visarga on Thursday January 26, 2023 @02:59PM (#34537863)
It's more pragmatic than that. Google can afford to run a 300M weights model on every search or click, but can't afford a 175B generative model for the same thing. Generative models require a forward pass per token, all done serially, while embedding or classification models only need one pass no matter the length of the input.

It's too expensive for Google.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by lagrange77 on Thursday January 26, 2023 @04:48PM (#34539349)
Do you think this is also the reason, why Translate is still not as good as e.g. DeepL?
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by hgsgm on Thursday January 26, 2023 @05:51PM (#34540032)
How could it be too expensive? OpenAI charges a penny per response. That's cheap enough to wrap in a premium subscription or product price.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by Jcowell on Thursday January 26, 2023 @08:55PM (#34541528)
One of the best thing about google is that outta free. Charging a premium subscription for Google is no longer google but a different product all together
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by fnordpiglet on Thursday January 26, 2023 @10:06PM (#34541981)
In my example of a voice assistant the lifetime fees are present valued into the upfront costs. These cloud enabled devices incorporate a portfolio effect pricing model for service usage.

Google can charge advertisers more if they can somehow figure out how to extract more personal information from you and shove ads in your face posing as information.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by rafaelero on Thursday January 26, 2023 @02:29PM (#34537410)
> Most people don't understand that ChatGPT has no idea what they're talking about.

I wonder why people keep saying this. Is it some type of psychological defense to future AI displacement?

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by DrSiemer on Thursday January 26, 2023 @02:40PM (#34537601)
No, it is an accurate evaluation of what ChatGPT does. The model is not trying to explain something to you, it is trying to convince itself that what it is writing looks like it could have been written by a human.

There is no logical thinking involved. Output can appear to be real communication, but it's basically just a very advanced trick, that has some serious drawbacks.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by jmugan on Thursday January 26, 2023 @02:40PM (#34537595)
It's because ChatGPT's knowledge doesn't come through interaction with the world. Words mean things to us because they point to world interactions. I saw a demo where ChatGPT could create a VM. It could be trained to interact directly with a VM, send commands to an interpreter. In this case, it would understand the response of the VM, although it wouldn't understand the design behind the VM, because humans did that based on interaction with the physical world.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by rafaelero on Thursday January 26, 2023 @02:49PM (#34537726)
Based on this alleged limitation, can you list tasks that you think AI's won't ever be able to succeed at?
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by jmugan on Thursday January 26, 2023 @03:08PM (#34538003)
I think we will see AGI. But for the AI to be robust, it has to interact with the world, even if it is a simulated one. We need to build an AI that knows what a toddler knows before we can build one that understands wikipedia.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by rafaelero on Thursday January 26, 2023 @03:36PM (#34538416)
Human text does interact with the real world, so I don't see the limitation. Adding more modalities (vision, sound, etc.) probably will increase performance, and I think this is where we are heading, but it's silly to say that any one of these modalities are not grounded in reality. It's like saying humans can't understand reality because we can't see infrared rays. I mean, yeah?, but it's not the only way of making sense of reality.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by jmugan on Thursday January 26, 2023 @05:25PM (#34539765)
Language is a representation medium for the world, it isn't the world itself. When we talk, we only say what can't be inferred because we assume the listener has a basic understanding of the dynamics of the world (e.g., if I push a table the things on it will also move). Having an AI watch youtube and enabling it to act out what it sees in simulation would give it that grounding. We are heading that direction. So, I agree ChatGPT is awesome. I don't believe it understands what it is saying, but it can if it trains by acting out what it sees on Youtube.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by boto3 on Thursday January 26, 2023 @02:50PM (#34537742)
It did, actually. The model was trained with multiple rounds of reinforcement learning where human judges provided the feedback: first with full answers, and then with ranking of answers as most relevant.

So the model in production is probably frozen, but before that it went through multiple rounds of interaction with the world.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by jmugan on Thursday January 26, 2023 @03:07PM (#34537985)
The reinforcement learning was on giving the right answer, not on interacting with the world. But there is movement in the right direction with https://ai.googleblog.com/2022/12/rt-1-robotics-transformer-... and other RL stuff. (RT-1 isn't RL but there is other related stuff that is)
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by boto3 on Thursday January 26, 2023 @03:57PM (#34538726)
Oh, you meant interaction as a joint training with images, actions, feedback etc. That would be the next generation I guess.

I am simply thinking of interaction here as similar to learning a language in a classroom. First the teacher provides sample questions/answers, then the teacher asks the students to come up with answers themselves, and tell them which one is better. The end result here is I think ChatGPT is quite good at answering questions and can pass as a human, especially if it's augmented with a fact database, so obviously wrong answers can be pruned.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by Turing_Machine on Thursday January 26, 2023 @04:59PM (#34539499)
> It's because ChatGPT's knowledge doesn't come through interaction with the world.

We don't interact with the world directly, either. All we have are signals mediated through our nervous system.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by jmugan on Thursday January 26, 2023 @05:04PM (#34539552)
Sure, but what happens when you touch a hot stove?
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by Turing_Machine on Thursday January 26, 2023 @05:32PM (#34539836)
The built-in weights in your brain are modified to discourage you from doing that again?
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by jmugan on Thursday January 26, 2023 @05:48PM (#34539998)
Right. Those are tied via evolution to the dynamics of the physical world. We can simulate the physical world and learn from that, but there needs to be a there there. Language assumes the listener already has that understanding.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by jmugan on Thursday January 26, 2023 @05:49PM (#34540011)
Sorry, I meant to say that it hurts. That's the dynamics of the world I'm talking about.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by fwsgonzo on Thursday January 26, 2023 @02:36PM (#34537524)
It definitely could be - it's on every thread here and on reddit.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by hprotagonist on Thursday January 26, 2023 @02:30PM (#34537434)
maybe, but it's also just the fact of the matter.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by rafaelero on Thursday January 26, 2023 @02:35PM (#34537505)
That's not even a testable claim. What does it even mean to "not know what it is talking about"? If OP tried to operationalize its beliefs by making some prediction, like listing tasks that AI will never be able to reproduce because it "doesn't know what it is talking about", then that would be a discussion.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by hprotagonist on Wednesday February 01, 2023 @12:07PM (#34613626)
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by visarga on Thursday January 26, 2023 @02:13PM (#34537118)
> and only gives you the most-likely response to your prompt

That "only" carries a lot of weight on its back.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by oxfordmale on Thursday January 26, 2023 @02:19PM (#34537232)
The most likely response is not necessarily an accurate response. It claims dogs do not give birth to babies. That is indeed the most likely response as babies are mostly, but not exclusively, associated with humans. However, a five year old child would understand me if I told them my dog had babies.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by fnordpiglet on Thursday January 26, 2023 @02:37PM (#34537536)
Have you used chatgpt much?

I asked your prompt:

Do dogs give birth to babies?

> Dogs give birth to puppies.

I had a long discussion about the expansion do space and the Hubble constant, and tried to trick it in a variety of ways. It was correct in everything even though it wasn’t able to go into proofs and likewise. I then tried to convince it that the universe began as a small potato and wasn’t able to do it without directly telling it to assume it is and to create fiction off that concept. It was then able to make a credible theory of what the universe would be like if it started as a small potato, even going into how the texture of the potato would have caused variations in the density of mass, etc. Even then it would find ways to sneak in this isn’t true and that the universe was never a small potato. This is remarkable no matter the actual reality of what’s going on behind the scenes.

I would also note that human thought is just a bunch of gradient functions in the electrochemical space, and thus also follows a most likely path.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by oxfordmale on Thursday January 26, 2023 @04:25PM (#34539057)
Yes, I have tried ChatGPT, and it told me dogs do not give birth to babies. However, ChatGPT doesn't provide deterministic answers and improves responses over time. Many of the internet examples no longer work.

I asked, "If a woman gives birth to a baby in 9 months, how many months does a dog take to give birth to a baby."

It answered, "Dogs do not give birth to babies."

Currently, it answers, "On average, a dog's gestation period (the amount of time it takes for a dog to give birth to puppies) is approximately nine weeks or 63 days."

It still doesn't understand pregnancy is a concurrent event (women can be pregnant at the same time)

Q: If a woman takes nine months to give birth to a baby, how long does it take a billion women to give birth to one baby?

A: It would take approximately 2.7 million years for a billion women to give birth to one baby.

I do love ChatGPT to help me write certain documents. For example, for a performance review, you can give it bullet points and ask to write a review. After correcting it with Grammarly, it is usually better than what I would have written.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by TchoBeer on Thursday January 26, 2023 @05:38PM (#34539897)
>I asked, "If a woman gives birth to a baby in 9 months, how many months does a dog take to give birth to a baby." It answered, "Dogs do not give birth to babies."

to be fair, I'd probably say the same thing lol. "Baby" in this context was already primed to mean "human child."

>Q: If a woman takes nine months to give birth to a baby, how long does it take a billion women to give birth to one baby?

>A: It would take approximately 2.7 million years for a billion women to give birth to one baby.

I asked the same and it answered "It would still take 9 months for a billion women to give birth to one baby, as the length of time it takes for a woman to give birth is not affected by the number of women giving birth simultaneously."

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by hgsgm on Thursday January 26, 2023 @05:56PM (#34540065)
> For example, for a performance review, you can give it bullet points and ask to write a review.

Please don't do this. It obfuscates what you are trying to communicate. You are harming your communication and your relationship.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by dns_snek on Friday January 27, 2023 @05:55AM (#34544693)
Still not as bad as it's going to be. I'd wager that within a year someone's going to write a script that compares your GitHub contributions against your peers, feeds all of your Slack/email conversations into ChatGPT for professionalism, tone, and helpfulness analysis and feeds it back into ChatGPT to produce a completely automated "performance review".
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by ryandvm on Friday January 27, 2023 @09:39AM (#34546519)
That's fine because within a year my and my peers are going to let ChatGPT write all our code and handle all our code review requests. It's going to be AI all the way down.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by oxfordmale on Friday January 27, 2023 @02:25PM (#34550545)
I am after a rewrite in more elegant language than I can produce myself. I do not auto-accept the review; it is a good starting point for me to tweak. Any written performance review is followed up by a face-to-face meeting to clarify anything written down. ChatGPT is a tool, not magic, however, I find it helps with writers block.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by smoldesu on Thursday January 26, 2023 @02:17PM (#34537191)
As far as most people can be fooled by coherent-looking text.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by helloplanets on Thursday January 26, 2023 @02:25PM (#34537334)
Hold on, at least for me, 90% of the stuff is not just coherent-looking, but coherent. I do think that when it's wrong, it should give a heads up about not being certain about a subject. It's certainly tweaked to sound almost overconfident about every subject, which gives off a bullshitty vibe when the details of the explanation are wrong.

What sorts of subjects have you been trying it out with?

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by trefoiled on Thursday January 26, 2023 @02:34PM (#34537486)
I think part of the point here is that ChatGPT has no clue what you might mean here by "certainty" or being "wrong," and the fact that people have the impression that it could have an idea of those concepts is indicative of people's poor understanding of what ChatGPT really is
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by visarga on Thursday January 26, 2023 @02:39PM (#34537583)
Are you talking about the same people who couldn't agree about a vaccine? How is logical consistency going with humans?

Not to mention that a mere 700 years ago we were dying of bubonic plague and with all our general intelligence could not muster up the germ theory of disease. Not even to save our lives, you see, we are not generally general in our intelligence either, and make tons of mistakes.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by helloplanets on Thursday January 26, 2023 @03:35PM (#34538397)
Could you elaborate on this?

It definitely is able to decipher whether it has knowledge about the future, or some specific political events. This is obviously a pretty straightforward bonus layer on top of the model itself, but couldn't there be an extrapolation of that system where it's not binary, but rather a range between 0 and 1? I'd imagine this wouldn't be the model itself doing the crunching of the previous tokens here, at least not the same instance of it, as it could be stuck in whatever character or loop of reasoning it has going on at the moment.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by fnordpiglet on Thursday January 26, 2023 @02:50PM (#34537737)
I’ve had pretty complex discussions about Buddhism, physics, and other subjects and it was generally erudite and accurate. In its current unrefined form it’s more useful than google is at providing understanding on most subjects, especially because it will attempt to answer my direct questions rather than providing documents with terms in them. In fact I think it’s probably one of the most useful tools I’ve ever used.

Has it given me wrong information? Absolutely. But it’s always been pretty obviously wrong, and I often use it to introduce me to a subject then follow up with google to verify details. I further fully expect this to improve.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by helloplanets on Thursday January 26, 2023 @03:46PM (#34538576)
Yes, I've been going back and forth between Google and ChatGPT quite a bit lately. Sort of using Google as the verification step, after getting deeper into a subject.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by fnordpiglet on Thursday January 26, 2023 @04:36PM (#34539177)
For the haters, it seems incredibly likely that an IR system like google will be augmenting LLM and semantic reasoning systems to form a solution to the problems folks point. The problems chatgpt suffer from are solved and are complementary.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by abm53 on Thursday January 26, 2023 @02:16PM (#34537179)
> thought and heuristic patterns

I’m not sure these are well-defined enough terms to make this claim.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by smoldesu on Thursday January 26, 2023 @02:19PM (#34537216)
Heuristics is the ability to recognize multiple options and reason which one is correct. ChatGPT (and all GPT models for that matter) have a single context that doesn't get compared to other possible generations. You can back-propegate and see which tokens influenced the output most, but there is no evidence that AI can conceive multiple hypothetical responses and select the most-preferred one. At least not ChatGPT.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by visarga on Thursday January 26, 2023 @03:01PM (#34537901)
Have you heard of "beam search"? The model keeps a number of optional lines of text without being greedy and picking only the most probable token every time. You can also sample with T>0 multiple times to get an ensemble of answers that can be used as input for another AI operation. You just need to do multiple interactions with the model.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by fsloth on Thursday January 26, 2023 @02:26PM (#34537352)
"hardly any more innovative than a Markov chain."

Innovation is not invention. Innovation includes combining pedestrian concepts in novel ways. ChatGPT has the best ux and richest corpus of any markov chain I've ever used. It fits my bill of innovation combining sevaral things into a dang appealing product.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by visarga on Thursday January 26, 2023 @03:02PM (#34537917)
Whatever they want to call it, it is here and it works.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by buescher on Thursday January 26, 2023 @02:25PM (#34537341)
Except that the Mechanical Turk is actually playing chess - there is no hidden chess master.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by canjobear on Thursday January 26, 2023 @02:16PM (#34537180)
Responses like this are highly predictable in response to prompts about ChatGPT.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by lostmsu on Thursday January 26, 2023 @02:24PM (#34537325)
The question is are we any more innovative than Markov chains.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by Swizec on Thursday January 26, 2023 @02:37PM (#34537534)
> ChatGPT is hardly any more innovative than a Markov chain

A smartphone is hardly any more innovative than the ENIAC. And yet ...

Do not underestimate the power of making tech useful and accessible. The lightbulb means nothing without the power grid after all.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by cwillu on Thursday January 26, 2023 @02:48PM (#34537707)
https://astralcodexten.substack.com/p/janus-simulators

    But the essay brings up another connotation: to simulate
    is to pretend to be something. A simulator wears many masks.
    If you ask GPT to complete a romance novel, it will simulate
    a romance author and try to write the text the way they
    would. Character.AI lets you simulate people directly,
    asking GPT to pretend to be George Washington or Darth Vader.
[…]

    This answer is exactly as fake as the last answer where it
    said it liked me, or the Darth Vader answer where it says it
    wants to destroy me with the power of the Dark Side. It’s
    just simulating a fake character who happens to correspond
    well to its real identity.
[…]

    The whole point of the shoggoth analogy is that GPT is
    supposed to be very different from humans. But however
    different the details, there are deep structural
    similarities. We’re both prediction engines fine-tuned
    with RHLF.

    And when I start thinking along these lines, I notice that
    psychologists since at least Freud, and spiritual traditions
    since at least the Buddha, have accused us of simulating a
    character. Some people call it the ego. Other people call
    it the self.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by turtleyacht on Thursday January 26, 2023 @04:05PM (#34538837)
What is the original code you had before your programmers were born?

The DM lowers the screen and looks at the group.

"That concludes this campaign. Next week, we'll start a new one."

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by cwillu on Thursday January 26, 2023 @07:04PM (#34540660)
Noteworthy that the brief openai outage coincided with this.

“…and then ChatGPT woke up and we had to put a bullet in the server. Service will be restored shortly.”

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by idopmstuff on Thursday January 26, 2023 @02:15PM (#34537157)
I don't think he's underestimating the UX/design piece, I think it's just outside of the scope of what he's talking about. It seems pretty clear to me that his statement is talking about the underlying AI technology (which makes sense given his background).

And in any case, I wouldn't call anything about the UX or design here innovative - it's obviously a huge improvement in terms of usability, but a chat UI is a pretty well-established thing.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by cl42 on Thursday January 26, 2023 @02:24PM (#34537329)
I think the idea of a "prompt" is actually pretty cool. I never saw that framing prior to GPT-3 and I think it reframes the entire idea behind what a model does and how you interact with it.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by Waterluvian on Thursday January 26, 2023 @02:17PM (#34537184)
Then what left is there to explain why this is so much different from the well-established stuff?
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by hnfong on Thursday January 26, 2023 @02:20PM (#34537255)
"The implementation is left as an exercise to the reader."
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by Waterluvian on Thursday January 26, 2023 @02:26PM (#34537356)
It’s easy. Just draw the rest of the owl!
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by visarga on Thursday January 26, 2023 @03:04PM (#34537937)
Nobody talks about the dataset. Yes, the model was not innovative. But why hasn't anyone equaled OpenAI yet? Maybe they innovated on data engineering.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by random3 on Thursday January 26, 2023 @02:25PM (#34537346)
Researchers point of view is based on their area of research and that's fair and expected.

Yann LeCun compares ChatGPT in the context of the related research. Imagine a ChatGPT equivalent that memorizes many questions and does a brute force strategy for an answer. It may "look" magic, but there's nothing magic about it. We all accepted that this is the case with Blue Gene - https://en.wikipedia.org/wiki/Deep_Blue_(chess_computer)

What's different here?

Productization and usability are difference concerns here and Yann LeCun is not a usability researcher. Granted, that doesn't mean usability/accessibility doesn't impact research outcomes.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by zozbot234 on Thursday January 26, 2023 @02:35PM (#34537506)
But ChatGPT does not use brute force search to look for an answer. It interpolates among the answers in its training set. I.e. in Yann LeCun's analogy of a cake, interpolation or unsupervised learning is the cake; direct feedback on each single data point or supervised learning is the icing on the cake; and general feedback of the "how I am doing" sort or reinforcement learning is the cherry on top. Now Yann LeCun is just saying that the cake is a lie, and leaving it at that. I don't think this is a helpful understanding.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by visarga on Thursday January 26, 2023 @03:06PM (#34537973)
He is saying "chatGPT is mostly cake and cake wasn't invented here".
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by cl42 on Thursday January 26, 2023 @02:30PM (#34537432)
OK, I'll defend the research, too.

OpenAI's really interesting approach to GPT was to scale the size of the underlying neural network. They noticed that the performance of an LLM kept improving as the size of the network grew so they said, "Screw it, how about if we make it have 100+ billion parameters?"

Turns out they were right.

From a research perspective, I'd say this was a big risk and it turned out they were right -- bigger network = better performance.

Sure, it's not as beautiful as inventing a fundamentally new algorithm or approach to deep learning but it worked. Credit where it's due -- scaling training infrastructure + building a model that big was hard...

It's like saying SLS or Falcon Heavy are "just" bigger rockets. Sure, but that's still super hard, risky, and fundamentally new.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by belval on Thursday January 26, 2023 @02:38PM (#34537554)
> OpenAI's really interesting approach to GPT

That's the issue though, Yann LeCun is specifically referring to ChatGPT as the standalone model, not the GPT family since a lot of models at Meta, Google, DeepMind are based on a similar approach. His point is that ChatGPT is a cosmetic additional training with prompt with a nice interface, but not a fundamentally different model than stuff we've have had for +2-3 years at this point.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by cl42 on Thursday January 26, 2023 @03:07PM (#34537988)
ChatGPT is build on GPT-3. GPT-3 was a big NLP development. The paper has 7000+ citations: https://arxiv.org/abs/2005.14165 It was a big deal in the NLP space.

It wasn't a 'cosmetic' improvement over existing NLP approaches.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by belval on Thursday January 26, 2023 @03:15PM (#34538101)
Respectfully I don't think you read my comment. GPT3 != ChatGPT. ChatGPT is built on GPT-3 and is not breaking new ground. GPT3 is 3 years old and was breaking new ground in 2020 but Meta/Google/DeepMind all have LLM of their own which could be turned into a Chat-Something.

That's the point LeCunn is making. He's not out there negating that the paper you linked was ground-breaking, he's saying that converting that model into ChatGPT was not ground-breaking from an academic standpoint.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by cl42 on Thursday January 26, 2023 @03:20PM (#34538180)
@belval -- sorry, can't reply directly. I understand what you're saying -- fair enough! I appreciate the clarification.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by scifibestfi on Thursday January 26, 2023 @02:23PM (#34537298)
And/or underestimate the importance of shipping.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by danenania on Thursday January 26, 2023 @02:37PM (#34537550)
This guy seems to go beyond underestimating it. The appearance is he has straight up contempt for anyone who would be so vulgar as to turn high-minded academic AI research into a useful and compelling product.

OpenAI has a massive advantage if people with this mindset are running the AI divisions at their big tech competitors.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by cl42 on Thursday January 26, 2023 @02:25PM (#34537344)
100%. I think Hugging Face is awesome here as well.

The # of times I've tried to clone a GitHub repo and run a model on my own only to get 50 errors I can't debug.... :)

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by MisterPea on Thursday January 26, 2023 @02:25PM (#34537348)
This is the bigger thing over UX/design.

Someone on twitter likened this to when Xerox invented the mouse, but Apple/Microsoft shipped it with their PCs

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by version_five on Thursday January 26, 2023 @02:26PM (#34537351)
chatGPT is innovative in the way the iPhone was innovative. That, of course can be market winning, but it's dangerous, especially for businesses and investors, to get caught up in hype that there is some new "AI" out there that can do things we previously couldn't
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by Gigachad on Thursday January 26, 2023 @06:32PM (#34540375)
Can you point to any product where I could do what chatgpt does before chatgpt? Feels like we just leapfrogged from dumb markov chains to something incredible that provides real value.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by sandworm101 on Thursday January 26, 2023 @02:16PM (#34537167)
>> using prompts for the first time -- and getting results that make sense -- is so incredible.

Years ago it was possible to insert prompts into Google and get results that made sense, results that were meaningfully tied to the information you requested. The young people today who missed that time think it magic.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by cl42 on Thursday January 26, 2023 @02:20PM (#34537237)
I think this is fundamentally different because you can play with it, and people want to play with it.

Google isn't a playground -- it's much more utilitarian. You go there when you need to find a doctor in Seattle, or to research politics in Poland, or something. You get results which are great, but you don't need to stick around.

GPT-3 and ChatGPT allow you to spend time playing with the system and having fun with it[1]. I think this is what makes it so viral and interesting to people.

[1] https://10millionsteps.com/ux-ai-gpt-3-click

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by lostmsu on Thursday January 26, 2023 @02:21PM (#34537264)
Yes, but the kind of responses were different.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by visarga on Thursday January 26, 2023 @03:08PM (#34537998)
That wonder might not be lost forever. Can't we have access to an old index and cached web pages?
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by ryanSrich on Thursday January 26, 2023 @02:06PM (#34537004)
> It's amazing how many researchers underestimate the importance of UX and design.

Yes. Also the fact that ChatGPT's UX and design leaves much to be desired. They could add/improve the product in so many obvious ways. I wonder if they either 1.) don't care, or 2.) have plans to, but are holding off until they take on a subscription fee.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by cl42 on Thursday January 26, 2023 @02:08PM (#34537035)
What would you change to the UX and design to make it even better?
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by liuliu on Thursday January 26, 2023 @02:19PM (#34537230)
Add "calculator" in a form of Python interpreter to give alternative result. Providing interactive graphical interface for some of the charts they shown. Connecting outside sites for booking, pictures, user comments.

To some degree, WolframAlpha putting more efforts into the UX than ChatGPT.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by cl42 on Thursday January 26, 2023 @02:22PM (#34537282)
I was going to ask you re: WolframAlpha. I think your suggestions are great ideas.

Citing sources, providing links, and even possibly visuals. That's also how I think ChatGPT could really attack the search space.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by baxtr on Thursday January 26, 2023 @02:16PM (#34537181)
The slow spell out is really annoying. If they want to rate limit fine. But give me the answer faster and then lock queries for a few seconds instead.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by cloudking on Thursday January 26, 2023 @02:19PM (#34537221)
The spell out is a limitation of the way the model works, it predicts the text word by word.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by baxtr on Thursday January 26, 2023 @02:20PM (#34537243)
Yes I get that. But just speed the god damn thing up.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by visarga on Thursday January 26, 2023 @03:10PM (#34538030)
The god damn thing has to do 175B multiply and add operations for each one of your words.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by baxtr on Thursday January 26, 2023 @03:29PM (#34538319)
Who cares? OP was asking about UX improvements. Maybe I missed the part, but IMHO no one was asking only for UX improvements that come cheap.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by visarga on Friday January 27, 2023 @02:24AM (#34543535)
They said "just speed the god damn thing up" - this is related to model inference speed. If your answer is one page long (about 2000 tokens), be prepared to wait for a whole minute.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by ryanSrich on Monday January 30, 2023 @04:00PM (#34586303)
I’d much prefer the type be hidden until it’s ready entirely.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by slaymaker1907 on Thursday January 26, 2023 @02:11PM (#34537087)
It seems to log me out every day unlike pretty much every other web service besides banks.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by elwell on Thursday January 26, 2023 @02:13PM (#34537122)
Probably related to a protection against abuse as a proxy or scraping.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by visarga on Thursday January 26, 2023 @02:14PM (#34537136)
The login captcha somehow irks me. It's a bot.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by cma on Thursday January 26, 2023 @02:14PM (#34537131)
Entering and editing text on a small touchscreen with it is a pretty bad experience. The keyboard covers the text, linebreaks can't be entered without pasting, viewing the output requires closing the keyboard. None of these are problems to the same degree in say Discord's mobile/tablet UI.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by visarga on Thursday January 26, 2023 @03:12PM (#34538054)
It has some issues when you edit a prompt that has very long text, it will snap to the end when you type losing focus.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by Turing_Machine on Thursday January 26, 2023 @05:30PM (#34539818)
Isn't there an API?
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by zwieback on Thursday January 26, 2023 @02:23PM (#34537302)
Agreed. He did say that it was "well put together and nicely done", though.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by PartiallyTyped on Thursday January 26, 2023 @02:11PM (#34537085)
It's quite fascinating how information retrieval and search engines have evolved..

From trying to teach people how to google via "I am feeling lucky", to using language models for ranking, to building LLMs to better adapt to user queries and move beyond keywords, to having arguably useful chatbots that can synthesize responses.

I am curious to see what the future holds. Maybe an AI that can anticipate our queries?

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by visarga on Thursday January 26, 2023 @02:18PM (#34537206)
I also saw the "LLM as database" metaphor. Up until 2020 we had databases in the backend, UI in front, now we can have LLMs in the backend.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by PartiallyTyped on Thursday January 26, 2023 @02:55PM (#34537806)
For the uninitiated https://news.ycombinator.com/item?id=34503418

Maybe, eventually, LLMs can be used to synthesize and cache certain APIs ... who knows :D

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by Turing_Machine on Thursday January 26, 2023 @04:47PM (#34539327)
Indeed. This reminds me of the people who were unimpressed by the iPod because there were other MP3 players already on the market.

Sometimes "making the first X that doesn't suck" is a lot more important than "making the first X".