News for hackers. Stuff that matters.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by random3 on Thursday January 26, 2023 @02:25PM (#34537346)
Researchers point of view is based on their area of research and that's fair and expected.

Yann LeCun compares ChatGPT in the context of the related research. Imagine a ChatGPT equivalent that memorizes many questions and does a brute force strategy for an answer. It may "look" magic, but there's nothing magic about it. We all accepted that this is the case with Blue Gene - https://en.wikipedia.org/wiki/Deep_Blue_(chess_computer)

What's different here?

Productization and usability are difference concerns here and Yann LeCun is not a usability researcher. Granted, that doesn't mean usability/accessibility doesn't impact research outcomes.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by zozbot234 on Thursday January 26, 2023 @02:35PM (#34537506)
But ChatGPT does not use brute force search to look for an answer. It interpolates among the answers in its training set. I.e. in Yann LeCun's analogy of a cake, interpolation or unsupervised learning is the cake; direct feedback on each single data point or supervised learning is the icing on the cake; and general feedback of the "how I am doing" sort or reinforcement learning is the cherry on top. Now Yann LeCun is just saying that the cake is a lie, and leaving it at that. I don't think this is a helpful understanding.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by visarga on Thursday January 26, 2023 @03:06PM (#34537973)
He is saying "chatGPT is mostly cake and cake wasn't invented here".
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by cl42 on Thursday January 26, 2023 @02:30PM (#34537432)
OK, I'll defend the research, too.

OpenAI's really interesting approach to GPT was to scale the size of the underlying neural network. They noticed that the performance of an LLM kept improving as the size of the network grew so they said, "Screw it, how about if we make it have 100+ billion parameters?"

Turns out they were right.

From a research perspective, I'd say this was a big risk and it turned out they were right -- bigger network = better performance.

Sure, it's not as beautiful as inventing a fundamentally new algorithm or approach to deep learning but it worked. Credit where it's due -- scaling training infrastructure + building a model that big was hard...

It's like saying SLS or Falcon Heavy are "just" bigger rockets. Sure, but that's still super hard, risky, and fundamentally new.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by belval on Thursday January 26, 2023 @02:38PM (#34537554)
> OpenAI's really interesting approach to GPT

That's the issue though, Yann LeCun is specifically referring to ChatGPT as the standalone model, not the GPT family since a lot of models at Meta, Google, DeepMind are based on a similar approach. His point is that ChatGPT is a cosmetic additional training with prompt with a nice interface, but not a fundamentally different model than stuff we've have had for +2-3 years at this point.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by cl42 on Thursday January 26, 2023 @03:07PM (#34537988)
ChatGPT is build on GPT-3. GPT-3 was a big NLP development. The paper has 7000+ citations: https://arxiv.org/abs/2005.14165 It was a big deal in the NLP space.

It wasn't a 'cosmetic' improvement over existing NLP approaches.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by belval on Thursday January 26, 2023 @03:15PM (#34538101)
Respectfully I don't think you read my comment. GPT3 != ChatGPT. ChatGPT is built on GPT-3 and is not breaking new ground. GPT3 is 3 years old and was breaking new ground in 2020 but Meta/Google/DeepMind all have LLM of their own which could be turned into a Chat-Something.

That's the point LeCunn is making. He's not out there negating that the paper you linked was ground-breaking, he's saying that converting that model into ChatGPT was not ground-breaking from an academic standpoint.

Re:Yann LeCun: ChatGPT is 'not particularly innovative' by cl42 on Thursday January 26, 2023 @03:20PM (#34538180)
@belval -- sorry, can't reply directly. I understand what you're saying -- fair enough! I appreciate the clarification.