News for hackers. Stuff that matters.

A comment on OpenAI, Google and Anthropic Are Struggling to Build More Advanced AI. Parent comment

Re:OpenAI, Google and Anthropic Are Struggling to Build More... by falcor84 on Thursday November 14, 2024 @01:40PM (#42139568)
> It’s like chaining up perceptrons hoping to get more expressive power for free.

Isn't that literally the cause of the success of deep learning? It's not quite "free", but as I understand it, the big breakthrough of AlexNet (and much of what came after) was that running a larger CNN on a larger dataset allowed the model to be so much more effective without any big changes in architecture.

[ Parent | on HN ]
Re:OpenAI, Google and Anthropic Are Struggling to Build More... by david2ndaccount on Thursday November 14, 2024 @02:06PM (#42139912)
Without a non-linear activation function, chaining perceptrons together is equivalent to one large perceptron.
[ Parent | on HN ]
Re:OpenAI, Google and Anthropic Are Struggling to Build More... by xanderlewis on Thursday November 14, 2024 @05:23PM (#42141849)
Yep. falcor84: you’re thinking of the so-called ‘multilayer perceptron’ which is basically an archaic name for a (densely connected?) neural network. I was referring to traditional perceptrons.
[ Parent | on HN ]
Re:OpenAI, Google and Anthropic Are Struggling to Build More... by falcor84 on Thursday November 14, 2024 @05:51PM (#42142074)
While ReLU is relatively new, AI researchers have been aware of the need for nonlinear activation functions and building multilayer perceptrons with them since the late 1960s, so I had assumed that's what you meant.
[ Parent | on HN ]
Re:OpenAI, Google and Anthropic Are Struggling to Build More... by xanderlewis on Thursday November 14, 2024 @06:31PM (#42142428)
It was a deliberately historical example.
[ Parent | on HN ]