More Agents Is All You Need: LLMs performance scales with the number of agents

Re:More Agents Is All You Need: LLMs performance scales with... by samus on Sunday April 07, 2024 @10:09AM (#39960882)

The attention mechanism is in practice implemented using three linear layers. The matrix multiplication to average the output and to implement the masking is the only non-neuronal part of that computation, but it can be seen as an activation function.

Usually, linear perceptrons and ReLUs or GeLUs are used. Due to the enormous compute requirements to evaluate models of interesting size, other types of neuronal networks and activation functions have received very little attention (pun intended) so far.

[ Parent | on HN ]

Re:More Agents Is All You Need: LLMs performance scales with... by dartos on Sunday April 07, 2024 @11:13AM (#39961304)

Using ReLU instead of sigmoid is a significant departure with regards to how closely it models actual neurons.

Using non fully connected layers is as well. Our brains likely aren’t fully connected, but the connections that matter are made stronger through living life and learning.

If you squint, it’s kind of like training a dense series of linear layers, but that’s not what we’re doing anymore (for the better)

Comparing NNs to organic brains is an apples to oranges comparison, is what I’m saying.

[ Parent | on HN ]

Re:More Agents Is All You Need: LLMs performance scales with... by samus on Sunday April 07, 2024 @11:33AM (#39961437)

I agree that the biggest difference is the missing ability of an artificial neuronal network to adapt.

[ Parent | on HN ]

Re:More Agents Is All You Need: LLMs performance scales with... by ben_w on Sunday April 07, 2024 @04:22PM (#39963559)

Lack of adaption is mainly a feature, we choose not to train them in real-time and instead make available fixed models with repeatable behaviour. We could, if we wanted to, update the model weights continuously in response to feedback.

I think the biggest difference is that they need far more examples than we need, to learn anything.

[ Parent | on HN ]