Re:Single cortical neurons as deep artificial neural networks
by shmageggy on Thursday August 12, 2021 @09:48AM
(#28156295)
No, because it's wrong. Threshold neurons are still differentiable almost everywhere, no different than Relus which are ubiquitous. They may not be very good activation functions but they don't prevent a network from learning.
[ Parent
| on HN ]
Re:Single cortical neurons as deep artificial neural networks
by sdenton4 on Thursday August 12, 2021 @10:29AM
(#28156782)
Sorry, but I think I disagree...
If I understand the PM neuron, it's outputs are boolean. This means there's no backprop signal, despite differentiability, since the outputs are constant (thus, gradient zero) in any neighborhood, which in turns zeros out any learning signal you would want to backprop through them. So you need a different learning strategy than backprop to use them.
(see also: the 'dead neuron' problem/phenomenon with ReLU activations.)
Re:Single cortical neurons as deep artificial neural networks
by orbifold on Thursday August 12, 2021 @01:48PM
(#28159278)
One can regularise the derivative (a delta distribution) in several ways (e.g. a triangle at zero) and that is good enough (even from a theoretical perspective) to find an approximate gradient. Experimentally it is then possible to train deep neural networks with such non-linearities.