Re:The hippocampus as a 'predictive map'
by orbifold on Tuesday October 03, 2017 @09:26AM
(#15392217)
Actually ReLU(x) = max(x,0) can be thought after a change of parameters as the firing rate response of a leaky integrate and fire neuron at least for a certain range of parameters. So it is not completely far fetched, but of course nature does not use rate coding exclusively.
[ Parent
| on HN ]
Yes. Though I think they stumbled on ReLU just from technical considerations and not taking nature into account?
That happens fairly often in engineering: looking around enough one can often find natural precedents for our artifices. But directly trying to copy nature did not lead to the plane or helicopter.
Wikipedia list this as the first paper involving ReLU in an ANN:
[1] R Hahnloser, R. Sarpeshkar, M A Mahowald, R. J. Douglas, H.S. Seung (2000). Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit.