News for hackers. Stuff that matters.
Re:The hippocampus as a 'predictive map' by eru on Tuesday October 03, 2017 @08:09AM (#15391820)
Current AI approaches don't have that much to do with the biological brain. Despite the name 'neuronal networks' that we have history to thank for, it's better think of them as just long compositions of functions with a lot of parameters one can tweak.

Historically researchers looked at activation functions that are at least somewhat justified by nature, like the sigmoid. More recently people have focused on the mathematical aspects and figured out that the rectified liner unit `ReLU(x) = max(x, 0)` works just as well if not better for most applications.

Maxout networks (https://arxiv.org/pdf/1302.4389.pdf) are another interesting development in that direction.

There's still lots we can learn from biology, of course; but the last decades rapid advances we saw from deep learning at most barely relate to real brains.

Re:The hippocampus as a 'predictive map' by mannigfaltig on Tuesday October 03, 2017 @10:06AM (#15392488)
> Despite the name 'neuronal networks' that we have history to thank for, it's better think of them as just long compositions of functions with a lot of parameters one can tweak.

Actually, the name is not so misleading because `ReLU(Wx + b)` is an approximation of the response of a large ensemble of integrate-and-fire neurons in terms of the firing rate [1] (it models populations of pyramidal neurons in the cortex). The ReLU activation function basically approximates a logarithmic activation function [2]. Obviously many kinds of computations are left out in this model, for example in pyramidal neurons, the integration of the incoming spikes is often sublinear rather than linear; the branches of the neurons that receive signals (dendrites) can themselves perform complex computations such as spatial clustering and logical operators basically sending small (dendritic) spikes forward and backward within an neuron [3]; short-term plasticity is left out entirely and so are countless types of interneurons which inhibit other neurons nearby and measure mean activity across populations of neurons etc.

[1] https://web.stanford.edu/group/brainsinsilicon/documents/Won...

[2] https://www.utc.fr/~bordesan/dokuwiki/_media/en/glorot10nips...

[3] https://neurophysics.ucsd.edu/courses/physics_171/annurev.ne...

Re:The hippocampus as a 'predictive map' by eru on Tuesday October 03, 2017 @10:13AM (#15392524)
Thanks! Yes, you can find that the ReLU does model some aspects---but if memory serves right people didn't switch from sigmoid to ReLU because they were interested in modelling the brain better, but mostly because they were looking for a structure that's simpler to train. Sigmoids notoriously smear out all your backpropagation 'juice' after only a few layers, don't they?

(I only mention ReLU because they are definitely simpler in mathematical structure than ye olde sigmoid. Max out is another interesting activation function that's first and foremost inspired by the math / pragmatics of drop-out training, and were any relations to biology more likely to be subsequent discoveries and not intentional design features.)

Re:The hippocampus as a 'predictive map' by orbifold on Tuesday October 03, 2017 @09:26AM (#15392217)
Actually ReLU(x) = max(x,0) can be thought after a change of parameters as the firing rate response of a leaky integrate and fire neuron at least for a certain range of parameters. So it is not completely far fetched, but of course nature does not use rate coding exclusively.
Re:The hippocampus as a 'predictive map' by eru on Tuesday October 03, 2017 @10:07AM (#15392494)
Yes. Though I think they stumbled on ReLU just from technical considerations and not taking nature into account?

That happens fairly often in engineering: looking around enough one can often find natural precedents for our artifices. But directly trying to copy nature did not lead to the plane or helicopter.

Re:The hippocampus as a 'predictive map' by zardo on Tuesday October 03, 2017 @10:20AM (#15392569)
Wikipedia list this as the first paper involving ReLU in an ANN:

[1] R Hahnloser, R. Sarpeshkar, M A Mahowald, R. J. Douglas, H.S. Seung (2000). Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit.

Re:The hippocampus as a 'predictive map' by derefr on Tuesday October 03, 2017 @04:34PM (#15395918)
Sure, but (our models of the functions of) neurotransmitters don't have that much to do with biology, either. For example, the local level of dopamine in a neuron is understood to be something like an encoding of a confidence interval associated with the computation, such that where synapses in the sensory (NMDA-excited) and model-predictive (AMPA-excited) regions of the brain come into contact, the dopamine in each region bubbles up as a bunch of coefficients that adjusts how comparisons between data and predictions shake out, temporarily biasing whether "branches" to error states (i.e. the activation of higher-layer neurons) are taken.

This works just as well whether you are modelling biological synapses with plasticity, or compositions of functions with training via backpropagation.

The only real difference is that NNs aren't usually "physically embodied" (i.e. embedded in a metric space with radiative signalling to non-connected but "local" neighbours) in the way they'd have to be to get the "area-effect excitation" effects of neurotransmission.