Re:The hippocampus as a 'predictive map'
by eru on Tuesday October 03, 2017 @10:13AM
(#15392524)
Thanks! Yes, you can find that the ReLU does model some aspects---but if memory serves right people didn't switch from sigmoid to ReLU because they were interested in modelling the brain better, but mostly because they were looking for a structure that's simpler to train. Sigmoids notoriously smear out all your backpropagation 'juice' after only a few layers, don't they?
(I only mention ReLU because they are definitely simpler in mathematical structure than ye olde sigmoid. Max out is another interesting activation function that's first and foremost inspired by the math / pragmatics of drop-out training, and were any relations to biology more likely to be subsequent discoveries and not intentional design features.)
(I only mention ReLU because they are definitely simpler in mathematical structure than ye olde sigmoid. Max out is another interesting activation function that's first and foremost inspired by the math / pragmatics of drop-out training, and were any relations to biology more likely to be subsequent discoveries and not intentional design features.)