Re:John Hopfield and Geoff Hinton Win Physics Nobel Prize [pdf]
by miven on Tuesday October 08, 2024 @07:44AM
(#41776250)
Residual connections are also worth mentioning as an extremely ubiquitous adaptation, one will be hard-pressed to find a modern architecture that doesn't use those at least to some extent, to the point where the original Resnet paper sits at over 200k citations according to google scholar[1].
[1] https://scholar.google.com/citations?view_op=view_citation&h...
There were also plenty of "hacks" involved to make the networks scale such as dropout regularization, batch normalization, semi-linear activation functions (e.g. ReLU) and adaptive stochastic gradient descent methods.
The maths for basic NNs is really simple but the practice of them is really messy.