In dropout, you select 50% (say) of the nodes in one layer of your network at random and force them to pass on no signal (neither positive nor negative). You then train the network to still return the result.
This is effectively training the network to be robust to failures (i.e. "harder to break").
While writing this, I've realized that we've both assumed that not only "networks which generalize better are harder to break" but also "networks which are harder to break generalize better". I think that's true, but worth being explicit.
(Still developing my intuition for machine learning, so I could be wrong...)
So, for example, if you train the network with a 50% dropout rate, dropout will encourage the network to be robust to dropping 50% of the units, but the network could completely fail once 51% of the units are deleted and the training objective would be perfectly happy. As a result, dropout doesn't change the shapes of ablation curves, but rather simply horizontally scales them such that the left edge of the curves is at the dropout fraction rather than 0. In contrast, we found that batch normalization actually pulls the curves up and to the right, rather than simply scaling them, though we only have hints as to why that is.
Hope that was helpful!
also the network could have weird results for dropout values around 20%-30% depending on how the robustness was 'learned'
it was 2004-2006, we used weight quantization as noise function and a modified levemberg-marquad for training, matlab was the hot stuff back then and by today standard everything is ancient, so there's that - the training alghorithm was changed to train toward a flatter area of the noise function, so it used the second derivative to hold gradient descent and overfitting.
I can answer specific questions if you want.
yep, that was my whole thesis dissertation. also you can actually include some confusion during the training itself by adding noise to the weights or the transfer function and the learning algorithm will find more stable solutions that are more robust on its own.
we used that to produce network that were more resilient to being quantized when moved to low power devices having only fixed point arithmetic available.