arimorcos

Author here. Indeed, our work is closely related to dropout. However, as we discuss in the paper (https://arxiv.org/abs/1803.06959), dropout doesn't really encourage the network to be robust to deletion generally; it only encourages this robustness up until the dropout fraction used in training.

So, for example, if you train the network with a 50% dropout rate, dropout will encourage the network to be robust to dropping 50% of the units, but the network could completely fail once 51% of the units are deleted and the training objective would be perfectly happy. As a result, dropout doesn't change the shapes of ablation curves, but rather simply horizontally scales them such that the left edge of the curves is at the dropout fraction rather than 0. In contrast, we found that batch normalization actually pulls the curves up and to the right, rather than simply scaling them, though we only have hints as to why that is.

Hope that was helpful!

You can't only look at demand. You also have to look at supply. Most machine learning engineers have a masters at least (many have a PhD), and almost all 'scientist' positions require a PhD. Of course there are fewer deep learning jobs available than front-end developers. That doesn't mean they're not highly coveted.

As an example, one of Fei-Fei Lee's recent grad students got multiple job offers upon graduating, one for more than a million dollars a year: http://www.nytimes.com/2016/03/26/technology/the-race-is-on-...

Not according to Demis Hassabis: https://twitter.com/demishassabis/status/739832323160563712

For a book published in late 2014, it's very strange that this book doesn't seem to mention any of the state of the art components in neural networks, including:

   - Convolutional neural networks 
   - LSTMs
   - GRUs 
   - Autoencoders 
   - Attentional mechanisms 
   - Variations on SGD (Adagrad, Adadelta, RMSProp, ADAM)

Generative adversarial networks can do this much much better. Check out this blog post:

http://www.foldl.me/2015/conditional-gans-face-generation/

and the corresponding paper:

http://www.foldl.me/uploads/2015/conditional-gans-face-gener...