Most of the "dinosaurs" around here should know by now that you don't get paid to do what works; you get paid to do what the boss/customer wants. (And that's not always going to be exactly what they asked for, either.)

As long as you continue to throw money at me, I'll continue to chase your wild geese, hunt your snipes, fish for your red herrings, and joust with your wind turbines. Most of us here can learn fast enough to always stay one or two steps ahead of the boss on whatever topic they might latch on to. And that becomes genuine experience if it somehow manages to graduate past the business fad phase.

And even if you don't want to use SGD to fit your model, the deep learning machinery can be reused for other things like bayesian inference, eg PyMC3 and Edward are built on Theano/TensorFlow respectively.

There have even been papers showing differentiable trees integrated into the deep learning framework. There is work on how to input graphs.

Maybe this is just my own desire for unifying frameworks shining through, but joint training seems to have real benefits and feature learning has clearly had a large impact on reinforcement learning.

Maybe it's not the best tool for day to day data science work, but that tends towards what tools are accessible and good enough, rather than what is best.

CNN's are great for some image recognition/classification problems, but that's just the start of the conversation.