Gradient methods have certainly gotten most of the attention, but there is the field of neuroevolution[1] and OpenAI recently released a paper covering evolutionary strategies for neural networks: https://blog.openai.com/evolution-strategies/

[1] https://en.wikipedia.org/wiki/Neuroevolution