I think you should start working on your Math (Khan academy courses) and ML foundations (Andrew Ng's coursera course). Then Geoffrey Hinton's coursera course on Neural networks could be a gentle introduction to Neural networks, deep learning and their applications. Last but not least, do a small project on Deep learning or try out few kaggle competitions to deepen your understanding.

Links:

https://www.coursera.org/course/ml

https://www.khanacademy.org/math/linear-algebra

https://www.coursera.org/course/neuralnets

www.iro.umontreal.ca/~bengioy/papers/ftml.pdf

http://www.iro.umontreal.ca/~bengioy/dlbook/

http://techtalks.tv/talks/deep-learning/58122/

This is a timely post for me because I also want to study deep learning and have been researching the best way to do that without going back to school and getting a graduate degree in CS.

My tentative plan is to start with Coursera's Neural Network course and implement as many things as possible (probably in Python) along the way. By the end of 2015, I'd like to build a deep learning application from scratch that does something interesting.

While online resources will likely be my primary source of information, I'd love to have a study partner (or partners) to discuss concepts and bounce ideas off of. If OP or anyone else is interested, drop me a note at [email protected] and we'll figure out the best way to do that.

As a side note, I built a few a-life/AI visualizations in JavaScript over the past year, including a simple neural network which might interest anyone reading this: http://www.emergentmind.com/.

Metacademy (http://www.metacademy.org/) is a wonderful resource for learning anything machine learning. Especially if you don't know what you want to know, or what you need to learn first. I also really like this coursera course: https://www.coursera.org/course/neuralnets

deeplearning.net and the ufldl tutorial are excellent places to start. I've also perused this ebook recently and found it to be pretty solid in terms of giving mathematically solid but still intuitive. last but not least, absolutely do no whatsoever "do a small project on deep learning or try out [a] few kaggle competitions." instead, pick up a paper that interests you and implement the methods they describe therein.

edit: here's the ebook link neuralnetworksanddeeplearning.com