The "fastai" course is free, and does a really nice job walking you through building simple neural nets from the ground up:

https://github.com/fastai/fastai

What's going on here is the exact same thing, just much, much larger.