If you want a nice, visual, explanation of what a neural network is actually doing, I found this blog post to be particularly helpful. [1] Although there is some math, the rest of the post gives a good intuitive understanding of what's going on without it.

[1] http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/