* initialize weights to small random values (otherwise it will quickly get stuck in a non-desirable local optimum)
* you need to decrease the learning rate of the stochastic descent according to a 1/(n+n0) or similar schedule, otherwise there's a risk of the whole thing swinging back and forth
* it often helps to normalize the input values so that they have a standard deviation of 1 and an expectation of zero. Or at least it's not good to have input values differ too much in scale.
* if you're training something that should give yes/no answers, you don't really want to minimize the mean square error, but a logistic loss.
Some of these parts are usually built into your NN toolkit, so you don't have to worry about them. Or they're not (e.g., choosing a good learning rate), in which case you're just screwed if you don't know what happens and why.
But you're right, mathematically ANNs are quite simple (and, as one has found out, don't really do similar things to what actual neurons do).
I was always turned out because it was always explained in some magical manner...
http://en.wikipedia.org/wiki/Logistic_regression
which has been around since the 1940s... (multilayer neural networks correspond to hierarchical logistic regression--just plug them together). this should be in any reasonable stats book.
if you mean mathematically sound in terms of learning, etc, then chris bishops "neural networks and pattern recognition" is pretty good, full of sage advice and justification.
Neural network learning algorithms are given (in,out) pairs and try to find A and B to minimize the mean square error, for example by using stochastic gradient descent.