Perhaps you could reduce the set of numpy functions used in your code to a minimal set (exp, sum, max, min, etc.) and then build fancier functions up from there. This affords you the speed and conciseness of using numpy arrays while limiting the abstractions that could obfuscate the inner workings of some of the fancier functions you might use (e.g. softmax).