This is a common criticism, but I think it is superficial. We don't use reason to derive solutions, we use reason to prune intuition and refine. I am sure that the next phase of these models will be setting them up so they work like importance sampling in MCMC, where the neural net spits out a small set of proposals and then a scoring function accepts or rejects based on learned heuristics with some simplicity reward (does it run, how fast is it, how simple is the code, etc). What we think of as reason is usually these heuristics used post hoc to check validity of proposals. I don't know how I come up with solutions to problems in the same way I don't know how I can catch a ball.