Actually, the minimal neural network that always outputs that string would basically have a weight vector equivalent to the ASCII coding of the string, so it would be equally easy to reason about compared to the Python equivalent.

You are comparing apples to oranges. The python line is a good representation of a part of a larger program, while the neural network you are talking about is not a good representation of any part of a larger neural network. You wont get that solution if you train it normally without knowing how to solve the problem yourself. Try setting up a network with 3 layers of nodes and 120 nodes in each layer and train it to output "Hello, world!" using a standard training method, you wont get something easily comprehensible from that.