The problem is that if you build the graph as-you-go, dataflow graph optimizations cannot be done efficiently (some high level optimizations such as data layout optimization, automatic data / model parallelism etc.). Swift can do all these because the compiler can extract the graph out ahead of time.
However, I'd like to see some numbers on how more efficient it is to build a graph in advance, given that the lion's share of the computations will be in tensor math anyway (which can be heavily optimized, and is independent of the graph).