Re:PyTorch – Tensors and Dynamic neural networks in Python
by smhx on Thursday January 19, 2017 @12:25AM
(#13433259)
the biggest hindrance to do this are language constructs that cannot be or are inconveniently expressed in the symbolic graph, such as python's if vs tf.if and for vs theano.scan, or conditioning on some python-code (not tensor operations). So to build an eagerly evaluating symbolic graph framework that is allowed to do arbitrary things would mean that you would (to an extent) reimplement the language you are working with.
Re:PyTorch – Tensors and Dynamic neural networks in Python
by liuliu on Thursday January 19, 2017 @01:04AM
(#13433451)
Let's assume Tensorflow has basic symbolic computation graph expressiveness. What you would do is to build a symbolic representation while executing your graph inline, your symbolic representation doesn't need to have any control structure, it is simpler than that. You execute while loop in Python as usual, and your symbolic representation won't have TF.While at all, it will simply be the execution you performed so far (matrix mul 5 times).
Once you have a reasonable symbolic computation graph library, you don't need to explicitly build a "tape" because the symbolic representation will record the order of execution and reverse AD even graph optimization (applying CSE etc) come naturally as well.
In fact, with DyNet or PyTorth, you still need to bookkeeping the graph you traversed (tape) because no one is doing forward AD. If that's the case, why not have a good library to do symbolic computation graph and build dynamic feature on top of it. (I am not saying Tensorflow is a good symbolic computation graph library to build upon just arguing that start with a define-compile-run library doesn't necessarily hinder your ability to support dynamic graphs).