News for hackers. Stuff that matters.
Re:PyTorch – Tensors and Dynamic neural networks in Python by liuliu on Wednesday January 18, 2017 @10:50PM (#13432843)
You can build both the symbolic computation graph and do the computation at the time when defining the network architecture, thus, gaining the ability to be "dynamic" and also supporting advanced features with the symbolic representation that you built on the side.

In fact, with DyNet or PyTorth, you still need to bookkeeping the graph you traversed (tape) because no one is doing forward AD. If that's the case, why not have a good library to do symbolic computation graph and build dynamic feature on top of it. (I am not saying Tensorflow is a good symbolic computation graph library to build upon just arguing that start with a define-compile-run library doesn't necessarily hinder your ability to support dynamic graphs).

Re:PyTorch – Tensors and Dynamic neural networks in Python by smhx on Thursday January 19, 2017 @12:25AM (#13433259)
the biggest hindrance to do this are language constructs that cannot be or are inconveniently expressed in the symbolic graph, such as python's if vs tf.if and for vs theano.scan, or conditioning on some python-code (not tensor operations). So to build an eagerly evaluating symbolic graph framework that is allowed to do arbitrary things would mean that you would (to an extent) reimplement the language you are working with.
Re:PyTorch – Tensors and Dynamic neural networks in Python by liuliu on Thursday January 19, 2017 @01:04AM (#13433451)
Let's assume Tensorflow has basic symbolic computation graph expressiveness. What you would do is to build a symbolic representation while executing your graph inline, your symbolic representation doesn't need to have any control structure, it is simpler than that. You execute while loop in Python as usual, and your symbolic representation won't have TF.While at all, it will simply be the execution you performed so far (matrix mul 5 times).

Once you have a reasonable symbolic computation graph library, you don't need to explicitly build a "tape" because the symbolic representation will record the order of execution and reverse AD even graph optimization (applying CSE etc) come naturally as well.