Re:TensorFlow: open-source library for machine intelligence
by albertzeyer on Monday November 09, 2015 @10:43AM
(#10533630)
I'm looking at the RNN implementation right now (https://github.com/tensorflow/tensorflow/blob/master/tensorf...). It looks like the loop over the time frames is actually in Python itself.
for time, input_ in enumerate(inputs): ...
This confuses me a bit. Maybe the shape is not symbolic but must be fixed.I also haven't seen some theano.scan equivalent. Which is not needed in many cases when you know the shape in advance.
Re:TensorFlow: open-source library for machine intelligence
by obstbraende on Monday November 09, 2015 @12:08PM
(#10534213)
I think this loop actually still only builds the graph -- what `scan` would do. The computation still happens outside of python. That is, in tensorflow they perhaps don't need `scan` because a loop with repeated assignments "just works"... Let's try this:
It seems like in TensorFlow you can say:
import tensorflow as tf
sess = tf.InteractiveSession() # magic incantation
state = init_state = tf.Variable(1) # initialise a scalar variable
states = []
for step in range(10):
# this seems to define a graph that updates `state`:
state = tf.add(state,state)
states.append(state)
sess.run(tf.initialize_all_variables())
at this point, states is a list of symbolic tensors.
now if you query for their value: print sess.run(states)
>>> [2, 4, 8, 16, 32, 64, 128, 256, 512, 1024]
you get what you would naively expect. I don't think that would work in Theano. Cool.
Re:TensorFlow: open-source library for machine intelligence
by benanne on Monday November 09, 2015 @12:42PM
(#10534466)
Why wouldn't this work in Theano?
>>> import theano
>>> import theano.tensor as T
>>> state = theano.shared(1.0)
>>> states = []
>>> for step in range(10):
>>> state = state + state
>>> states.append(state)
>>>
>>> f = theano.function([], states)
>>> f()
[array(2.0),
array(4.0),
array(8.0),
array(16.0),
array(32.0),
array(64.0),
array(128.0),
array(256.0),
array(512.0),
array(1024.0)]
Re:TensorFlow: open-source library for machine intelligence
by obstbraende on Monday November 09, 2015 @01:11PM
(#10534670)
Thanks! When I tried this before, I thought compilation was stuck in an infinite loop and gave up after about a minute. But you're right, it works. Though on my machine, this took two and a half minutes to compile (ten times as long as compiling a small convnet). For 10 recurrence steps, that's weird, right? And the TensorFlow thing above runs instantly.
Re:TensorFlow: open-source library for machine intelligence
by benanne on Monday November 09, 2015 @08:07PM
(#10536979)
Agreed. Theano has trouble dealing efficiently with very deeply nested graphs.
Re:TensorFlow: open-source library for machine intelligence
by yablak on Monday November 09, 2015 @01:37PM
(#10534845)
You're right. There is not currently a theano.scan equivalent that dynamically loops over a dimension of a tensor.
That said, you can do a lot with truncated BPTT and LSTM. See the sequence modeling tutorial on tensorflow.org for more details.
It looks like using TensorFlow from Python will feel quite familiar to a Theano user, starting with the separation of graph building and graph running, but also down into the details of how variables, inputs and 'givens' (called feed dicts in tensorflow) are handled.