Re:New kind of recurrent neural network using attention
by jostmey on Tuesday March 07, 2017 @10:07PM
(#13817534)
The RWA model is described in equations (7) of this paper: https://arxiv.org/pdf/1703.01253.pdf
I think you will see that the cost per training set is just as cheap if not cheaper than a LSTM model. On my machine, the RWA model completes each training step faster than a LSTM model. I don't know if that will hold for GPUs.
Re:New kind of recurrent neural network using attention
by singularity2001 on Wednesday March 08, 2017 @05:34AM
(#13819343)
Yes, give us wall time please
Let me know when you have more results! I want to start doing things with text soon and I would love to try RWA