News for hackers. Stuff that matters.
Re:New kind of recurrent neural network using attention by jostmey on Tuesday March 07, 2017 @09:54PM (#13817488)
The RWA model requires fewer training steps than a LSTM model on the tasks I tested (https://arxiv.org/pdf/1703.01253.pdf). Even better, the RWA model uses over 25% fewer parameters (because it doesn't have lots of gates like a LSTM model). So the RWA uses less computer memory than a LSTM model.

To answer your second question: Yes I do plan on testing it on more challenging problems! I just need enough time and the GPU rig.

Re:New kind of recurrent neural network using attention by jorgemf on Tuesday March 07, 2017 @10:00PM (#13817507)
I was wondering the time per step, I am aware it converges faster.

Let me know when you have more results! I want to start doing things with text soon and I would love to try RWA

Re:New kind of recurrent neural network using attention by jostmey on Tuesday March 07, 2017 @10:07PM (#13817534)
The RWA model is described in equations (7) of this paper: https://arxiv.org/pdf/1703.01253.pdf

I think you will see that the cost per training set is just as cheap if not cheaper than a LSTM model. On my machine, the RWA model completes each training step faster than a LSTM model. I don't know if that will hold for GPUs.

Re:New kind of recurrent neural network using attention by singularity2001 on Wednesday March 08, 2017 @05:34AM (#13819343)
Yes, give us wall time please
Re:New kind of recurrent neural network using attention by kastnerkyle on Wednesday March 08, 2017 @04:31PM (#13824008)
So in that case it has the same amount of parameters as GRU - did you compare with that?
Re:New kind of recurrent neural network using attention by jostmey on Wednesday March 08, 2017 @05:00PM (#13824254)
No, I have not tried the GRU model