News for hackers. Stuff that matters.
Re:New kind of recurrent neural network using attention by jostmey on Tuesday March 07, 2017 @10:04PM (#13817522)
In my opinion, that's the beauty of the approach. The attention mechanism reuses the same weights for each symbol in the sequence. The attention mechanism is essentially a weighted average that filters which pieces of the sequence are feed through each processing step. So no, it does not need O(n^2) weights.

My guess is that the RWA model would be well suited for natural language processing. I am biased. Of course, you would need to first convert each word into a vector representation (i.e. word2vec). You might also want to add a decay term in the weighted average to force the model to focus on the recent past and not the "deep" past.

Re:New kind of recurrent neural network using attention by amatsukawa on Tuesday March 07, 2017 @10:08PM (#13817537)
It still scales O(N^2) in terms of computation though right?
Re:New kind of recurrent neural network using attention by eximius on Tuesday March 07, 2017 @10:19PM (#13817581)
If you save numerator_t and denominator_t from a previous computation, then h_t = f( (numerator_(t-1) + z + a) / (denominator_(t-1) + a) ). So it should be linear with t?

(where z and a are z(x_i, h_(i-1)), etc)

Re:New kind of recurrent neural network using attention by jostmey on Tuesday March 07, 2017 @10:39PM (#13817685)
Yes, I do save the numerator and denominator.

The model I used is: h_t = f( (numerator_(t-1) + z x e^a) / (denominator_(t-1) + e^a) )

Equations (7) in the arxiv link (here it is again: https://arxiv.org/pdf/1703.01253.pdf) provide the update equations. When implementing the equations, you sometimes have to scale both the numerator and denominator back by a constant factor to avoid an overflow error

Re:New kind of recurrent neural network using attention by p51ngh on Saturday March 11, 2017 @07:02PM (#13848023)
I don't follow the intuition behind using an exponentiation of a as in e^a. You refer to this as a "context model" in your paper. Could you please elaborate? Thanks!
Re:New kind of recurrent neural network using attention by jostmey on Sunday March 12, 2017 @11:17AM (#13851020)
The name "concept model" is borrowed from one of the papers I cited. The concept model is just "a(x_i,h_{i-1})". It decides what is important. You can sort of think of it like a filter or gate. When the concept model returns a large value, creating a large exponent, the information encoded by "z(x_i,h_{i-1})" dominates the weighted average.