News for hackers. Stuff that matters.
Re:New kind of recurrent neural network using attention by p51ngh on Saturday March 11, 2017 @07:02PM (#13848023)
I don't follow the intuition behind using an exponentiation of a as in e^a. You refer to this as a "context model" in your paper. Could you please elaborate? Thanks!
Re:New kind of recurrent neural network using attention by jostmey on Sunday March 12, 2017 @11:17AM (#13851020)
The name "concept model" is borrowed from one of the papers I cited. The concept model is just "a(x_i,h_{i-1})". It decides what is important. You can sort of think of it like a filter or gate. When the concept model returns a large value, creating a large exponent, the information encoded by "z(x_i,h_{i-1})" dominates the weighted average.