Re:My phone case, the hidden state of an RNN
by only_as_i_fall on Tuesday September 07, 2021 @03:52PM
(#28449060)
Looks great but I'm surprised they aren't way noisier. Is that just a result of having a limited number of neurons compared to the number of pixels, or is there some specific trick to it?
[ Parent
| on HN ]
Re:My phone case, the hidden state of an RNN
by douglasorr on Thursday September 09, 2021 @04:05AM
(#28467413)
I think at least some of the structure is down to the update equation, roughly 'new = sign(gate * old + ...)'. Since 'gate' is always positive, this means that even when the network is randomly initialized, the state is more likely to stay the same between two time steps than to change.
I tried "deep transitions" which repeats this equation a couple of times, and the results were indeed noisier.
Re:My phone case, the hidden state of an RNN
by maxwells-daemon on Wednesday September 08, 2021 @12:22AM
(#28453004)
It's mostly that these are very narrow and shallow networks -- if I remember right, 5 layers and 30 units wide?