douglasorr
Procedural Textures with Hash Functions
Llama 3.2 Vision – A Deep Dive
Benchmarking std:{string, string_view, reference_wrapper}
ML papers (June): Mamba-2 and Matmul-free Models
Almost-scaled dot-product attention
Metaphor Thinking in Deep Learning
Rasterized ray tracing of sound in JavaScript
GitHub Game Off – “Bug”
Autodiff from scratch
How to train your classifier – deep learning training objectives
Re:My phone case, the hidden state of an RNN
on Thursday September 09, 2021 @04:05AM (#28467413)
I think at least some of the structure is down to the update equation, roughly 'new = sign(gate * old + ...)'. Since 'gate' is always positive, this means that even when the network is randomly initialized, the state is more likely to stay the same between two time steps than to change.
I tried "deep transitions" which repeats this equation a couple of times, and the results were indeed noisier.