Re:Do Machine Learning Models Memorize or Generalize?
by qumpis on Thursday August 10, 2023 @01:41PM
(#37079224)
Slightly related but sparsity-inducing activation function Relu is often used in neural networks
Re:Do Machine Learning Models Memorize or Generalize?
by medium_spicy on Thursday August 10, 2023 @01:44PM
(#37079280)
Short answer: if the inputs can be represented well on the Fourier basis, yes. I have a patent in process on this, fingers crossed.
Longer answer: deep learning models are usually trying to find the best nonlinear basis in which to represent inputs; if the inputs are well-represented (read that as: can be sparsely represented) in some basis known a-priori, it usually helps to just put them in that basis, e.g., by FFT’ing RF signals.
The challenge is that the overall-optimal basis might not be the same as those of any local minima, so you’ve got to do some tricks to nudge the network closer.
Second, the article correctly states that typically L2 weight decay is used, leading to a lot of weights with small magnitudes. For models that generalize better, would it then be better to always use L1 weight decay to promote sparsity in combination with longer training?
I wonder whether deep learning models that only use sparse fourier features rather than dense linear layers would work better...