News for hackers. Stuff that matters.
Re:The Fourier transform is a neural network by AndrewGYork on Thursday April 29, 2021 @11:01AM (#26982043)
One of the questions raised at the end of the article:

"Do we ever benefit from explicitly putting Fourier layers into our models?"

Has a simple, partial answer here:

https://docs.scipy.org/doc/scipy/reference/generated/scipy.l...

"...multiplying a vector by the matrix returned by dft is mathematically equivalent to (but much less efficient than) the calculation performed by scipy.fft.fft."

fft-as-a-matrix-multiply is much slower to compute than a standard fft, especially on large input.

Re:The Fourier transform is a neural network by modeless on Thursday April 29, 2021 @11:09AM (#26982159)
You can do the linear parts of neural nets in the frequency domain, but AFAIK you can't do the nonlinearity, so you have to inverse transform back to the spatial domain for that. The nonlinearity is an absolutely essential part of pretty much every neural net layer, so there is no big win to be had unfortunately. For convolutional nets in particular there are other ways of going faster than a naive matrix multiply, e.g. winograd convolution.
Re:The Fourier transform is a neural network by aesthesia on Thursday April 29, 2021 @12:12PM (#26983015)
But if your network contains a layer whose linear part performs an (approximate) DFT, you will get an efficiency gain by replacing it with an exact FFT.

You wouldn't want to use an FFT for most CNNs anyway because the kernels have very small support. Convolution with them is O(n) in the spatial domain as long as you recognize the sparsity.

Re:The Fourier transform is a neural network by evancox100 on Thursday April 29, 2021 @12:13PM (#26983027)
Why can't you apply the nonlinear activation functions in the frequency domain? What's stopping this or making it not work?