https://github.com/riicchhaarrd/nn

I wanted to try to compress images and export them as a function, it didn't work that great but I learned something. I don't fully understand everything but my assumption was that taking inspiration from JPEG and DCT and projecting onto basis functions for Spherical Harmonics the neural net would do the same using different activation functions as basis. Trying out different functions I found that cos/sin work really well for capturing details with low amount of layers and neurons without exploding even when setting the learning rate lower than I would have to with tanh or ReLU/Softmax. Afterwards I found papers talking about this.

https://arxiv.org/abs/2006.09661 https://arxiv.org/abs/2006.10739

Maybe this is just a really terrible CNN, convolution/kernels vs random sampling pixels and inputs dimensionality, input space constrained to activation functions and so on.