Re:New neural network architecture inspired by neural system...
by f_devd on Wednesday February 08, 2023 @02:21PM
(#34713359)
Learnable activation functions are a thing famously Swish[0] is is a trainable SiLU which was found through symbolic search/optimization [1], but as it turns out that doesn't magically make make neural networks orders better.
[0]: https://en.m.wikipedia.org/wiki/Swish_function [1]: https://arxiv.org/abs/1710.05941