Re:Any Deep ReLU Network Is Shallow
by visarga on Friday June 23, 2023 @06:45PM
(#36453234)
> So any N-layer network with the relu _/ activation function
The paper talks only about models with additive operations among activations. It doesn't say anything about more complex networks like transformers.
In transformers there are multiplicative interactions between activations inside the attention matrix, it is unclear if they can be approximated with just a 3 layer ReLU network, or if such conversion would be practical at all.
The paper talks only about models with additive operations among activations. It doesn't say anything about more complex networks like transformers.
In transformers there are multiplicative interactions between activations inside the attention matrix, it is unclear if they can be approximated with just a 3 layer ReLU network, or if such conversion would be practical at all.