Any Deep ReLU Network Is Shallow

Re:Any Deep ReLU Network Is Shallow by madmaxmcfly on Friday June 23, 2023 @05:26PM (#36452356)

Isn't that just like a Taylor series?

[ Parent | on HN ]

Re:Any Deep ReLU Network Is Shallow by charliea0 on Friday June 23, 2023 @05:34PM (#36452445)

Yes - or Fourier series or other decomposition into a sum of orthogonal basis functions.

[ Parent | on HN ]

Re:Any Deep ReLU Network Is Shallow by mkl on Friday June 23, 2023 @09:41PM (#36454599)

I don't think orthogonality is important here. Taylor series basis functions are not orthogonal, and nor are ReLUs.

[ Parent | on HN ]