News for hackers. Stuff that matters.

A comment on Any Deep ReLU Network Is Shallow.

Re:Any Deep ReLU Network Is Shallow by sp332 on Friday June 23, 2023 @03:40PM (#36451321)
Nice! I have to admit I didn't follow all the math, but the layers get really wide, right? You still need to describe all the partitions between the peices of the piecewise linear functions. Does this save any computation at inference time?
[ Parent | on HN ]
Re:Any Deep ReLU Network Is Shallow by polygamous_bat on Friday June 23, 2023 @05:03PM (#36452126)
My hunch is that you can really easily find (adversarial) counterexamples that may show that in the worst case you will always use more compute by shallow-ing. In my head it should be related to no free lunch theorem.
[ Parent | on HN ]