Re:A visual proof that neural nets can approximate any function
by throwawaymath on Saturday April 20, 2019 @08:07PM
(#19709715)
You could also just cite the Weierstrass function, which is continuous everywhere and differentiable nowhere. But that's separate from the meat of my point, which is that C(R) is a vector space.
The standard proof uses some functional analysis techniques but nothing too complicated to show you can get arbitrarily close to any continuous function with an NN. That includes things like step functions whose derivatives are not defined everywhere.