Why train when you can optimize?

Re:Why train when you can optimize? by SemanticStrengh on Saturday April 30, 2022 @01:10AM (#31213501)

And then your NN can't represent anything else than periodic functions.. If we had to build separate programs for each product requirements variations.. programming would not be viable. More generally neural networks can't even imitate a dumb calculator without throwing absurd errors despite the rules of calculus being trivial and well defined. And matching a calculator is a task order of magnitudes easier than the semantic Causal reasoning abilities of human NLU that is involved in argumentation, inferences and understanding. But people are pathetically fooled by the fallacy of it being a uNiVeRsAl aPpRoXiMaTor.

[ Parent | on HN ]

Re:Why train when you can optimize? by viraptor on Saturday April 30, 2022 @01:43AM (#31213649)

> And then your NN can't represent anything else than periodic functions

Significant parts of x+sin are close to linear. You don't need to use this activation as the result layer either. Why would we lose anything?

> If we had to build separate programs for each product requirements variations

We pretty much do? We use both extremely generic frameworks both in technical sense (.net) and organisational (sap). But we also have software written to specific requirements where needed (there's millions of very specific ways to invoice someone, companies get invoicing platforms written just for them from scratch). There's space for both approaches.

[ Parent | on HN ]

Re:Why train when you can optimize? by telchar on Saturday April 30, 2022 @01:41AM (#31213640)

And yet neural networks can solve symbolic integral and derivative problems and differential equations better than other computer algebra programs. Sure one network might fail to compute sin(x) numerically, but another could easily tell you its derivative is cos(x). Turns out they are pretty flexible. Do they need to do everything?

[ Parent | on HN ]

Re:Why train when you can optimize? by ABeeSea on Saturday April 30, 2022 @02:50AM (#31213950)

Source? I would be very surprised if there was a neural symbolic PDE solver better than what’s in wolfram mathematica.

[ Parent | on HN ]

Re:Why train when you can optimize? by SemanticStrengh on Saturday April 30, 2022 @03:07AM (#31214026)

[ Parent | on HN ]

Re:Why train when you can optimize? by mhh__ on Saturday April 30, 2022 @02:53AM (#31213962)

At the cost of how many parameters?

[ Parent | on HN ]

Re:Why train when you can optimize? by PartiallyTyped on Saturday April 30, 2022 @01:49AM (#31213677)

You are wrong.

https://www.vincentsitzmann.com/siren

[ Parent | on HN ]

Re:Why train when you can optimize? by SemanticStrengh on Saturday April 30, 2022 @01:52AM (#31213694)

? I did not say that NNs can't represent periodic functions. Show me a NN calculator with >99% accuracy if you wanna refute me.

[ Parent | on HN ]

Re:Why train when you can optimize? by PartiallyTyped on Saturday April 30, 2022 @02:09AM (#31213782)

> and then your NN can't represent anything else than periodic functions..

This is a quote from the parent comment where you state that NNs with periodic activations can't represent non periodic functions.

Siren (what I linked) uses periodic activations and is able to represent non periodic functions.

[ Parent | on HN ]

Re:Why train when you can optimize? by SemanticStrengh on Saturday April 30, 2022 @02:33AM (#31213872)

Oh OK, thanks yes that's what I was looking for! However if this NN can represent non Periodic functions, what cost does the Periodic function support incur on non Periodic function accuracy, if any?

[ Parent | on HN ]

Re:Why train when you can optimize? by PartiallyTyped on Saturday April 30, 2022 @04:36AM (#31214378)

It actually behaves a lot better than non periodic functions in some scenarios. It seems that period functions enable a network to encode positional information within [-1,1]. This is supported by the fact that positional encoding is needed in transformers to make them work.

[ Parent | on HN ]