When to assume neural networks can solve a problem

Re:When to assume neural networks can solve a problem by ludwigschubert on Monday March 30, 2020 @11:42AM (#22728884)

How would one extract an algebraic function from the network’s weights? Why would the function the network approximates necessarily be algebraic?

[ Parent | on HN ]

Re:When to assume neural networks can solve a problem by Plough_Jogger on Monday March 30, 2020 @12:46PM (#22729603)

The function would simply be a chain of multiplications and (typically) non-linear transformations.

For classification problems, there is often a final non-linear transformation, like a softmax.

Example with ReLu activation:

Output_i = max((input_i * weight_i),0)

[ Parent | on HN ]

Re:When to assume neural networks can solve a problem by jawarner on Monday March 30, 2020 @12:58PM (#22729736)

I don't understand. The example you gave is not an algebraic function.

[ Parent | on HN ]

Re:When to assume neural networks can solve a problem by Plough_Jogger on Tuesday March 31, 2020 @06:43PM (#22743552)

You are correct. It is possible to construct a linear neural network, where an algebraic function could be extracted, but in practice, almost all networks use non-linear activation functions.

In the case of a linear network, the function would be a dot product between the input and the weights: 𝑥1𝑤1+𝑥2𝑤2+𝑥3𝑤3 ... for all inputs (xi) and weights (wi)

[ Parent | on HN ]