>We can throw darts to the board and measure how close we are to the bullseye. Then we can tune parameters like throwing force, elbow angel, etc. These parameters have physical meaning. We can reason that by throwing harder, the trajectory of the dart can be so and so affected, hence causing the dart to fall closer to or further from the bullseye.

This is how Newtonian physics is derived. Similar to neural networks you create an approximation based off your observations. The actual reason why those approximations work is not necessary to be known to utilize Newtonian physics.

>in each layer of a CNN, is it doing Fourier transform, low pass filtering, rotating, scaling or whatever? And even if they are, why does it work? Why does it give the result that we want?

What the layers themselves are actually doing doesn't really matter and each one is just a part of a larger system. What is important is finding a set of weights that approximates a function that minimizes loss. People care about the ability to find this approximations. What exactly it is found is less of a concern.

>Or we are happy with getting results of hot dog/not hot dog

Neural networks are not limited to binary classification.