In my mind, the layers make the network "non-trivial". A Fourier transform is only a neural network in a trivial, definitional sense.

The activation function is what keeps layers separated. When it isn't there, a pair of layers devolves to a matrix multiplication, which can be replaced by its resulting matrix, a single layer.

That is a thoughtful distinction, too. I think you're right.

Biological neural networks do not use backpropagation, or at least, there's no evidence of that yet.

Backpropagation is a placeholder for stuff we don't understand.

It seems that backpropagation might be a good abstraction for biological learing rules after all: "local predictive coding converges asymptotically (and in practice rapidly) to exact backprop gradients on arbitrary computation graphs using only local learning rules" from https://arxiv.org/abs/2006.04182

In biology there are excitatory networks and inhibitory networks.

Isn't "backpropagation" just a synonym for the partial derivative of the scalar cost function with respect to the weights? And in the mathematical formulation of biological neural networks these derivatives can't be computed analytically. Your comment sounds like "backpropagation" is some kind of natural phenomenon.

> Your comment sounds like "backpropagation" is some kind of natural phenomenon.

No it's not, read again.

Not exactly the same kind of backpropagation but signals fly the other way in neurons as well https://en.wikipedia.org/wiki/Neural_backpropagation