Actually, the name is not so misleading because `ReLU(Wx + b)` is an approximation of the response of a large ensemble of integrate-and-fire neurons in terms of the firing rate [1] (it models populations of pyramidal neurons in the cortex). The ReLU activation function basically approximates a logarithmic activation function [2]. Obviously many kinds of computations are left out in this model, for example in pyramidal neurons, the integration of the incoming spikes is often sublinear rather than linear; the branches of the neurons that receive signals (dendrites) can themselves perform complex computations such as spatial clustering and logical operators basically sending small (dendritic) spikes forward and backward within an neuron [3]; short-term plasticity is left out entirely and so are countless types of interneurons which inhibit other neurons nearby and measure mean activity across populations of neurons etc.
[1] https://web.stanford.edu/group/brainsinsilicon/documents/Won...
[2] https://www.utc.fr/~bordesan/dokuwiki/_media/en/glorot10nips...
[3] https://neurophysics.ucsd.edu/courses/physics_171/annurev.ne...
(I only mention ReLU because they are definitely simpler in mathematical structure than ye olde sigmoid. Max out is another interesting activation function that's first and foremost inspired by the math / pragmatics of drop-out training, and were any relations to biology more likely to be subsequent discoveries and not intentional design features.)
That happens fairly often in engineering: looking around enough one can often find natural precedents for our artifices. But directly trying to copy nature did not lead to the plane or helicopter.
[1] R Hahnloser, R. Sarpeshkar, M A Mahowald, R. J. Douglas, H.S. Seung (2000). Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit.
This works just as well whether you are modelling biological synapses with plasticity, or compositions of functions with training via backpropagation.
The only real difference is that NNs aren't usually "physically embodied" (i.e. embedded in a metric space with radiative signalling to non-connected but "local" neighbours) in the way they'd have to be to get the "area-effect excitation" effects of neurotransmission.
Historically researchers looked at activation functions that are at least somewhat justified by nature, like the sigmoid. More recently people have focused on the mathematical aspects and figured out that the rectified liner unit `ReLU(x) = max(x, 0)` works just as well if not better for most applications.
Maxout networks (https://arxiv.org/pdf/1302.4389.pdf) are another interesting development in that direction.
There's still lots we can learn from biology, of course; but the last decades rapid advances we saw from deep learning at most barely relate to real brains.