Not necessarily. It turns out to really matter how the weights are initialized (randomly, but from which distribution?) for ordinary feed-forward neural nets that only uses standard neurons -- different activation functions have different optimal initialization distributions.

https://machinelearningmastery.com/weight-initialization-for...

A big breakthrough was Kaiming He's work on initialization for ReLU:

https://arxiv.org/abs/1502.01852

(Kaiming is the personal name, He is the family name. He Kaiming is the native Chinese order. Lots of citations use "Kaiming" by mistake.)