This suggests that the effective number of parameters is far lower than the nominal number. My head canon for neural networks as overparametrized models still holds.