They do though. A ton of the neurons end up not activating on any particular layer, leading to a huge waste as you're passing a zero down the layers.

I spoke with a PM at TT and he told me an important idea is that we're spending a lot of electricity multiplying things with zeros.