Clock only needs to be distributed to sequential components like flip flops or SRAMs. The number of clock distribution wire-millimeters in typical chip is dwarfed by the number of data wire-millimeters, and if a neural network is well trained and quantized activations should be random, so number of transitions per clock should be 0.5 (as opposed to 1 for clock wires), meaning that power can't be dominated by clock. The flops that prevent clock skew are a small % of area, so I don't think those can tip the scales either. On the other hand, in asynchronous digital logic you need to have valid bit calculation on every single piece of logic, which seems like a pretty huge overhead to me.
There's more promise in analog chip designs, such as here:
https://spectrum.ieee.org/low-power-ai-spiking-neural-net
Or otherwise smarter architectures (software only or S/W+H/W) that design out the unnecessary calculations.
It's interesting to note how extraordinarily wasteful transformer-based LLMs are too. The transformer was designed part inspired by linguistics and part based on the parallel hardware (GPU's etc) available to run it on. Language mostly has only local sentence structure dependencies, yet transformer's self-attention mechanism has every word in a sentence paying attention to every other word (to some learned degree)! Turns out it's better to be dumb and fast than smart, although I expect future architectures will be much more efficient.
Partly, but also because the brain has an asynchronous data-flow design, while the GPU is synchronous, and as you say clocked at a very high frequency.
In a clocked design the clock signal needs to be routed to every element on the chip which requires a lot of power, the more so the higher the frequency is. It's a bit like the amount of energy used doing "battle ropes" at the gym. The heavier the ropes (cf more gates the clock is connected to), the more power it takes to move them, and the faster you want to move them (cf faster clock frequency) the more power it takes.
In a data-flow design, like the brain, there is no clock. Each neuron fires, or not, independent of what other neurons are doing, based on their own individual inputs. If the inputs are changing (i.e. receiving signal spikes from attached neurons), then at some threshold of spike accumulation the neuron will fire (expending energy). If the inputs are not changing, or at a level below threshold, then the neuron will not fire.
To consider the difference, imagine our visual cortex if we're looking at a seagull flying across a blue sky. The seagull represents a tiny part of the visual field, and is the only part that is moving/changing, so there are only a few neurons who's inputs are changing and which themselves will therefore fire and expend energy. The blue sky comprising the rest of the visual field is not changing and we therefore don't expend any energy reprocessing it over and over.
In contrast, if you fed a video (frame by frame) of that same visual scene into a CNN being processed on a GPU, then it does not distinguish between what is changing or not, so 95% of the energy processing each frame will be wasted, and this will be repeated frame by frame as long as we're looking at that scene!