Where does the energy go in neural network workloads?

Wires.

To and from memory, in the memory dice, on the compute die, in the register files, etc.

Check out the paper "DNN Dataflow Choice is Overrated". And that's for the most compute bound cnns!

Edit: and if you think there's enough SNR to make analog "deep" processing in memory work, go make a quick billion dollars by expanding commodity memory with multi level. If you can add another level to flash you'll get rich quick.

I dispute this. The problem with analogue processing (besides noise, of course) is that the dynamic range sucks compared to plain old floating point. Fine for inference, not for training. And most of the compute power in NN is spent on training, inference is trivial.

Analogue processing has certainly been successful for people, so I wouldn't be too dismissive.

I would not be certain of the premise in your statement. We do not understand the brain well enough to say whether error correction (which is inherently digital) is utilized. There are some pretty solid clues that some of the processing in the brain is not analog (e.g. the step-like behavior of the activation function of biological neurons).

And coming from the other side: If it turns out that the brain is actually an analog computer without anything that can be rephrased as some form of sloppy noisy approximately-digital error correction, this would be incredibly important discovery, completely changing our understanding of theoretical computer science and complexity theory. P-vs-NP or the eventual creation of quantum computers would pale in comparison to the impact of discovering scalable analog computers.

Agree mostly, except for "error correction .. inherently digital"

For an example of an analogue error corrector, see https://en.wikipedia.org/wiki/Centrifugal_governor

PS. cool fact; "governor" has the same etymology as kubernetes.

Inference is typically much more power constrained, as it is often performed on a battery. For example in a mobile phone, or in an IoT sensor unit. This makes approximate computing for neural network accelerators an interesting field of research.

Nailed it: https://www.mythic-ai.com/technology/