I always thought that part of the reason that Apple measured energy impact in arbitrary units was because they're trying to take into account the energy used by the rest of the system. So, for example, if a task takes half the energy when running on E cores, but causes the screen, wifi, disk, etc. to stay on longer, then perhaps it really is causing more total power to be used.

But, OTOH, if it's happening in the background while you're already using the computer, then those parts were probably going to be on anyways, so I suppose it's fairly context-dependent.