The 3.5x energy-efficiency gap between serial and SIMD code becomes even larger when

A. you do byte-level processing instead of float words;

B. you use embedded, IoT, and other low-energy devices.

A few years ago I've compared Nvidia Jetson Xavier (long before the Orin release), Intel-based MacBook Pro with Core i9, and AVX-512 capable CPUs on substring search benchmarks.

On Xavier one can quite easily disable/enable cores and reconfigure power usage. At peak I got to 4.2 GB/J which was an 8.3x improvement in inefficiency over LibC in substring search operations. The comparison table is still available in the older README: https://github.com/ashvardanian/StringZilla/tree/v2.0.2?tab=...

It makes intuitive sense that going wide in SIMD uses less power than going wide over cores, because extra SIMD lanes take up much less silicon area than adding another whole core. If they didn't then we wouldn't bother with SIMD, we'd just make more cores and save ourselves the hassle of juggling two different types of parallelism.

GPUs take this even further by going even wider, with each "core" typically executing 1024-bit SIMD operations (i.e. 32x FP32, 64x FP16, etc). CPUs have roughly settled at 128-bit (most ARM) or 256-bit (most x86) with a little bit of 512-bit (x86 with AVX512) which grew out of Intels earlier aborted attempt at making a dedicated GPU.

Not only that, but you need to decode the SIMD instruction only once, and not N times. Same with other stages of the pipeline.

Yep, I've heard that actually something like 80% of the energy of the CPU is used by the front-end! Beyond decoding, you also have all the energy spent on scoreboarding data dependencies and reordering the instructions and scheduling to execution ports. If you can do that 8x less as in the case for SIMD, that's a huge win!

That's definitely wrong (unless you use an incredibly expansive definition of "frontend"). The simplest way to prove that it's wrong is that heavily vectorized workloads can easily cause a 30% increase in power consumption over scalar workloads even though the instruction density is similar.

Wouldn't that support the idea that most energy consumption is in decoding? If you're getting 2x, 4x, 8x ect as much value computation per instruction and yet only a 30% increase in power then clearly most the power is not used by computing the values.

no, because there's a lot the CPU does that is neither decoding nor execution. There's also caches, register renaming, branch prediction, inter core communication for atomics, and a dozen other things.

Sorry poor terminology use on my part. I mean more broadly that most energy is used on frontend and middle end, rather than backend and that this is what vectorisation improves in regards to energy consumption. Register renaming and branch prediction energy consumption should be improved in the same factor as decoding. Caching probably less so (depending if we are talking instruction, data or combined).

I don’t think inter-core communication is too relevant when comparing vectored and non-vectored on a single core, but definitely would be when batching across multiple cores.

I wonder how efficient a modern x86-64 CPU could be — on a scale from “CISC” to “single-purpose ASIC” — if you could compile and load microcode directly into cache lines, pre-baking all of this just-in-time silicon allocation decision-making into a static microcode loop, so that the frontend can shut off entirely.

(Presumably doing this would only be possible in the context of a single-threaded uninterruptible unikernel workload with system-management-mode functions disabled — but I’m sure a lot of “one powerful single-core SoC”-type embedded systems would be happy to make that trade off!)

Forgive me if i'm misunderstanding, CPU's have micro-op caches that will bypass decoding instructions altogether if its in that cache. This means that variable length instructions will be decoded to a micro-op in the cpu and will get reused for as long as you have cache hits. Which does mean you"ll have ghost performance penalties if your code/gcc doesn't respect the size and limitations of hidden caches. Otherwise you can pretend that any instruction length gets shortened to its micro-op.

I think I was imagining a useful operation level lower than the regular uOp — one less akin to programming a RISC processor, and more akin to the direct control over individual RTL signal lines — similar to the "instruction word" of signal-line states directly encoded by a row of the op-decoder PLA of a 6502.

Right now, AFAICT, even when executing a uOp stream from L0 cache, several "planning" systems are still active — juggling caches around and deciding routing between them; renaming registers; switching between power "license" states; etc. None of these decisions are explicit even in uCode for modern CPUs, so they have to be made, over and over again, even when running from uOp cache. Which means the silicon that's making these decisions can't ever go dark, even when running from uOp cache.

A "nanocode" would be a version of microcode that burns in all these "planning"-system decisions — and which can thereby put all the "planning" silicon to sleep. There would be "nanoOps" for explicitly wiring registers to other registers or cache buffers, for adding precise numbers of delay cycles, etc. And these would all happen at Nth-of-a-cycle-precise points in the execution stream, indicated by either entire nOp instructions, or pragma bits on nOp instructions.

(Given this, to usefully function, these "nanoOps" would either likely need to be bytecode and be decoded+executed at some ridiculously high frequency relative to the regular CPU clock — or they would need to be some arcane parallelized packing of what was originally a serial stream of uOps, such that you get VLIW nOps where all the pragmas you want to have happen for the next CPU cycle are indicated in the instruction-word at once as individual signal-line bits. Just like the 6502 PLA output "instruction word", actually!)

I know this probably sounds like nonsense — the instruction stream would be so much more bloated than microcode that it probably wouldn't be worth it. But that assumes an instruction stream that needs to live in RAM and get shuttled through layers of caches to reach the decoder. But what if the instruction stream could be loaded into a reserved SRAM area within the CPU itself — an SRAM that would effectively act as EEPROM (in that it would be loaded from CPU NVRAM on boot); and which you could directly drop the instruction pointer into (in nCode decode mode), skipping RAM+caching entirely?

I mention this, because I've always had a vague hypothesis that Intel x86 CPUs specifically already have something akin to this implementation of a "nanocode" + reserved SRAM area that holds some of it — specifically for the purpose of programming custom dynamic RTL for instructions after CPU release to hotfix CPU errata. It's something they had to learn the necessity of over and over again, after releasing CPUs with bugged instructions, all the way back to the FDIV bug.

Sadly I wouldn't know enough about microarchitecture to understand whether nanocode would actually be a benefit. surely if CPU's became big and complex enough there would be a benefit to have hierarchical translations down to micro-ops and smaller subdivisions for execution units. What you're describing seems to come close to FPGA's. The possibility to write very complex sequences to pipeline different payloads trough the system while maintaining proper state. Though I think the economical solution might be more boring, the intel ME and AMD PSP hardware is probaly responsible for alot of management of state that could flush or change how instructions are executed with very basic logic.

If esoteric hardware interests you I would recommend IBM system/360 as reading material. Sadly hardware like that has fallen out of fashion and we've been stuck with the x86 and arm dichotomy. if anyone else has interesting/newer non-x86 hardware to link I'm more than happy to delve into that.

Given the massive # of instructions that an M3 core can keep in flight, I suspect that Apple’s CPU engineers could (but won’t) write some really interesting papers on this challenge.

Memory traffic too, assuming the SIMD code is properly optimized, you can slam nicely aligned blocks of memory straight into a SIMD register rather than loading each element one at a time.

this is less important than you would expect because memory gets fetched into caches 512 bits at a time. As such, the difference in power use is only the cost of loading from L1 cache (since the first instruction will bring the line to l1)

There's a talk by a Nvidia scientist I just saw where he talks about compute efficiency in terms of flops/joule and explicitly mentions vectorized instructions as one of the energy savers. I'll see if I can find it.

Edit: found it (I think) I can't remember where in the presentation but he does mention energy efficiency as a proxy for performance. https://m.youtube.com/watch?v=kLiwvnr4L80

Edit 2: about 26:40 he starts talking about energy use in the context of performance.

I guess it also makes sense if you figure that dedicated hardware for things like AES, video codecs, etc. are similar to SIMD

Anybody know whether the same is true for gpu vid decode va cpu?

> if your tasks complete much faster, you are likely to use less power, even if you are using more energy per unit of time.

isn't this another formulation of the race-to-idle thinking?

everyone who has early batches of Alder Lake should try AVX512-FP16. Extremely fast for AI tasks.

Please look at these number with the following grain of salt, when optimizing programs. They are trumped by cache usage patterns [1]:

    Loading (fig IV)
    
    Location    Energy (pJ = pico Joules)
    L1          64   pJ/Byte
    L2          121  pJ/Byte
    L3          254  pJ/Byte
    RAM         1250 pJ/Byte
    
    Adding integers with SIMD (fig VI)
    428 pJ/op, for 8 Byte/op; this means: 
    53 pJ/byte

So it takes actually 20 times to fetch data from RAM than to add it to something! And most often this is also the source of latency. Generally energy is linear to the distance signal has to travel, and that's the same for latency.

That's why successful data structures are sized a tiny bit under L1/L2 sizes! (BTree chunks, ring buffers).

If you've been following hardware, it's all about putting RAM closer to compute at the moment with chiplets at the moment.

[1] https://tu-dresden.de/zih/forschung/ressourcen/dateien/abges...