That scaling has a limited range before static power becomes dominant. Compute efficiency is compute / energy (total power * time). Total power includes static power. An SBC pulls far less from the wall than an x86 desktop could ever hope to when calculating the first million digits of pi.

As an example: even if I halted my desktop at 0 MHz and it still magically took the same time to calculate the first million digits of pi as a raspberry pi, it still would be using far more power.