> RISC-V shuffles the immediate bits on its instruction encoding to speed up decoding the immediate in hardware

You're right that this might be a problem, in principle. Surprisingly enough, however, RV is actually more than competitive as a system-level virtual ISA. People have experimented with this on existing emulation platforms, and that was the result they got.