Well, at some point you might run out of space in the opcode byte (or multiple bytes), and then you need longer sequences that put up more cache pressure and require more time decoding, right?

Also, the developers time budget and the VMs complexity migth keep you from optimizing all possible paths.