The reason why struct packing would be a win is probably not performance, but code readability and less bugs.

Making be/le/native conversion a job for the programmer is not only error-prone and a waste of time.

The compiler could safely optimize the byte-swizzles away on non-arithmetic operations, whereas most programmers tend to covert everything before they start working on it.