Not just that. If you store a lower precision type e.g. 2 byte integer inside a large precision type e.g. 8 byte integer, you can read the 2 byte integer by just reading two bytes. Extending or shrinking data types leads to a very natural way of implementing arbitrary precision arithmetic. To get the same capability with big endian your pointer has to point at the end of the number. If you have byte arrays, like a string, you would have to swap the order in which you allocate data, starting from the end of the array and always decrement your index from the array pointer. This would then also apply this to struts. You point at the end of the struct and subtract the field offsets.

Overall this seems like a pretty weird choice on a planet where the vast majority of text is written from left to right and only numbers are written right to left. Especially since endianness only affects byte order but not bit order, as you said.