Specifically, the problem is with "carries". When you're adding (or subtracting, or multiplying, or dividing, I'll just discuss adding) two binary values you might have to carry a 1 to the next place.
If you've got a BE value and an adder stage smaller than that value (say, a 32-bit number and 1-bit adder stages) you have to carry a 1 many times to output the result. If you're receiving the value in BE order 1 bit at a time you can't start the computation until you have the LSBs of both values, since if they're both 1 they'll affect the second bit by their carry. So you're stuck waiting for the entire value to start the computation. Further, during the computation you have to wait for every 1-bit adder in sequence.
There are "fast" adder designs that don't have to wait for every bit, but can instead work on groups of multiple bits with a carry-out at the end of the group. So if you've got an 8-bit group size, you'd have at most 3 carry delays during the computation of the output. For BE, you'd have to wait for all 4 bytes to be received, then wait for all the carry delays. For LE, you can start the computation as soon as the first byte is received, saving some time.
The larger the adder group size the more die area is needed, the stronger the drive strength of the transistors needs to be (bigger fan-out), and the slower the maximum clock of the overall system. On the other hand the bigger the group size the fewer carry delays, so addition can take fewer cycles. Most CPUs and MCUs implement single-cycle addition of their word size. Some CPUs even implement single-cycle multiplication at their word size.
There are also non-Linux cases, mostly microcontrollers. EG the Arm Cortex M0 doesn't have a hardware multiplier, the M0+ does.
And then there's that one guy who got Linux running on an 8-bit AVR by emulating a 32-bit ARM and running it on that[1]. I'd consider this a silly edge case. Too fun not to mention though.
[1] https://dmitry.gr/?r=05.Projects&proj=07.%20Linux%20on%208bi...
Obviously this example is very contrived. This sort of thing was much more of a concern on 8-bit computers. But little endian still seems more natural to me.
Some of the other replies have minor technical reasons, but I've always preferred big endian for the readability. Having said that, I'm happy to part with the idea of big endian if it means an end to having 2 options to worry about. One thing that bothers me a lot about RISC-V is that the standard claims to allow either big or little endian implementations. Little has won and nothing new should support big endian IMHO. The benefits of either are largely irrelevant, but the existence of both is a problem. Or maybe it's that the existence of code that cares is the real problem ;-)
To me, big endian makes a lot more sense - integers are stored in the order which you read them (mentally).
I do remember reading about how little endian enabled some type of optimization inside the CPU, but I forget the specifics.