Edited to add: A lot of performance loss for large, complex systems is death-by-a-thousand-cuts -- 20ms here and there can make a real, user-perceivable difference.
When optimizing boot time of an embedded system something like 100 ms is a significant saving once you've picked all the low hanging fruit in the user space init side. In the end, every millisecond counts as it all adds up when there are hundred or so places where they may end up being spent needlesly. An embedded product, especially automotive, might have rather strict boot time requirements.
The time it takes for a bootloader to load a kernel image can be a decent part of the boot time. The bootloader most likely won't be running the SDHCI controller in any of the faster modes that require implementing bus training, calibration, and tuning algorithms of varying complexity.
Plus there are things like [e]MMC's "boot" area, where a special storage area is simpler to read using a simpler special protocol. That area has a limited size.