Yeah, that was one of the reasons I said "just about". But minuscule compared to millisecond seek latencies on a spinning disk.
So the problem getting orders of magnitude smaller while the costs of the cure increasing a lot swung the equation away from defragging being a good thing overall.
If the file is fragmented, this may require more reads than if it's contiguous and uses an extent.
Exact scheme is filesystem-specific.