Does that really make that big of a difference? You can still tell the difference...

It does perceptually, not intellectually. Obviously anyone can see that it's offset, but the perception is still smaller than it actually is. There's no easy marker to re-align and try to make a better estimate.

Let's take a step back though- why are the phones in this image offset at all? Apple made the image, it's explicitly for a size comparison, so why make it harder to comprehend?

Good catch, I missed that.