Wow, I never realized my intuition was off by so much. Speed of light in vacuum is still about 16ms between coasts, which is orders of magnitude worse than I thought. I thought most latency came from routers and switches but a significant portion of delay is indeed speed of light.