The existence of the request itself (for the handshake) is most of the cost. If a client is on the other side of the world, hundreds of ms. Otherwise if you have servers everywhere, still probably 50-70ms for the handshake.

Edit: in my experience.