I think relativity makes this impossible; you're essentially saying there's a particular trajectory that has an absolute velocity of zero. Relativity is founded on the idea that there's no such thing; any trajectory has an equal claim to be the zero velocity.

Maybe time travel would stay in the same reference frame, so if you're on the Earth's surface you end up in the same position on the Earth. On the other hand if you're on an object that experiences non-gravitational directional changes in the span of your time travel, you might be in trouble.