If you think about it in terms of the book Time Machine, the machine itself is still affected by gravity as it travels through time. You would expect it to remain stationary relative to the the objects it started traveling on, assuming it wasn't affected by some external force.