Assume that the volume of the visible universe grows, and then consider a ball that just circles on larger and larger circles. Clearly, the ball does not need to get back arbitrarily close to where it started.