I think it is relative. Your argument considers time as a non relative entity and the past and present as static points in a timeline. So the same "event" is one man's past and one man's future. It depends on your space coordinates from the coordinates of the even if this is your past or present or how back in past.

Disclaimer: This is deduced logic and not scientific information.