While that’s a problem in theory, to moderate accuracy it’s not a practical issue. Most objects in the universe are roughly at rest relative to the cosmic background radiation (in other words they don’t see it as Doppler shifted in any particular direction), so we can use that as a common frame of reference. When we’re talking about ballpark integer billions of years, or even to a few decimal places, that’s easily good enough.