If every action that can happen, does happen, then the "universe" that you left only exists in that exact moment of departure because every quantum interaction from that point forward creates more branches of infinite universes. Thus, which of those would you return to? You could only return to the exact moment you left or to some random branch that's created after you leave.