I take it with a grain of quantum physics. When traveling back in time you are entangling the state of the moment of time travel with a past state of the universe, and the universe must collapse into a steady state with a single history... The history in which the time travel did nothing.