Evolution only evolves to evolve faster where that is itself an evolutionary advantage; likewise slower only when it’s an advantage. (Where there is no advantage either way you can get drift; also fitness is at a much higher level than a particular gene).

This is why we don’t have the 30-minute reproductive period of some bacteria.