Actual premature optimisation is harmful for the same reason as premature abstraction. In both cases, improvements that come "for free" should be done without thinking, but one they start to impact readability, we should consider how beneficial those changes really are.