There is such a thing as premature pessimization: making choices based on intuition about productivity benefits or personal taste that force you down the road to poor performance.

I think it's smart to include performance requirements in your specifications before starting a project (or when taking on the maintenance of one too). Set an upper limit on your TTI or server response times. Turn it into a budget. That's not optimization that's just engineering.