I agree
>(i.e. you've identified your bottleneck and determined its worth the time to improve it).
For this, though, it is often possible to identify this BEFORE writing any code. As opposed to refactoring an inner loop, Big-O challenges may often stem from (at least in my experience) the top level design of the code, and changing it may require rewriting almost everything.
This is particularly true in a microservices world, where the problem may often be how you split a computation task up into services in the first place.
I was in a design discussion this morning for some new functionality which requires hitting a pay-for API; and much of the design was based on minimising how many calls we'll make. A naive polling approach would require O(t×e×u) requests, where t is time, e is the number of the events we care about and u is the number of users affected by those events. The design we came up with uses a separate data source to identify when those events occur, which removes the factor of t. We also came up with some de-duplication logic which reduces the constants.
Both of the quotes in your comment resonate with me, in the sense that they can avoid unnecessary refactors. But IMO the big-O/performance part should really mostly just be done on a by-need basis (i.e. you've identified your bottleneck and determined its worth the time to improve it). I've seen way too much unreadable code because "otherwise it wouldn't be performant" in places where performance doesn't matter, and similarly I've seen people sacrifice a clean modular interface for the sake of performance.
Sure thinking ahead of time makes sense, as does choosing a design which puts flexibility in the right places (i.e. gives you room to adapt to what you learn in the future which is currently uncertain). But that same uncertainty also means you're probably not going to get the design fully right now, so don't overthink it.