A thing I remind myself of regularly: Avoiding worrying about optimisation up front too much is not avoiding caring about optimisation at all, it's reserving the number of hours you can expend on optimisation until you have enough working to have a profile available, because you are inevitably going to be wrong about which part is actually slow until that point so your optimisation hours budget will be better spent with a profile in hand.

There's a parallel here to "no, you did not find a bug in the compiler". Yes, ok, once every five years or so I actually did find a bug in the compiler, but assuming you aren't that smart is still a far far better default approach.