One of my favorite examples is Ubuntu - they run their 'One Hundred Papercuts' program [1], which is about fixing and optimizing little things that on their own would never get fixed, but together they are more than the sum of their parts in terms of user experience.
http://permalink.gmane.org/gmane.comp.db.sqlite.general/9054...
Occasionally performance is a feature, but a lot of times it's just an excuse for developers to have fun writing assembly.
300ms still doesn't put a dent in 8 or 9 seconds.
> So keep calm, optimize on. This is how compilers get better over time.
So work with the compiler rather than against it. As others have pointed out, with the correct -march setting you get code that's just as good as this hand-tuned assembly - and much more maintainable, and the correct compiler setting will improve the rest of your code as well.
I have a different opinion on those 3ms optimizations:
"It all adds up"
Stopping at one 3ms optimization is not going to move mountains. But doing that 10 times is already 30ms. Imagine that you found 100 micro optimizations. Now we are talking!
So keep calm, optimize on. This is how compilers get better over time.