I think it's the opposite. A good annotated code which can use SIMD instructions accelerates tremendously with -O3. -march=native -mtune=native generally improves things if you can saturate the cores with instructions, which can be measured by running perf record and looking to IPC & instruction retirement numbers.

When I was using Eigen on my code, the biggest performance boost came from -O3. -march and -mtune did minimal improvements on the systems which I've ran benchmarks on.

ffmpeg and associated projects do not rely on autovectorization, which pretty much only works on naive scientific code. They write their SIMD in assembly and don't need to care about compiler settings.

If you're compiling hand tuned assembly, -march and -mtune probably will have more effect when compared to compiling and optimizing C/C++ code.

OTOH, I'd like to underline that heavily optimized scientific code and libraries are neither naive (in terms of algorithmic complexity/implementation) nor straightforward :D

E.g.: This is how Eigen configures its internal vectorization parameters: https://gitlab.com/libeigen/eigen/-/blob/master/Eigen/src/Co...