At least with MSVC if you use SIMD heavily the debug build will be 100x slower easily. I believe it creates debugable SIMD by storing each operation to memory, though I'm not entirely sure how they managed such a massive slowdown even so:)

Anyway the way I deal with it is to mark functions I want to debug with a macro that disables optimizations just for that function.