The only problem being if you want to debug the optimized code ;)

This is true a lot in games - often the "debug" version of a game is literally too slow to reproduce issues, so you have to work with the optimized builds.

...even if the actual debugging would be much easier with a debug build.

Understood. However even then I would hope that games have unit tests and other module tests that can run just a subset of code and not the entire game. When I'm writing code I want to know it builds and the basics work before I run the entire program.

I work on seasonal programs. I can only do fully realistic tests in the month of April (depending on the weather it shifts a few weeks) , meaning my window to test is closed for the next 11 months. I have learned to find ways to run my code in less realistic situations.

There are also those bugs that only manifest in the release build.

That happens, but fortunately not often. Most problems show up in either optimized or unoptimized code. If it is in optimized code only it is a compiler bug (rare but I have seen it) or a race condition that won't show up in the debugger either. (there are a few more choices but the above are the big ones in my experience)

In optimized code, it is far more likely that one has a programming error in one's source code that an aggressive optimization revealed, rather than a bug in an industrial compiler. For example, an uninitialized variable that ends up sharing a memory location with some other variable whose life-time doesn't overlap in optimized mode rather than having its own memory location without optimization.

That one is caught be a warning, but there are thousands of possible errors that can be exposed this way.

Compilers have bugs all the time, but code in a write-test-debug-cycle typically has many more.

Which is why equally aggressive debug checks are important