Not my experience at all. I can only safely refactor major parts of the large complex C++ software I am maintaining because of the tests.

If you can’t refactor your code without breaking tests then your tests are testing implementation details and not higher level specification details. Your tests should test your use cases not your implementation details.

In other words, if you can’t refactor your code without breaking your tests then you are doing automatic testing wrong. It’s that simple.