Do them regularly, and keep your playbook of mock failure scenarios up to date with good coverage of all your systems. It's especially critical for disaster recovery (a DR plan that's never tested isn't worth the paper it's written on).

Consider going one step further and randomly injecting artificial failures into production shards, so handling them becomes a regular affair. When you build out monitoring, that might be a good time to think about the kind of stimulus that would be effective at exercising those monitors/metrics/alerts (think unit testing analogy). You can automate an evil adversary bot, or have humans do Red team / Blue team challenges. Yes, you're limited how much you can break without too severely impacting production, but if you've designed enough redundancy you should be able to achieve reasonable coverage. The more you can engineer simulated failures into your regular workflow, the less of a big deal it'll be when real ones occur.

Instead of watching the watchers, engage them with stimulus that keeps them sharp and prevents them from getting bored.