A model being coupled to a system == a model that can influence the system's state through a means

> a model that is meant to guide action (basically any useful model) needs to be tested in action

No, it doesn't. For example, the vast majority of work on modeling the stock market is done on machines completely sandboxed from any ability to make trades and are owned by companies who will never make a trade themselves but instead return an API response with a yes/no. Whether that is fed directly into some sort of automated action is largely irrelevant as the ability for an individual trade to cause a measurable impact on the market is negligible until it isn't. So, these systems are built separate from the system they model and learn entirely through observation.

tl;dr: weather forecasting models don't have an action to take and also can't influence their system. And yet they learn and grow more accurate.

OK that's a fair criticism. Then perhaps we can divide models into those that influence the system they observe (regulatory systems) versus those that only measure, or whose influence is negligible. Models that aim to influence a system do indeed need to be used to test their efficacy.

It's not just about testing their efficacy it's about the theoretical limits of pure observation when doing causal reasoning. We know that we are better served by avoiding causal certainty when using purely observational studies. It seems like the base assumption should be that similar epistemic constraints apply to machine learning.

Yes the best way to understand a system is to interact with it. But there are scenarios where that simply isn't possible and yet we can still model causality, like the weather example musingsole gave.