It's not just about testing their efficacy it's about the theoretical limits of pure observation when doing causal reasoning. We know that we are better served by avoiding causal certainty when using purely observational studies. It seems like the base assumption should be that similar epistemic constraints apply to machine learning.

Yes the best way to understand a system is to interact with it. But there are scenarios where that simply isn't possible and yet we can still model causality, like the weather example musingsole gave.