It also depends on your model. If you do not need to estimate the entire distribution, but are fine with getting some moment estimates (aka - an average effect, the marginal of the expected output and so forth) then you do not even need independence. What you need is no correlation with confounders. Since this includes latent factors, the philosophical issue is, of course, the same.