2. It really depends on the experiment. For causal inference, both type s and type m need to be very low, but type I can be high (I don’t care about rejecting a null). For inference where I only care about final predictive accuracy, this may not matter. For example in an extreme case I could have two perfectly collinear predictors, which means their coefficients can be arbitrary as long as the sum yields the true coefficient on the underlying linear component. If my goal is causal inference of the effect size, this would ruin it, since type m error can be unboundedly bad. But if the goal is overall predictive accuracy, it doesn’t matter at all - I just ignore the coefficients.
> “ it still seems like the final sample sizes you calculate could end up being very sensitive to the prior.”
I prefer to flip this around. A frequentist model has a prior too whether anyone wants to admit it or not - usually it is some unrealistic flat / uninformative prior or improper prior. The results of a frequentist method are equally sensitive to this implicit (huge) assumption. A Bayesian approach at least makes it explicit, admits the sensitivity, puts the range of prior choices out in the open for skeptical review, lets you carry out sensitivity analysis, and very often relies of real data and domain expertise to posit a much more physically plausible prior.
I'd be very curious about two follow-up questions here:
1 - A frequentist approach to the above could still be formulated in a minimax sort of way, but then you have to deal with alternatives which are close to your null. It's not like this problem goes away for Bayesians, it still seems like the final sample sizes you calculate could end up being very sensitive to the prior. Does this happen in practice?
2 - What kind of optimization goals do your users prefer when trading off power, type m, and type s? My guess based on this formulation it's something of the flavor "max power st P(type m or type s or type I) <= alpha", but wanted to check.