From a science angle it makes more sense to just collect this data from everybody all the time and let the studies be post hoc explorations of that giant dataset. But how do you get people to cooperate when they have reason to believe that the data will be used against them (by third parties who end up with the data)?

Either we accept the diminished value of isolated study-at-a-time datasets (as we're doing now), we figure out how to do this kind of research through a layer of homeomorphic encryption (a tall order), or we wait for an era with more trustworthy institutions (potentially a long time).

For this reason, I think it'll be like fusion power: the perpetual "next big thing".

It won't be. We aren't able to culture most gut bacteria. Most die immediately in oxygen.

The best we can do at the moment is correlation.