While OSI don't require the training data itself to be under a FOSS license, they do still at least require "the complete description of all data used for training", including where to obtain it, such that "a skilled person can build a substantially equivalent system".

I think there's a legitimate tension between pragmatism and ideals, without necessarily a clear answer. Needing to be able to relicense/distribute all training data would be a significant constaint on the feasibility of developing open-source models, possibly entirely suffocating when even simple models tend to be pretrained on ImageNet/etc.