Agreed, in addition LLMs are trained on a lot of useless data, such as the whole Reddit dataset which has to be at least 90% Reddit garbage, and that doesn’t seem to be a problem. I don’t see why ai labs wouldn’t want to also train on older data