If the problem is shallow correlations, then we can use hard example mining to clean up the dataset. Train a simple model and skip all examples that are easily being solved. What remains is a harder dataset where simple correlations don't cut it.