Also good to note that that the Pile contains lots of curated sources and recent trends have been to take curated data sources and combine them with filtered webcrawls (i.e. commoncrawl with heavy processing). See dolma or the stack v2 (for code models) as others have mentioned.