Because it would have tore up the social and economic fabric of significant parts of the US. The Founders were ahead of their time, but they still recognized that the vast majority of their countrymen were not.