Politics is the art of the possible. Given what post-slavery America looked like for the former slaves (not much different), the question becomes: do you care so much about slavery, which is an ancient institution, that you give up the idea of forming a united front?

Was it good? No, it was a bad idea. But that's something we say comfortably from our homes in a large, powerful country. In 1787... this was a remote and weak place. If you wanted the slave-dependent colonies to join, you had to either buy their slaves from them or allow it to continue. And there wasn't enough money to buy them.