Slave trade brought in massive wealth into Europe back then. That is why people did it, really. Otherwise they would not bothered with it.

The south had slavery, because it was massively profitable. They did not done it for fun and giggles. Owners fought to keep slavery multiple times, because they wealth was in slaves and in what slaves produced. The slave owning families were the richest and most powerful.

Same with colonialism. It was not done out of boredom, it was done because resources you get from all of it.