West Africa was never "incredibly wealthy". Yes, before Europeans it was part of long-range trading routes with the Muslim world, but empires were feudalistic and only a tiny elite had access to that trade-related wealth, while the vast majority of the population was barely surviving through subsistence farming and pastoralism, just like so many do today, or hunting and gathering.

Lets also not forget the slavery and slave trade that was practiced in Western and Northern Africa.