West Africa was never "incredibly wealthy". Yes, before Europeans it was part of long-range trading routes with the Muslim world, but empires were feudalistic and only a tiny elite had access to that trade-related wealth, while the vast majority of the population was barely surviving through subsistence farming and pastoralism, just like so many do today, or hunting and gathering.