The big problem is the manual pollination and harvesting. Growing the vanilla can be done with a suitable climate in many tropical locales, but pollination requires a lot of care--you need to balance getting deep enough to actually pollinate the plant with being gentle enough to not destroy the plant. Most vanilla farmers have special tools they use to cut away parts of the flower and quickly pollinate it.