The downside, mostly, is that your next job may need you to move again. Many people prefer stability in their housing arrangements, and are willing to trade off a lot for that.