On the other hand, if you accept contributions while under a strict license like the AGPL but then decide to go MIT, you'll have to ask each contributor to relicense his or her work or remove that work altogether.

OP, if you see your project switching licenses down the road, you might as well look into CLAs, a whole other can of worms, or just stick with the least restrictive license that achieves your goals. I'd personally use LGPLv3.