Make sure you have on file release forms for every copyright holder who submits code that grant you permission to relicense later on. For one line bug fixes it can be a pain, but it's worth it.
One problem with this though is that it places an unfair advantage with the current maintainer. A fork of the project wouldn't inherit the relicensing permissions from all the contributors.
Ogre3D was able to successfully transition from LGPL to MID/BSD because of contributor agreements.
The Apache Software Foundation, Free Software Foundation, and others all require contributor agreements for very good reasons.
If your projects starts out with very liberal licensing, it's not necessary strictly speaking (unless you're concerned about patents). But that's the tradeoff.
Contributor agreements are a necessary mechanism to protect the project and ensure its future when dealing with patents or preparing for the potential of relicensing.
By choosing the (A)GPL but forcing contributors to grant relicensing rights, you've excluded both those who consider copyleft licenses non-free, and those who consider copyleft licensing an important requirement.
And while the Apache Foundation already has a "permissive license" it's not quite as permissive as BSD / etc.
And if the Apache Foundation wants to relicense, to say, a future version of their license, they're still going to potentially need that contributor agreement.
Again, as I said before, there are tradeoffs.
BTW, beware of code ownership of pull requests from corporations.