I'm sure this is covered thoroughly in legal studies, and IANAL. But here's my current take on it:

If we're going to let people sell patent rights in a free market, I'm guessing we need to let patent holders chose to not sell at a given price. Otherwise they have severely limited negotiating power.

I.e., suppose we did add a use-it-or-lose-it patent rule. Think of the ways that you, as a would be rights-purchaser, could exploit that.