It's not retroactive if you know about it ahead of time. It would be known when the contract is made that the money would by law have to be returned in the event the patent is invalidated.

The consequence would obviously be that all the licensees of a patent that has any chance of being invalid would subsequently try to get it invalidated so they can get their money back, but isn't that the intended result?