It wouldn't be unconstitutional for private contracts, i.e. you can make any private agreement restricting your speech - but as a condition of being in public office.