You don't have to support the Citizens United decision to think that contributing small amounts to political campaigns is effectively speech - because the money is used to promulgate speech (ads), as well as in the more abstract sense that donations are viewed by society as a normal and encouraged outlet for people to promote their viewpoints.
(Of course the humongous donations the decision enabled are also used to promulgate speech, which is why I have some sympathy for it, but there's just a huge difference in practice between giving $1,000 and giving millions.)
So in a very literal way, he was pushed out due to the manner in which he chose to speak.
0: http://www.scotusblog.com/case-files/cases/citizens-united-v...