I think you lost all credibility when you made that tweet. If you want to regain it, you should be completely transparent about which sites you have taken any specific ranking action regarding, and why. This should be relatively easy to do, as you say you take action very rarely.