I'd personally find removing favorite rather frustrating as it's a nice way of differentiating posts than just upvoting so that I can easily find them again in the future...