I used to pepper my comments with phrases like "to be honest" but I never use it in speech. I've tried to eliminate it recently because it usually isn't needed.

For me, I think phrases like "however" and "to be fair" may come up a lot because I actively try to remain objective in conversations, and now I subconsciously add those into what I say as an indicator of considering different viewpoints. Then again, I may just be over-analyzing it.

(While we're at it, add "then again" to that list, haha.)