Yes, I do that in speech and written text. "However" is another one I use a lot.

For me, I think phrases like "however" and "to be fair" may come up a lot because I actively try to remain objective in conversations, and now I subconsciously add those into what I say as an indicator of considering different viewpoints. Then again, I may just be over-analyzing it.

(While we're at it, add "then again" to that list, haha.)