Usually not in my favor, especially if it's my word against somebody else's.

I wonder if there's a way to change how one thinks, to not assume that the other person won't trust what one says, and so avoid getting nervous / acting like a culprit

I wonder if thinking "I get paid anyway the same each month, what does it matter if they believe me or not" helps (unless one might get fired)

Yes, not giving a fuck is a good solution to many social anxieties.