Yikes, how did you respond to this? Was it an off-hand comment or a serious accusation on their part (were they legitimately offended)?

Is this co-worker a peer, a superior, or are you their superior?