But as an atheist of Jewish descent, what do I say to my co-workers, who are from Turkey, India, Korea and China? Of those who I know to follow a religion, none are Christian. It would feel outright silly to say "Merry Christmas".

(For comparison, try saying "Happy Hanukkah" to your non-Jewish friends as a non-Jewish person when the season comes, and see if it makes much sense to you).

My question is more whether you care if someone you don't know says it to you on the street or when you buy a coffee or whatever.

On the other hand, different people are rubbed the wrong way by many different things. I do try to avoid going out of my way to avoid offending people, by, say, assuming they're Christians. And on the other, other hand, I'm fine with offending people who want to go out of their way to be offended.