I have done that before in a previous job, the end result is I was further alienated. Members of the team distanced themselves from me for fear of saying something wrong or reported. In the end, I had to change jobs because I couldn't fit in.
I suppose its not a problem of race, but a problem of just being different from others, like Age in you case.
Dunno what to do ati that point other than go to HR/their superior and file a proper complaint. Assuming your coworkers aren't some best-friends-5ever thing and your HR is competent, you could always claim hostile work environment