"Culture fit" can indeed be code for ageism and sometimes sexism, but more often it is just superficial cultural signifiers, such as dressing too formally / not formally enough, the wrong sort of facial hair, too much / not enough makeup, being the wrong kind of health fanatic, too many / too few positions in your job history, a percieved difference in appetite for financial risk, and so on.