I think is used for arbitrary discrimination (unlawful or otherwise), but I'd suspect the majority of what's folded in there is just stuff that you don't want to put into official response.
I've conducted interviews as a chair-filler (not my interview, not my decision, but just to chip in and discuss the victim afterwards).
I have awarded negative points for: Mentioning your father's job 3 times. Continuing to draw on a whiteboard with a marker that's run out, when there were 4 others next to the pen you selected. Being rude to the receptionist. Telling me to be quiet until you'd finished drawing, when I tried to nudge you back to answering the question I actually asked. Not being able to provide any response/elaboration on a CV bullet-point.
Some of these you could shape into constructive feedback, but easier to just say they "don't fit"
While proof of violation may in some cases be difficult, because its very hard to measure the impact of one among several subjective criteria, I find it difficult to believe that there are actual “culture fit” decisions that are not disparate-impact discrimination. To wit, I do not believe that it is ever the case that “culture fit” decisions either:
(1) fail to have a substantial adverse impact on one or another protected class, or
(2) have such an impact but, in fact, effectively measure the minimum qualifications for successful performance of the job in question, so as to fall within the “business necessity” defense.
In fact, I think the predominant purpose of “culture fit” is to provide a cover for arbitrary discrimination by hiring decisionmakers with which unlawful discrimination can be masked in a way which makes it more difficult to litigate, as a liability risk mitigation technique.