Maybe this isn't a typical view of 'culture fit' but I would think of it more in terms of whether a candidate is reasonable, pleasant, friendly, etc. (matching the extent of those traits between the candidate and the company average -- maybe a company of argumentative folks wants to continue along that path instead). Such matching would, I think, have little impact on diversity of perspectives, neutrality regarding protected classes, etc.