I agree with your points entirely, but "culture fit" is a very ugly thing when used to exclude people that cannot be legally discriminated against. There's a reason these employment laws exist; people have a right to a job they are qualified to do regardless of whether they share the hobbies or opinions of the interview team.