Indeed, it is frustrating. It may give the impression that you are defective in some way (to improve yourself), but doesn't necessarily mean that. As the job candidate, I have declined offers because of culture fit. I have also left some companies due to culture fit. It's really a two-way street.

In a lot of cases, I think it's better to get rejected up front than to get hired on only to realize later on that you've landed in a bad situation. To me if a company wants to reject you based on culture fit, many times it's a blessing in disguise.