Yes, sure - any time different people will be getting different sets of warnings, -Werror (or equivalent) is just going to make your life harder.

You should always strive for zero warnings on all builds across all compilers. It's a realistic, achievable and worthwhile goal. But while pretty much all warnings are worth looking at, a much smaller set are worth breaking the build over. You're much better off (in my view) promoting those particular warnings to errors, and leaving the rest as-is.