I think it depends to some extent on one's dialect of English–and that includes regional and subcultural dialects, as opposed to merely national variants. If one grows up in a region/subculture in which its use in a gender-neutral way is common, it is natural and automatic to view some uses of it as gender-neutral. If one grows up in a region/subculture in which gender-neutral use of the term is rare, viewing it as gender-neutral is much less natural and automatic.

It is also worth pointing out while sometimes its plural uses can be perceived and intended as gender-neutral (especially vocative plural uses), singular uses are almost always gender-specific, and even many plural uses are gender-specific too. A person who says "hey guys" is quite possibly intending it in a gender-neutral way; if the same person were to say "most guys are like that", that plural use is obviously talking about males only

The fact that the term is still gender-specific in many contexts, even among speakers and listeners who sometimes use and understand it in a gender-neutral way, leads some people to consciously choose to reject it as a gender-neutral term – sometimes even people who may have grown up with its gender-neutral use