There are lots of languages where the gender-neutral term and one of the gendered terms are the same. Consider the Spanish "ellos", meaning either "the masculine group" or "the masculine and feminine group". The word "ellas" denotes a feminine-only group.

It seems very reasonable that "guys" could function similarly in English. Moreover, I might not say "the guys I've dated" as a straight man, but if all the women I've ever dated were talking together in a room, I wouldn't bat an eye asking "what are you guys all doing?".

Similarly, I wouldn't say "all the folks I've dated", but I might say "good evening, folks" -- the word is a misfit solely because of the idioms and context, not because the word is inherently communicating something else.