A big part of it you're missing is that it's a joke. Anime fans definitely know they're weird, and are very passionate about the things they like (and are conscious of that), and thus a lot of humor in the community is self-deprecating and ironic (for example, calling your favorite fictional character your "waifu" or "husbando"). The fact that outside observers might think it's "problematic" is kind of the whole point.