The word “fan” has become really watered down over time. Didn’t it used to be short for “fanatic”? If someone isn’t deeply familiar with the landscape of a genre, then it seems logical to say they aren’t a fanatic.

It may just come down to whether one uses the watered down meaning of “fan” or the original.