But in a way, a lot of things are gambling - up to and including investing, particularly in startups. The desirability of a particular form of gambling in society is tied to the negative effects it has on people (e.g. how easy is it for a player to get addicted), and how exploitative the organizations operating a given game are. Under those metrics, fantasy sports are very much like lottery or one-armed bandit, and very much unlike investing.
And hell, it's not just about being condescending to people. I'm perplexed by how people tolerate, and even vocally support companies that intentionally fuck over poor people.
Cutting right to the more meta part of the discussion, it's just false equivalency constructed that hits both a desire to dump on sports fans and find a way to make an absurdly irresponsible habit practiced mostly by lower income people more palatable.