This has become a colloquial term for describing all sports, not the insult you're perceiving it to be.
Rather than projecting your own hangups and calling people names, try instead assuming that they're not trying to offend you personally and are just using common vernacular.
If only there was an existing way to refer to sports generally! And OP was referring to a specific sport (baseball), not sports generally.
The Rogers Centre hosts baseball, football, and basketball games - so in this case "sportsball" was just a shorthand for all these ball sports.
Ah, Mr. Kettle, I see you've met my friend, Mr. Pot!
Would you get incensed by "petrolhead", "greenfingers" or "trekkie"? Is that what you choose to be emotional about?
This is not the flex you think it is. You don't have to like sports, but snarking on people who do doesn't make you intellectual, it just makes you come across as a douchebag, no different than a sports fan making fun of "D&D nerds" or something.