Did it really say "vacation"? I think they meant "travel" as in going to another city (say, in a chartered bus packed full of fans, and which goes back home afterwards) to see your team's away game; at least that kind of travel is what brings most of the out-of-town spectators -- who are what's under discussion here, right? -- to most of the big inter-city sports events I know of.
For that kind of "travel", going to see a regular game of some sport is the main objective.
But I know people who plan trips abroad, for instance from Helsinki (Finland) to Liverpool and Manchester (UK) etc, mainly to see football. Sure, going that far they do other stuff to -- visit team museums, for instance:-) -- but the matches are the backbone the rest of the journey is scheduled around. Maybe there aren't all that much fewer of that kind of people than there are fanatic foodies? I really don't know. But yeah, of course both categories are a drop in the ocean compared to ordinary tourists.