"Hey, who's your pick in the superbowl?" "I like to golf!"
"I'm actually more of a golf guy, myself."
"Oh really? What's your favorite club?"
"I love my MegaPutt 20x6 Owninator"
"Oh yeah that's a classic! The 20x6 line has really turned MegaPutt around in recent years. Myself, I can't get enough of anything SportStick comes up with, I got walls of their clubs at home."
The idea's that a conversation gets redirected, not just stopped outright like it would with a reply of "I don't watch sports."
DISCLAIMER: I don't know how golf works beyond what Wii Golf taught me, and that was years ago.
That never sufficed for knowing about football. People want to talk football.
They like talking pool with me too, and it's not like I was cast out of the social circle over it--but most sports conversations, in my experience, start out very specifically.
"Who you like in the SEC this year?"
"You been watching this cheating stuff with Belichick?"
Don't know who Belichick is? You're not going to fudge a conversation, most likely.
Plus it's OK if you suck at golf, everyone you're playing with sucks too. And if they don't suck, you make them look even better. And you get to walk around on a (usually) nicely manicured bit of grass on a (hopefully) sunny day.