Most folk will have headphones, most folk are focusing on their routine / program; themselves.

Being a regular is kind of a must, know that the person you wish to strike conversation with is a regular. Find a time when they're not plugged in to their zone and then start with a compliment. Keep it small and lite, everyone likes to be complimented now on their workout now and then.

Further it on if you've noticed that they've completed a set on new weights, extra speeds et cetera. Then the next time you see them at the gym, make a friendly hello.

Over the periods of small talk and rapport you can make larger gestures such as sharing sets, running next to them if it's a treadmill.