https://goo.gl/maps/DYvaWBSLFcP2 1h15m from Rye train station to Fulton subway station on a Tue morning
https://goo.gl/maps/TdQN5FhWF132 55m from Pelham station to Fulton subway.
So door to door, you're still looking at 70 min to 90 min commute one way, with the added stress of possibly dealing with train delays. I wouldn't be able to do that more than a couple years.
My point is, if you want a door to door commute of sub 45 min, consistently, then you would need to be in Hoboken/Jersey City/Secaucus/Weehawken/etc. That whole Hudson County corridor doesn't offer much in the way of decent gyms (swimming pools, racquetball/squash courts, basketball, etc), and neither does it have any decent hiking. People don't really have yards there either.
If you move out of that circle, you do start getting some amenities, but your commute starts looking like 45 to 60+min, and you have to park your car at the train station or get dropped off, etc. It's all personal preference, but for me, I would want all the benefits of owning a car, or all the benefits of not needing to own a car. Living in NJ and working in NYC seems like compromising on both.