My point is, if you want a door to door commute of sub 45 min, consistently, then you would need to be in Hoboken/Jersey City/Secaucus/Weehawken/etc. That whole Hudson County corridor doesn't offer much in the way of decent gyms (swimming pools, racquetball/squash courts, basketball, etc), and neither does it have any decent hiking. People don't really have yards there either.
If you move out of that circle, you do start getting some amenities, but your commute starts looking like 45 to 60+min, and you have to park your car at the train station or get dropped off, etc. It's all personal preference, but for me, I would want all the benefits of owning a car, or all the benefits of not needing to own a car. Living in NJ and working in NYC seems like compromising on both.