The question is whether or not it has to be this way. Short of politics there is no reason that the LIRR and Metro-North could not also work for intra-city or intra-regional travel; in fact, many umbrella organizations contain merged separate railroads for exactly this purpose (STIF operates the Paris RER, Berlin's BVG operates the S-Bahn, TfL operates the Overground and TfL Rail, etc.)

Without the regional integration, we would probably never get things like the Freedom Ticket; utilizing commuter rail lines as de facto subways, even premium ones, is much preferable to the much more expensive option of duplicating existing corridors.