Segregation would take them out of the ecological niche where normal people provide for them. They’d be back to Stone Age survival outcomes.

Besides there is already a social taboo against engaging in sexual intercourse with them because of the substantial power asymmetry. So really the only impediment to speciation is the lack of fertility.