I recognize the feeling somewhat. First time I experienced it was when considering traveling out in the solar system, but perpendicular to the ecliptic. I liken it to thalassophobia or some deep fear of being completely isolated.