Despite the sword and sorcery veneer this is actually the premise of several of CJ Cherryh's works, for example : https://en.wikipedia.org/wiki/Exile's_Gate