Your last paragraph reminds me of Hindsight Experience Replay[1] where they use goal-conditioned policies that learn how to reach essentially any state. The idea is that, even if you don't reach your true goal, you've learned how to reach some other goal, so you can successively learn to navigate the environment until the end goal is easy. It's a really nice paper, highly recommended!

[1] https://arxiv.org/abs/1707.01495