Imagine leaving to travel one day, for a year, no contact with anyone. Then imagine who will be excited to see you versus who will be bitter and distant because you left. You don't need the latter.
I am of the view that you should be happy with what time you get with others, because it is a privilege to be thankful for. No one owes you their time, nor do you owe yours to others, and that doesn't have to be a bad thing.
I was livid. I fired off an angry text when he asked me where I was and he chose to argue rather than apologize. In these kinds of situations you have to decide when, why, and how, and whether to cut them off.
I decided that I would use the same rule for him as I do for my lady friends. Always meet somewhere I know and am comfortable with, never somewhere new. That way if they don't show up it's not a total loss. Also never drive anywhere but home, the coffee shop, or my bar right after work. The better and more flexible and more intelligent boundaries you put up, the more and better people you'll bring into your life.
But it takes fearlessness and guts.