Imagine leaving to travel one day, for a year, no contact with anyone. Then imagine who will be excited to see you versus who will be bitter and distant because you left. You don't need the latter.
I am of the view that you should be happy with what time you get with others, because it is a privilege to be thankful for. No one owes you their time, nor do you owe yours to others, and that doesn't have to be a bad thing.
I was livid. I fired off an angry text when he asked me where I was and he chose to argue rather than apologize. In these kinds of situations you have to decide when, why, and how, and whether to cut them off.
I decided that I would use the same rule for him as I do for my lady friends. Always meet somewhere I know and am comfortable with, never somewhere new. That way if they don't show up it's not a total loss. Also never drive anywhere but home, the coffee shop, or my bar right after work. The better and more flexible and more intelligent boundaries you put up, the more and better people you'll bring into your life.
But it takes fearlessness and guts.
One or two children will be all I have, I am never going to start a startup, band, or other such long shot waste of time. I only have so much to give, and I ultimately will always put myself first, because when I put myself first, the quality of what I have to give to the people I truly care about and who care about me goes way up. The people who fall short of that all have their own lives that I am not and never will be responsible for. I've learned this the hard way too many times.