If you feel that way about everyone, then you are a very different person to me (and probably OP).
If you feel these are the only two possibilities, we're definitely very different people.
It's mostly just 'yooooo I didn't realize you were near X' or 'how was Y? I saw it when checking where my buddy was'
Dunno maybe someone's has been messing with me all these years and I'm oblivious
Great! You tell them honestly, what could possibly go wrong. I'm sure nobody's feelings would be hurt when you tell them you weren't feeling like meeting them.
Now next time you drop by and still don't manage to catch them for whatever reason. Now your prior "candor" becomes a lie -- one that never needed to arise in the first place.
Maybe you enjoy getting yourself into these kinds of situations, but hopefully you can understand why others might not.
The heck? Is it that hard for you to imagine these occurring without lying? You have X planned, and now your plans change to Y. Now you owe everyone whose invitation you'd declined an explanation, or they wonder if you're a liar. Or I guess if they're like you, they already assumed you're a liar.
Because sometimes people can be unreasonable. (Bad day, drunk, generally difficult personality, etc.) The more people you add to your circle, the more you are likely to run into this.
Because it’s none of their business. You are not owed my time just because I’m nearby. That’s not a healthy boundary to have. Location sharing encourages “boundary creep” that forces you to more frequently justify and reinforce your personal boundaries, adding friction to the relationship.
You seem to have missed a crucial step in the conversation.
What happened is that the person listing said benefits explicitly asked "Why not?" and so received a response answering their question.
E.g. "it's draining" is... self-explanatory? I'm not sure what else you want to hear on this. Keep throwing draining problems in front of people and they will get tired of it and try to avoid the situation entirely. And if somehow it's not draining for you, surely you can understand your stamina doesn't generalize to that of the entire human population.
"If they're unreasonable then that's their problem" is just a silly strawman. If unreasonable people have a problem with you that can and often will quite easily become your problem...
"If you don't want to justify it then just don't" is basically the same as above.
Etc.
All of this "surely you can see" that you're saying is presumptuous and a strawman. Of course I understand not everyone feels the same as me, should I just shut up because my experience isn't universally transferable? Why aren't you telling the people who find it draining that not everyone finds it draining?
To avoid some awkward conversations many people would rather avoid, I guess?
"You were nearby? Why didn't you attend blah/come say hi/etc.?"
"You're here? Didn't you say you're going to be out of town?"
"What were you out doing at {time/place}?"
etc.