Velocity in physical terms is an instantaneous measure. It's the same in Agile, you've done X things in Y time. But that ignores the context of those X tasks and the context of Y time.
The X+1 task might turn out more difficult or have some other challenge that makes it take longer. Time keeps passing so at time Y+2 the velocity measurement is lower than before. Now all the PMs and other team members jump in demanding answers which now makes the task take even longer.
As you mention, some span of time can see team members out or unavailable. So the velocity is "low" but rarely is that context captured or bubbled up the management chain. So your team has "low velocity" which ends up generating worthless meetings, bad reviews, or all sorts of problems.
It's not part of Agile. It's specific to Scrum.
I've found that most Product Owners really care about predictability and less about actual estimates. They care that if you say you're gonna finish something by the end of the sprint, much more often than not, you will. Estimates and velocity and such things are just means to the end of trying to figure out which items are reasonable to take into a sprint together.
If your Product person knows that they can throw 10 "random" (but currently most wanted by stakeholders) work items into a "sprint" and 9 times out of 10 you will actually finish all of those, they're probably gonna be very very happy as they won't have to explain delays over and over again on their end.
I don't see how this can be the case. If each ticket is estimated to be about the same size, and it sounds as though they should be, then if you're doing say 5 tickets a week, 10 a sprint, and they're all about the same size then each ticket you could say is 5 points, and you're doing 50 points a sprint.
That's probably pointless (hah) if they are all the same size; ticket sizing is more for if you have tasks of different sizes and you're trying to roughly combine them into something you can predictably deliver.
But if you're at a stage when your delivery is already this predictable, you may not need to use story points particularly. If it's mandated for some silly reason then you could use the above method to bat it away quickly!
I mean, just the difference in an iteration with a couple of people on PTO, or a holiday or two intruding into it, will throw the numbers off. Now factor in all of the other fuzziness that's inherent in the process... yeah, no. Don't bother trying to calculate or track "velocity". You'd probably get better results from Tarot cards or animal entrails.