But this is exactly how it's supposed to work. It's efficient to table side discussions that don't apply to everyone. Otherwise, you're just wasting a subset of the participants' time.
A proper agenda should contain everything that absolutely needs to be talked about at that time. That is the focus of the meeting. If additional discussions are spawned that aren't covered by it, they should be taken offline, either with an additional meeting, or with an informal discussion after.
I agree in this kind of situation, unexpected items are of limited scope, shouldn't affect the meeting agenda, and can be dealt at other time.
Now, this would be more the exception than the rule, at least in my experience. And if you can reach this point in your organization, I think you can get rid of the meeting altogether, and just validate the different points by mail or group chat.
If you've got 10 people in the meeting and 6 people start discussing something, you've got 4 people who are having their time wasted and were given no choice in the matter.
It's the same pattern as when you need to be in a meeting about a specific point, but not attend all the other points. You ask to have the stuff you care about come first and you can leave it after. If some unexpected things needs your input, you'll be called in anyway.
You don't have to schedule an entirely different meeting room to talk about it - more often than not at my work, the people who want to discuss the unexpected point just stay after the original meeting has ended to discuss it.
To everyone who was there for the agenda, the meeting is effectively over - for the others, you could say the meeting continues on with the unexpected topic.
The important part is that everyone meets for as short as is needed - they don't have to keep running in and out of meetings to see if things are relevant, or stay and listen to things that don't matter to them.
Of course if nobody cares about some tangent point, it should be stopped on the spot. If a discussion appears while some of the stakeholders are missing, it might be postponed as well, but if it's not the case there's no reason no to go on and discuss it, just because it's not on the agenda.