Exertion of power does not require a pseudo-court-system and it can be exerted in the furtherance of fairness just as much as in its hindrance (people are also punished because there's no enforcement of some behaviour). Again, to discuss whether something is worth doing or not, you have to consider all the upsides and all the downsides of doing it as well as those of not doing it.
Would you accept (or would one even send) an ad-hoc post-conference invite to 'talk about something you did wrong and how you should be punished' from a committee member?