I agree with you, but the issue is that the Uber driver doesn't have control over it. If the driver were in charge, they would make their policy that the ride starts on arrival of the passenger and ends on departure of the passenger. But Uber doesn't do that -- they start the ride when the car moves and ends it when the car stops at the destination. So the driver loads and unloads on their own time. They aren't compensated for differences in accommodations per passenger.

I try to make up for this by telling the driver to start the ride the moment I arrive and not end until I leave, but not everyone does that.