Yes, you can extend the logic forever, however that ignores the obvious and enormous power differential between the two parties.

The first has to go down and get their food, a minor inconvienience.

The other has to risk losing their delivery vehicle / property when it gets stolen while they are in the building (an enormous, possibly income-destroying inconvienience)

I'm saying that the person in the building should maybe have a little gratitude for their position and empathy for the delivery driver