The issue is that very often it would only take you a couple seconds to go through anyway, so you end up forcing the other person to adapt in a way that leaves everyone either the same or delayed.

There are definitely places where letting someone go is appropriate, especially if it's leaving a gap so they can make a turn. But if you're on a road and you have to wave them through then most of the time you're not actually helping, and fake-helping is an annoying thing.