Or pretend to have lost their passport and get a new passport without stamp. Or get a secondary passport that's made available by most governments because of issues like this one.

But you are right in principle. Letting people through because they look important is a big security hole.