For the same reason height is, it is one of the first thing we notice when we look at a person and it rarely changes.

The literal answer is that it's a fashion from early 1800's Europe that carried forward, fell out of favour in 1861 when France and other European countries abolished passports and that came back again as World War I loomed. *

I mean that's the history of passports, which started out with an indication of the person, but it's a fair question to ask why the hell are there passports anyway.

( * Sure, you could start with limited issue by King Henry V of England in 1414 but that's really getting in the weeds here. )