A passport is only used for international travel, which was already under the purview of the feds. It is also only used for travel, which means that many people don't have one.

Drivers licenses (/IDs) are used for everything else pertaining to someone's daily life. In this way it affects how/who/where/when people drive, vote, buy and sell controlled things, engage in some financial transactions, travel domestically, obtain social benefits, etc.

It allows local authorities to be gatekeepers. A commonly used and carried federal ID would shift that gatekeeping authority to the feds. It could also enable additional gatekeeping which is not currently possible in the absence of this ID, which is a primary concern of the people who are against this.