I know many people don't want us to risk becoming a "show your papers" country, but A) We already kinda are (ever been pulled over?), and B) It just makes more sense to have something like ID be centralized, preferably with a much better model then SSN's.
1. The ACLU-style left fear it will lead to more pervasive, easier surveillance, and more "papers please" style checks on poor people and immigrants.
2. The right hates it because it's an extension of government power, arguably a 10th amendment violation, and it would greatly simplify voting for people who traditionally vote democrat.
3. A nontrivial number of people believe (no-joke) that it would be a portent of the apocalypse, relating to the number of the beast in the book of revelation. This actually came up in a number of state legislatures as they standardized drivers licenses after 9/11.
The few polls I've ever seen actually say it's fairly popular with people, but those interest groups are non trivial.
People mainly rely on their state drivers licenses because states regulate driving. (And most other day-to-day government interactions that require ID)
If you’re the authority asking for ID, you get to decide which one to ask for.