I hate calling things like this a privilege. It's licensed and regulated for the safety and benefit of the people who fly and live underneath airspace (all of us).
There are too many things involving the government that have turned into privileges where it takes knowing someone or being able to afford access to obtain action.
- Having the money to afford the training for the pilots license
- Having the time to complete the training and stay current
- Having no health issues that may deny you a medical
It is pretty out of reach for most people with out the time / funds for it.
I do get what you're saying, but there are people using "privilege not a right" literally to mean that someone gets to decide whether the person is worthy beyond some standard criteria.
We’ll likely see amendments codifying those unwritten rules into existing legislation soon enough.
And I doubt this is the final action. FAA can't criminally prosecute but they can refer to the DOJ.
When you're in the air alone, there is no one to rely on but yourself, and failure to exercise proper judgment can kill people on the ground. So the FAA has broad leeway to determine that you don't have the right mindset for a private pilot's license.