More density does not mean more crime or more "deplorables" unless you actively enable people get into that position. It does not tarnish the environment worse than endless seas of sprawl housing and resultant insane amounts spent on transit. Literally the only reason to not enable growth is to profit from its absence.
For example, the entire point of doing this is to get housing costs down, and people hear that and think it will cause their property values to go down.
But those are actually two different things.
Housing costs are per square foot. They go down when there is a bunch of new housing because of supply and demand.
Property values include the value of the land. If you zone for high density, it causes the land to be worth significantly more because people will pay you a million dollars for it so they can bulldoze your house and build a tower there.