I don't think it's particularly bad per GWh of energy produced. You just don't need that much uranium. Compare coal (still being actively mined and burned for energy!), natural gas, exotics for batteries/solar.

In Australia at least most of it comes from a copper mine that happens to intersect a uranium vein, so as long as the demand for copper exists its extraction is essentially environmentally free

Olympic Dan is the fourth largest copper deposit and the largest known single deposit of uranium in the world.